Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
👋
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Generative AI for Autonomous Driving
·
Module 4
Modern Transformer Components
1
RMSNorm: Simplifying LayerNorm
Explain RMSNorm and how it simplifies LayerNorm by removing the mean-centering operation
2
SwiGLU: Gating for Enhanced FFN Capacity
Describe the SwiGLU activation function and how its gating mechanism increases FFN capacity
RoPE: Position as Complex Plane Rotation
Derive how Rotary Positional Embeddings (RoPE) encode position as a rotation in the complex plane
4
RoPE: Relative Attention for Extrapolatable Context
Explain why RoPE produces attention scores that depend only on relative position, enabling context length extrapolation
5
Pre-norm vs. Post-norm Transformers: Stability Analysis
Compare pre-norm and post-norm transformer block configurations and their effects on training stability
Previous module
Transformer Architecture and LLM Basics
Next module
Efficient Attention: From MHA to MLA