Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
👋
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Large Language Models
·
Module 8
Advanced Architectural Innovations
1
RoPE vs. Absolute Positional Encodings
Explain Rotary Positional Embeddings (RoPE) and contrast them with absolute positional encodings.
2
Mixture of Experts: Scaling Parameters, Not Compute
Describe the Mixture of Experts (MoE) architecture and explain how it increases parameter count while keeping computational cost per token constant.
3
Router Networks and Load Balancing in MoE Layers
Explain the role of the router network in an MoE layer and the concept of load balancing loss.
4
Beyond Vanilla: RMSNorm & SwiGLU in Modern Transformers
Compare modern architectural components like RMSNorm and SwiGLU to their original Transformer counterparts (LayerNorm, GELU).
Previous module
Scaling Laws and Emergent Capabilities
Next module
Decoding Strategies and Inference Optimization