Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
👋
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Generative AI for Autonomous Driving
·
Module 6
Scaling with Mixture-of-Experts
1
Sparse Activation for Efficient Model Scaling
Explain conditional computation and how sparse activation allows scaling model parameters without proportional compute cost
2
Mixture-of-Experts Layer Architecture
Describe the architecture of a Mixture-of-Experts layer: router network and parallel expert FFNs
3
Mixture of Experts: Gating and Routing Explained
Explain the gating/routing mechanism that assigns tokens to a subset of experts
4
Load Balancing in MoE: Understanding Auxiliary Losses
Describe the load balancing problem in MoE and the auxiliary losses used to mitigate it
5
DeepSeek-V3 MoE: Shared and Routed Experts
Explain DeepSeek-V3's MoE design with shared and routed experts
6
MoE Parameters: Total vs. Active Per Token
Distinguish total parameters from active parameters per token in MoE models
Previous module
Efficient Attention: From MHA to MLA
Next module
Reinforcement Learning Foundations