Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
đź‘‹
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Training Mixture-of-Experts Models
·
Module 1
Sparse Routing and Parameter Accounting
1
Tracing Tokens Through a Sparse Mixture-of-Experts Transformer Block
Trace a token through a transformer block whose feed-forward sublayer is replaced by a sparse mixture of experts.
2
Calculating Router Probabilities from Token Representations and Learned Weights
Calculate router probabilities from token representations and learned router weights in a small numerical example.
3
Top-k Expert Selection and Gate Weight Renormalization
Perform top-k expert selection and distinguish full-softmax gate weights from weights renormalized over the selected experts.
4
Calculating Total and Per-Token Active Parameters in MoE Transformers
Calculate total and per-token active parameter counts for an MoE transformer under an explicit convention for counting shared parameters.
5
Estimating Expert-Layer Arithmetic from Width and Expert Selection
Estimate expert-layer arithmetic per token from expert width and the number of selected experts, distinguishing parameter count from computation.
6
Gradient Flow Through Hard Top-k and Renormalized Top-1 Routing
Determine which router and expert parameters receive task-loss gradients through hard top-k routing, including the special case of renormalized top-1 gates.
Previous module
Next module
A Minimal Sparse Expert Layer in PyTorch