Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
👋
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Training Mixture-of-Experts Models
·
Module 4
Training and Measuring Expert Specialization
1
Reproducible Training with a Tiny MoE Transformer
Run a reproducible short training experiment using the tiny MoE transformer and a supplied synthetic next-token dataset with identifiable token groups.
2
Comparing MoE and Matched-Compute Dense Models
Compare the MoE model with a supplied approximately matched-active-compute dense baseline, reporting validation loss alongside total and active parameters.
3
Quantifying Expert Specialization via Held-Out Routing Distributions
Quantify expert specialization using held-out routing distributions conditioned on the synthetic dataset's token groups.
4
Testing Specialist Contributions via Ablation and Held-Out Loss
Test whether an apparent specialist contributes to predictions by measuring held-out loss changes when that expert's contribution is ablated.
5
Diagnosing Routing Collapse vs. Useful Specialization
Diagnose routing collapse from expert assignment counts, router entropy, and overflow traces, distinguishing it from useful specialization.
6
Diagnosing and Stabilizing Router Numerical Instability
Diagnose router numerical instability from logit and gradient traces and select a targeted stabilization intervention.
Previous module
Load Balancing, Router Stability, and Capacity
Next module
Expert Parallelism and Communication Costs