Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
đź‘‹
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Training Mixture-of-Experts Models
·
Module 3
Load Balancing, Router Stability, and Capacity
1
Calculating Hard Assignment Fractions and Mean Router Probabilities per Expert
Calculate hard assignment fractions and mean router probabilities per expert, using consistent normalization for top-k routing.
2
Auxiliary Load-Balancing Loss with Router Probabilities and Assignment Statistics
Implement an auxiliary load-balancing loss using router probabilities and non-differentiable assignment statistics.
3
Router Z-Loss: Controlling Logit Scale, Not Expert Balance
Implement router z-loss and explain how it constrains router logit scale rather than directly enforcing balanced expert usage.
4
Per-Expert Token Capacity and Overflow Calculation
Calculate per-expert token capacity and overflow counts from batch token count, top-k, and a stated capacity-factor convention.
5
Capacity-Limited Dispatch with Residual Preservation
Implement capacity-limited dispatch with an explicit policy that drops overflowing expert contributions while retaining the transformer's residual path.
6
Capacity-Limited vs. Dropless Routing on Skewed Batches
Compare capacity-limited and dropless routing on a fixed skewed batch using dropped assignments and peak expert batch size.
Previous module
A Minimal Sparse Expert Layer in PyTorch
Next module
Training and Measuring Expert Specialization