Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
đź‘‹
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Training Mixture-of-Experts Models
·
Module 5
Expert Parallelism and Communication Costs
1
Expert vs. Data Parallelism in Sharded Model Layouts
Identify expert-sharded and replicated model components in a simple expert-parallel layout, distinguishing expert parallelism from data parallelism.
2
Tracing Token Dispatch and Output Return in Two-Rank Top-k Routing
Trace token dispatch and output return through all-to-all exchanges in a two-rank top-k routing example.
3
Estimating Forward and Backward Communication Volume
Estimate forward and backward communication volume from token count, hidden width, top-k, element size, and the fraction of assignments crossing ranks.
4
Per-Rank Memory Estimation for Sharded Expert and Replicated Shared Weights
Estimate per-rank parameter, gradient, and optimizer-state memory for expert-sharded weights and replicated shared weights under stated precision assumptions.
5
Two-Rank Expert Dispatch and Combination Simulation Validation
Implement a single-process simulation of two-rank expert dispatch and combination, checking its output against the unsharded sparse layer.
6
Diagnosing Throughput Limits from Phase-Timing Traces
Identify whether expert computation, communication, or rank imbalance limits throughput from a supplied phase-timing trace.
Previous module
Training and Measuring Expert Specialization
Next module
Failure Diagnosis and Readiness to Scale