Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
đź‘‹
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Training Mixture-of-Experts Models
·
Module 2
A Minimal Sparse Expert Layer in PyTorch
1
Tracing Tensor Shapes and Index Operations in a PyTorch Mixture-of-Experts Router
Trace tensor shapes and index operations in a supplied Python/PyTorch scaffold containing a router and a bank of feed-forward experts.
2
Top‑K Router: Expert Indices and Gate Weights for Flattened Tokens
Implement a top-k router that returns expert indices and gate weights for a flattened token batch.
3
Token-to-Expert Dispatch with Position and Slot Preservation
Implement token-to-expert dispatch while preserving each token's original position and selected-expert slot.
4
Weighted Expert Output Combination with Indexed Accumulation
Implement weighted combination of expert outputs back into the original token order using indexed accumulation.
5
Validate Sparse Layer Outputs and Gradients Against a Dense Reference
Check a sparse layer's outputs and gradients against a reference that evaluates every expert and masks unselected contributions.
6
Replacing a Feed-Forward Sublayer with a Sparse Expert Layer
Replace one feed-forward sublayer in a supplied tiny decoder-only transformer with the tested sparse expert layer.
Previous module
Sparse Routing and Parameter Accounting
Next module
Load Balancing, Router Stability, and Capacity