In this course, you will move from the mechanics of sparse routing to a small implementation, then examine training behavior and the costs of distributing experts across devices. This first module establishes what happens inside one MoE transformer block. The key change is local: attention still produces a contextual representation, but a router chooses a small subset of feed-forward networks to process each token instead of sending every token through the same feed-forward network.
By the end of this lesson, you should be able to trace one token from the block input, through attention and routing, to the block output. The selection calculations themselves come in the next lessons.
Locate the replacement
A dense pre-normalization transformer block can be written schematically as
The first residual addition produces , a context-informed representation. The FFN then acts independently on each token position. In a sparse MoE block, the second equation changes; the attention sublayer and both residual paths remain:
The diagram shows this arrangement in an encoder block. A decoder-only block can use the same MoE placement, with causal masking in its attention sublayer.
Each box marked “FFN” is an expert: a feed-forward network with its own learned weights. The boxes are alternatives available to the router, not stages that a token passes through in sequence.
makeMoE: Implement a Sparse Mixture of Experts ...
Read the “Creating an Expert module i.e. a simple Multi Layer Perceptron” subsection of makeMoE to connect the diagram’s expert boxes to an ordinary FFN implementation.
Start at the replacement. Continue through the Expert class, ending at its return self.net(x) line. Focus on the up-projection, nonlinearity, and down-projection: an expert still accepts and returns a vector of the model’s embedding width. You can leave the preceding attention code for later.
That matching input and output width matters. It lets the MoE result be added to by the same residual connection used around a dense FFN.
Follow one token through the router
Let be the vector at token position after the attention residual addition, and let . The router takes and produces one score per expert. If there are four experts, its output for this token has four entries. A top- choice retains the indices of the highest-scoring experts.
Watch this short visual account of scoring, selection, and combination. It uses a particular weighting convention; for now, concentrate on which experts actually run.
Mixture of Experts (MoE), Visually Explained
Watch “Mixture of Experts (MoE), Visually Explained” by Jia-Bin Huang for a token-level view of the router and expert outputs.
Watch token routing. Follow the distinction between the router’s scores for all experts and the small set of experts selected to evaluate the token. The video uses full-softmax probabilities as output weights; another common convention renormalizes weights over selected experts, which you will compare in a later lesson.
Suppose the router selects experts and out of four. The token’s normalized vector is sent to both selected experts. Neither nor needs to evaluate this token. Selection is made for each token at each MoE layer: two tokens in the same sequence may choose different experts, and the same token position may make a different choice in a later layer.
This is what sparse describes. The router can score every expert, but only a subset performs the comparatively substantial FFN computation for a given token. It does not mean that attention is sparse or that unselected experts cease to exist in the model.
Combine expert results and restore the residual
Each selected expert receives and returns a vector with the same width as . The router also supplies a weight for each selected contribution. In the simple case where the selected weights sum to one, the MoE output is
where is the set of experts selected for token . The block finishes with
For a concrete trace, suppose . The router selects and , with weights and . Suppose their outputs on are and , respectively. Their weighted contribution is
The block output for that position is therefore . The experts did not each produce a separate replacement token. Their contributions were combined into one vector at the token’s original position, then added to the attention-side residual .
A useful distinction is that the selected indices determine which FFNs run, while the weights determine how strongly their outputs contribute. Some implementations calculate a softmax over all experts and retain the selected weights; those retained weights need not sum to one. Others normalize only the selected scores, as in the example above. Both preserve the same essential token path. The next lessons will make their arithmetic precise.
Read the block as an implementation
For a batch shaped , attention returns a tensor shaped . The router produces scores shaped , where is the number of experts. Top- selection records expert indices per token. Implementations commonly gather tokens assigned to each expert into a temporary batch, run that expert once on its assigned tokens, and accumulate weighted outputs back at the original positions. This batching rearranges where computation happens; it must not rearrange the final token sequence.
makeMoE: Implement a Sparse Mixture of Experts ...
Return to makeMoE to see the token path expressed in code, from selected expert calls through the block’s final residual addition.
In “Creating a sparse Mixture of Experts module,” read the dispatch explanation, then examine the SparseMoE.forward method from gating_output, indices = self.router(x) through return final_output. Track how expert_mask identifies tokens for one expert and how weighted results are added to final_output. Then, in “Putting it all together,” read the block introduction and the Block.forward method, from x = x + self.sa(self.ln1(x)) through return x. Relate its two additions to the two residual additions in the equations above. Stop before the full language-model class.
One training consequence is already visible in this forward pass: selected experts see the tokens routed to them, while an unselected expert performs no FFN evaluation for those tokens. Across a batch, this can give experts very different amounts of work. Later modules will address how that affects training, capacity limits, and communication; the immediate point is to keep per-token selection distinct from the set of experts used somewhere in an entire batch.
Takeaway
A token enters the block through the usual attention and residual computation. Its normalized, context-informed vector is scored by a router, sent to only the selected FFN experts, and represented by a weighted combination of their outputs. That single combined vector is added back to the attention-side residual at the token’s original position.
You now have the complete forward-path picture. The next lesson will open the router box and calculate its scores and probabilities from a token vector and learned router weights.