Hello. Earlier modules established how a router selects experts and how a sparse layer dispatches and combines token outputs. This module turns to training behavior: whether experts receive a reasonable share of work, whether router scores remain stable, and what happens when an expert reaches capacity.
The first step is measurement. In this lesson, you will calculate two batch-level quantities for each expert: the fraction of actual assignments it receives and its mean router probability. They sound similar, but top- selection makes them different.
Two views of the same routing batch
Suppose a batch contains tokens, there are experts, and each token selects distinct experts. For token , let be the probability assigned to expert by a softmax over all router logits. Let be that token’s set of selected experts.
The hard assignment fraction for expert is
There are exactly selected token–expert pairs, so this denominator makes the fractions sum to one:
By contrast, the mean router probability is
Think of as a count of decisions made and as an average of scores assigned before selection. An expert can have substantial without appearing often in the top : it may consistently receive a respectable probability but rank just below the selection cutoff.
In “MoE Token Routing Explained: How Mixture of Experts Works (with Code),” Hugging Face shows how top- masking turns unselected positions into zero-weight positions. Watch the normalization portion with one distinction in mind: the selected-only weights shown there are not the full-softmax probabilities used to calculate in this lesson.
MoE Token Routing Explained: How Mixture of Experts Works (with Code)
In “MoE Token Routing Explained: How Mixture of Experts Works (with Code),” Hugging Face visualizes the difference between selecting experts and assigning weights to their outputs.
In the segment on top-k expert logits, watch masking and softmax. Notice that excluded experts receive zero selected-output weight. Keep separate the full distribution over all experts, which we will average to obtain P_i.
A useful notation separates the two kinds of weights. If a sparse layer renormalizes probabilities over the selected experts, its output gate is
The gates determine how selected expert outputs are combined. They do not replace in the definition of , and their magnitudes do not replace the indicator in . For example, a selected expert with gate still receives one hard assignment.
Fix the counting convention before calculating
Top- load measurements are easy to misread if their denominator is left implicit. The following practice resource states the convention we will use: count all assignments for , but average the full-softmax probabilities over tokens for .
Top-k MoE gating and the Switch load-balancing loss - Practice - AI Engineering from Scratch
Read “Top-k MoE gating and the Switch load-balancing loss” from AI Engineering from Scratch to pin down the two normalizations before working through a batch.
In the “Task” section, read the “Routing” paragraph and the following “Load-balancing loss” paragraph. Start where the routing instructions say to sort experts by logit, and stop after the statement about all tokens going to one expert: routing and loss. Focus on why the assignment denominator is Tk, while each P_i uses the full softmax and denominator T. The resource’s observation about all tokens going to one expert concerns top-1 routing; with distinct top-k selections for k>1, each token makes several assignments.
Some descriptions instead report the fraction of tokens selecting expert :
This is a valid statistic, but for fixed top- routing it sums to , not one. Under our convention, . For top-1 routing the two definitions coincide. Neither statistic should be silently substituted for the other in a formula.
The supplied diagram depicts this top-1 case. Its counts tokens whose single argmax is expert ; its averages the corresponding full-router probability across the batch.

A complete top-2 calculation
Consider four tokens, three experts , and . Assume these are the probabilities after softmax over all three experts:
| Token | Selected experts | |||
|---|---|---|---|---|
| 1 | 0.60 | 0.30 | 0.10 | |
| 2 | 0.20 | 0.50 | 0.30 | |
| 3 | 0.40 | 0.35 | 0.25 | |
| 4 | 0.45 | 0.40 | 0.15 |
First count selections, ignoring the probability magnitudes. Expert appears in three selected pairs, in four, and in one. There are assignments altogether:
Now average each probability column across the four tokens, including probabilities for experts that a token did not select:
Both vectors sum to one, but they tell different stories. receives the most assignments, appearing in every token’s top two. has the highest mean probability, despite being selected only three times. Selection depends on each token’s ranking, not simply on which expert has the largest batch-average score.
For token 1, the selected-output gates would be for and for . Neither nor enters the calculations above: token 1 contributes one count to each selected expert’s , and contributes to the three probability averages.
To see how the same probabilities behave under top-1 selection, choose only the largest entry in each row. The selections become , giving
The values do not change, because the full-softmax probabilities did not change. Only the hard selection rule changed.
What to check in a real batch
When inspecting a router, first specify which tokens and decisions are being counted. For the convention in this lesson, is the number of included tokens, and every included token supplies exactly distinct selections. If padding positions are excluded, exclude them from both the count and the probability average. Measure the selected assignments before any capacity limit drops contributions; post-drop workload is a different statistic, addressed later in this module.
Then apply two checks: the expert counts should total , and both and should equal one up to numerical rounding. If the assignment fractions sum to , you have likely divided selection counts by instead of . If the mean probability vector reflects zeros for every unselected expert, you have likely averaged selected-only gates rather than full-softmax probabilities.
These measurements describe routing, not whether an expert has learned a useful specialization. The next lesson uses the hard assignment fractions and mean probabilities to construct an auxiliary load-balancing loss; later lessons will distinguish balanced usage from useful expert behavior.