Welcome back. The previous lesson tested whether a frequently selected expert actually contributes to held-out predictions. That test matters here: a specialist can be selected often and be useful. Routing collapse is a different claim—that the router increasingly concentrates work on too few experts, leaving others without meaningful opportunities to train.
In this lesson, you will diagnose that claim from three traces taken together: assignments before capacity is applied, router probabilities, and overflow. You will then use the token-group and ablation measurements from earlier lessons to distinguish a failing router from useful, uneven specialization.
What collapse looks like over time
An expert that wins one batch has not collapsed the layer. The concern is a persistent feedback loop: a preferred expert receives more assignments, gets more task-loss updates, and becomes an even more attractive choice, while rarely selected experts fall further behind. Diagnose a trajectory across checkpoints, not a single utilization chart.
Watch two short passages from Jia-Bin Huang’s Mixture of Experts (MoE), Visually Explained:
Mixture of Experts (MoE), Visually Explained
Jia-Bin Huang's video gives a compact visual account of the training feedback loop and of why soft router probabilities cannot substitute for hard assignment counts.
Watch the feedback loop, from the first uneven assignments through experts becoming inactive. Then watch the counterexample: apparently even probability mass can coexist with experts receiving no tokens. The later balancing methods are not needed for this diagnosis.
The counterexample is important. A probability assigned to an expert is not a training example processed by that expert. Top- selection turns scores into discrete assignments, and a small, consistent ranking advantage can win nearly every token even when the softmax distributions look fairly flat.
For a second account of the training-time warning signs, read the following focused portions of MoE Load Balancing: Token Distribution & Expert Collapse:
MoE Load Balancing: Token Distribution & Expert Collapse
This article describes collapse as a persistent training trend and defines probability-based measurements you can compare with actual dispatch.
In "Expert Collapse," read the subsections "How Collapse Happens" and "Detecting Collapse." Begin with the progression and warnings. Then, under "Load Metrics," go to "Router Probability Metrics" and read from the opening distinction between assignments and probabilities through the entropy discussion. Treat entropy as one diagnostic, not a standalone verdict; we will distinguish two kinds of entropy below.
One caution about terminology: collapse usually means that a small subset repeatedly wins across the data, while other experts become persistently idle or nearly idle. Load imbalance is broader. A working specialist can be busy because its token group is common. Such imbalance may still create an operational problem, but it does not by itself show that the model has lost useful specialization.
Measure what the router intended, not only what capacity admitted
At one MoE layer, suppose valid tokens each select of experts. Let be token 's selected set and its full-softmax probability for expert . Count selections before capacity filtering:
The fractions sum to one if each token selects distinct experts. For top-1, the denominator is simply . Also log , the assignments that survived capacity filtering, and . The sum of processed counts can be less than , so do not quietly treat processed counts as if they describe all router choices.
For probability traces, the mean probability per expert is
Together, and reveal whether concentration appears in both the hard decisions and the underlying scores. Neither implies the other must be equally concentrated: top- selection depends on rank, not just probability magnitude.
“Router entropy” can refer to two different quantities. Using natural logarithms, and optionally dividing by to put each on a scale from zero to one, they are
The first asks how uncertain the router is for an individual token. The second asks how widely its average probability mass is spread across experts. Compute both from the full-softmax distribution, even if your implementation renormalizes only the selected experts’ gates for combination. Renormalized top-1 gates would always be one and would carry no useful entropy information.
Consider four equally frequent token groups, each consistently sent to a different expert. The router might give the relevant expert probability and each other expert . Its per-token entropy is low: it makes confident choices. Yet its average probabilities and assignment counts are uniform, so its population entropy is maximal. Low per-token entropy is not, by itself, collapse. Conversely, if every token has probabilities close to , top-1 always selects expert 1 despite high per-token and population entropy. Hard counts expose that failure.
Here is a small diagnostic you can attach to the existing router instrumentation. chosen contains pre-capacity top- indices, and kept records which of those assignments survived the capacity policy.
import math
import torch
@torch.no_grad()
def routing_snapshot(probs, chosen, kept):
# probs: [T, E], full-softmax probabilities
# chosen, kept: [T, k], with kept a boolean mask
T, E = probs.shape
k = chosen.shape[1]
assert T > 0 and E > 1
assert kept.shape == chosen.shape
selected = torch.bincount(chosen.reshape(-1), minlength=E)
processed = torch.bincount(chosen[kept], minlength=E)
p = probs.float().clamp_min(1e-12)
mean_p = p.mean(dim=0)
token_entropy = -(p * p.log()).sum(dim=1).mean() / math.log(E)
population_entropy = -(mean_p * mean_p.log()).sum() / math.log(E)
return {
"selected": selected,
"processed": processed,
"dropped": selected - processed,
"selection_fraction": selected.float() / (T * k),
"mean_probability": mean_p,
"token_entropy": token_entropy.item(),
"population_entropy": population_entropy.item(),
"dropped_assignment_fraction": (~kept).sum().item() / (T * k),
"tokens_with_no_expert": (~kept.any(dim=1)).sum().item(),
}
Exclude padding and ignored positions consistently. Record a snapshot per MoE layer at several training checkpoints on the same held-out token stream, with router noise disabled. Keep the expert count, top-, and capacity policy fixed when comparing snapshots. If batches differ in size, aggregate counts before calculating overall assignment fractions; an unweighted average of batch fractions gives small batches disproportionate influence.
Your existing group labels add the decisive view. Apply the same snapshot to tokens in each group, alongside the all-token snapshot. This shows whether an expert’s rising share comes from the group it serves or from taking over every group.
Read overflow as a consequence—and a possible disguise
Capacity filtering can make a bad router look less bad if you inspect only processed counts. The Switch Transformers paper explains the fixed-capacity mechanism and what happens to overflowed assignments:
arXiv:2101.03961v3 [cs.LG] 16 Jun 2022
Fedus, Zoph, and Shazer's paper connects uneven router choices to fixed expert capacity, skipped expert computation, and the residual path.
In Section 2.1, "Simplifying Sparse Routing," inspect Figure 3 and its caption for the contrast between evenly dispatched tokens and overflow at a fixed capacity. Then, on the printed page marked 6, use Section 2.2, "Efficient Sparse Routing," under "Distributed Switch Implementation." From the paragraph beginning "A capacity factor greater than 1.0," read the capacity trade-off. Focus on the difference between a token missing this expert computation and a token disappearing from the transformer entirely.

Use an explicit capacity convention when interpreting your own trace. For top-1 with , , capacity factor , and integer capacity , each expert can process at most tokens. Suppose the pre-capacity counts are
With a drop-overflow policy, the processed counts are
and of the assignments——overflow. Expert 1’s processed count has been clipped. Reporting only “120 processed” conceals its attempted assignments and makes the original routing skew harder to see. For top-, repeat the calculation using your implementation’s capacity convention and distinguish dropped assignments from tokens that lost every selected expert.
In the residual architecture used earlier in the course, an overflowed contribution is skipped while the token continues through the residual path. This can prevent an immediate catastrophic loss spike; it does not make the missing expert computation harmless. Look for group-conditioned validation loss changes as well as total validation loss.
Overflow is also not proof of collapse. The same pre-capacity counts could arise because token groups occur in exactly those proportions and each has a reliable, useful specialist. It is still a serious capacity problem: that specialist cannot serve many of its assigned tokens. The appropriate diagnosis is useful but overloaded specialization if group-conditioned routing and held-out contribution support it, not automatically “expert collapse.”
Put the traces together
Compare successive checkpoints rather than applying a universal entropy or imbalance cutoff. The following patterns separate questions that a single metric conflates:
| Observed pattern on fixed held-out data | Strongest interpretation | Further check |
|---|---|---|
| Counts spread across experts; confident choices differ by token group; little overflow | Plausible useful specialization | Confirm group-conditioned ablation effects |
| One expert’s pre-capacity share rises across checkpoints for most groups; other experts approach zero; overflow rises | Evidence of developing collapse | Check whether probability concentration also rises and whether idle experts stop receiving task updates |
| Hard counts concentrate, but soft probabilities remain diffuse | Ranking or tie-like selection bias can still cause hard-routing collapse | Inspect score margins and consistently preferred indices |
| A specialist’s group becomes common; its count and overflow rise, while other groups retain their own experts | Data-driven overload, not necessarily collapse | Compare group frequencies, group loss, and ablation results |
| Processed counts look capped or fairly even, but pre-capacity counts are severely skewed | Capacity is hiding router demand | Examine drops by expert and token group |
A useful reporting convention is to place, for each checkpoint and layer, the pre-capacity count vector, processed count vector, dropped-assignment fraction, both entropies, and validation loss on the same timeline. Add group-conditioned counts when a trend appears. A falling population entropy strengthens the case for global concentration; a falling per-token entropy only says choices are becoming more confident. Neither replaces the hard-count and overflow traces.
Finally, connect this diagnosis to the previous lesson’s intervention. If a heavily used expert serves a stable token group and its ablation selectively worsens held-out predictions for that group, there is evidence of useful contribution—even if capacity needs attention. If it increasingly wins unrelated groups while others become idle, “specialization” is a weak explanation for the concentration. An ablation result is evidence about the current checkpoint, not a license to ignore persistent overflow.
Conclusion
Diagnose routing collapse from persistent, cross-group concentration of pre-capacity assignments, interpreted alongside full-softmax probabilities and overflow. Track both per-token and population entropy: confident specialists can have low per-token entropy without collapsing, while near-tied soft probabilities can conceal a dominant hard top-1 choice. Capacity-capped processed counts alone are especially misleading because they hide attempted assignments.
Next, the module turns to router numerical instability. You will use logit and gradient traces to separate that failure from the load and capacity problems diagnosed here.