The previous lesson used assignment counts, probabilities, and overflow to diagnose where tokens go. This lesson examines a different failure: whether the numbers used to make those decisions remain reliable during training. A router can have reasonably distributed assignments and still develop unstable logits; conversely, severe load imbalance does not prove a numerical problem.
You will read logit and gradient traces together, distinguish drift from ordinary confident routing, and choose an intervention that addresses the observed failure rather than changing load balance indiscriminately.
What can go wrong before a token reaches an expert?
For a token with router logits , the full-softmax probability of expert is
Softmax is invariant to adding the same constant to every logit. The vectors and give identical probabilities and identical top- choices. Multiplying logits by a positive constant is different: it preserves their ranking but generally makes the probabilities sharper.
That distinction gives you two phenomena to look for. A growing common offset increases absolute logit values without improving routing decisions. A growing gap between logits makes decisions more confident and can leave softmax gradients very small. Both matter, but they call for more careful interpretation than “large logits are bad.”
Watch the stability portion of Jia-Bin Huang’s Mixture of Experts (MoE), Visually Explained. You have already encountered its routing and load-balancing portions; focus here on the shift-invariance argument and the difference between stable softmax computation and controlling the logits produced by the model.
Mixture of Experts (MoE), Visually Explained
In Mixture of Experts (MoE), Visually Explained, Jia-Bin Huang explains why router logits can drift even when their routing probabilities do not, then introduces safe softmax and router z-loss.
Watch router stability. At about 00:27:23, follow the constant-shift example; at about 00:28:40, note what subtracting the maximum fixes; and from about 00:30:22, follow why the squared log-normalizer is used. Treat the claim that z-loss gradients leave probabilities unchanged as an intuition about controlling a shared offset, not a general guarantee about an optimizer step.
A standard, max-subtracted softmax prevents an avoidable overflow in exponentiation. It does not restore a logit difference already rounded away in low precision, nor does it make an extremely peaked softmax provide strong task gradients. On the other hand, finite large logits do not automatically cause exploding gradients: a saturated softmax often has vanishing derivatives. An inf or NaN gradient is a more serious observation, but its source must still be localized.
Read logit traces alongside gradient traces
The ST-MoE paper provides a useful engineering account: its authors compare stability interventions and explain why router calculations can be sensitive to precision, even when softmax uses max subtraction.
ST-MoE: Designing Stable and Transferable Sparse Expert Models
Read the stability and precision passages in ST-MoE: Designing Stable and Transferable Sparse Expert Models. They connect the router z-loss to observed training stability and give a concrete example of a routing-relevant difference lost to low-precision rounding.
In Section 3.3, “Stability and Quality Tradeoffs when Constraining Activations and Gradients,” begin where the authors turn from update clipping to router logits. Read the intervention comparison through their discussion of the update-clipping and z-loss results. Then, in Section 3.4, “Selecting a Precision Format: Trading Efficiency and Stability,” read from the discussion of lower-precision benefits through the subsection “Understanding precision format and roundoff errors.” In the precision passage, pay particular attention to the example with ten logits at 128 and one at 128.5. Finally, find the aside titled “An aside on the router z-loss.” Read the clipping comparison, which ends just before Section 4.
In the paper’s example, rounding to removes a meaningful difference before softmax. Subtracting the maximum afterward cannot reconstruct it. This is why a targeted precision intervention—computing router logits and softmax in float32—can help when the trace points to rounding or non-finite router calculations. Casting already-rounded logits to float32 is too late to recover information they lost.
For the tiny MoE you have been training, collect traces per MoE layer on a consistent held-out token stream, with router noise disabled. Record the router’s raw logits before softmax, top-, or capacity filtering. Useful summaries are:
| Trace | What it helps distinguish |
|---|---|
| Mean and upper-tail log-sum-exp of logits | Growth in the softmax log-normalizer targeted by z-loss |
| Upper-tail maximum absolute logit | Large positive or negative values, including outliers hidden by averages |
| Upper-tail largest-minus-smallest logit and top-two margin | Increasing spread or increasingly decisive rankings |
| Router-logit and router-parameter gradient norms; finite-value checks | Vanishing, unusually large, or invalid gradients |
| Validation loss, pre-capacity assignments, and overflow | Whether a numerical trend coincides with an actual training or routing failure |
“Upper-tail” can mean the 99th percentile, supplemented by a maximum. Log the precision used for router computation as well. When comparing training gradients, record them after any loss-scaling unscale operation and before clipping; otherwise a change in scaling or clipping can masquerade as a change in the underlying gradient. Keep the loss-term coefficients alongside the trace.
One caveat from the earlier routing lesson is essential. With renormalized top-1 gates, the selected gate weight is always one. Away from a top-1 selection boundary, the task loss then supplies no router gradient through that gate. A small task-loss router gradient in this configuration is expected—not evidence, on its own, of softmax saturation. Auxiliary losses may still supply router gradients.
What z-loss controls—and what it does not
For valid tokens, define each token’s log-normalizer and the router z-loss as
The training objective can include alongside task and load-balancing losses. Compute from all raw expert logits, not only selected gates or probabilities; a stable logsumexp implementation avoids explicitly forming potentially enormous exponentials.
Why square the log-normalizer? Adding a constant to every logit for one token adds to , even though it does not change routing probabilities. The penalty therefore provides pressure against an unconstrained common shift. Its logit gradient is
This also explains a limitation of the video’s shift-only intuition: gradient components depend on , so an actual parameter update is not guaranteed to preserve probability differences or top- choices.
The following toy simulation illustrates the intervention’s intended effect. Its training objective was deliberately constructed to encourage logit growth; these curves are not results from the tiny language-model experiment, and its coefficients are not recommended settings for that experiment.

The next plot shows the z-loss gradient alone, not the full training gradient. As one expert’s probability dominates in its toy setup, that expert receives most of the penalty’s logit-gradient magnitude.

Z-loss is not a bound on every absolute logit or every gap. For example, logits have a log-sum-exp very close to zero, hence almost no z-loss, but an extreme softmax spread. This is why you should inspect both and the logit range. Likewise, an isolated large negative logit can raise your maximum-absolute-logit statistic without implying that the log-normalizer has drifted upward.
Select the intervention from the failure signature
Start with the earliest abnormal trace, rather than treating a later loss spike as the entire diagnosis.
-
Log-normalizers and absolute logits climb across checkpoints; instability follows. This is evidence for logit drift. Check that the router uses a stable softmax, then compare a controlled run with a modest z-loss coefficient against the current run. The ST-MoE authors selected for their setting; it is a starting reference, not a universal threshold. Keep load-balancing strength and other training settings fixed, and check whether the logit trend improves without harming held-out loss or useful group-conditioned routing.
-
Router outputs change abruptly with precision, or become non-finite in low-precision computation. Prioritize computing the router projection and probability calculation in float32. Inspect where non-finite values first appear; z-loss cannot repair information that was already rounded away before it is computed. Avoid treating post-hoc clipping of rounded logits as equivalent to producing smaller logits.
-
Gradients spike while logit scale stays bounded. Z-loss is not the first explanation. Check the learning rate, loss scaling, the first tensor with a non-finite gradient, and whether the spike originates outside the router. Appropriate responses may include correcting mixed-precision handling, lowering the learning rate, or clipping genuinely excessive updates. A total gradient norm alone does not locate the problem.
-
Logit gaps grow and task router gradients shrink, but log-normalizers remain controlled. Investigate saturation and the gating convention before increasing z-loss. If gates are renormalized top-1, near-zero task router gradients are structural. Otherwise, inspect margins and whether routing can still change on held-out data. Z-loss may help with some scale trajectories, but the example shows why it cannot be assumed to solve every large-gap problem.
-
Assignments skew or overflow rises, while logit and gradient traces remain healthy. Return to the previous lesson’s load and capacity diagnosis. Load-balancing pressure or a capacity-policy change addresses a different mechanism from numerical stabilization.
A convincing comparison changes one stabilization setting at a time, begins from the same pre-failure checkpoint when possible, and uses the same held-out stream. Put the logit summaries, finite-gradient checks, assignment counts, overflow, and validation loss on one timeline. “No loss spike” is insufficient if the intervention has merely flattened useful routing or degraded predictions.
Conclusion
Router instability has several signatures, not one: common-offset drift, excessive logit gaps, low-precision rounding, unusually large updates, and non-finite gradients can look similar when viewed only through training loss. Stable softmax protects an operation; float32 router computation protects precision; z-loss discourages growth of the log-normalizer. None substitutes for diagnosing load imbalance from pre-capacity assignments and overflow.
This completes the module’s progression from balancing and capacity to training-failure diagnosis. Next, you will examine what changes when experts live on different devices: token dispatch and return become communication operations, and uneven expert loads can become per-rank stragglers.