Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 6

Gradient Flow Through Hard Top-k and Renormalized Top-1 Routing

Recommended resources

A sparse expert layer changes more than which feed-forward networks run: it changes which parameters can learn from each token. The key is to separate two operations that are easy to conflate. Selecting expert indices is discrete; weighting the selected outputs may still be differentiable. This lesson traces task-loss gradients through both operations and shows why renormalized top-1 routing has a special limitation.


Start with the forward computation

In a transformer block, a router scores the token representation for each of experts. Write its logits as . A hard top- operation chooses a set of expert indices; only those experts evaluate their feed-forward networks. Their outputs are combined as

where is the gate weight. The residual connection and subsequent layers receive as usual. For this lesson, assume every selected contribution is executed and retained; capacity-related dropping comes later in the course.

The short lecture excerpts below establish the routing computation and the precise difficulty posed by discrete selection.

Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 4: Mixture of experts

Watch “Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 4: Mixture of experts” from Stanford Online for the mathematical setup and the training problem.

Watch router formulation for learned expert scores, top-k selection, and gated aggregation. Then watch discrete gating for the difficulty of differentiating a sparse choice. Focus on the distinction between an expert's index and its combination weight; the derivation below makes their different gradient paths explicit.

Gate weights depend on a convention established during forward routing. In the chart, experts 1 and 3 are selected, but their displayed full-softmax probabilities add to only . Those numbers could be used directly as gates, or divided by so the selected gates add to one.

Full-softmax probabilities for the two selected experts, 1 and 3, among eight candidates. Their sum is 0.48, so these are not yet weights renormalized over the selected pair.

The following reading describes that normalization and the continuous-weight gradient path. We will refine one potentially ambiguous phrase in it: “gradients only flow through selected experts” is correct for expert-network parameters, but does not always mean that only selected router rows receive a task-loss gradient.

Gating Networks: Router Architecture in Mixture of Experts - Interactive

Read the sections on hard top-k routing and end-to-end router gradients in “Gating Networks: Router Architecture in Mixture of Experts” by Martin Brendle, to connect the forward gate definition to backpropagation.

First, in “Hard Routing with Top-K Selection,” read from the four-step recipe, the gating recipe. Stop after the explanation of why selected probabilities are renormalized; focus on the denominator. Then go to “Router Training” and its “End-to-End Gradients” subsection. Begin in the opening paragraph where discrete choices are described as discontinuous, and read the gradient explanation. Stop after the sentence about adjusting the router for similar future inputs. Keep separate what is said about selected expert outputs and what may happen to unselected router logits.



Hold the hard selection fixed during backpropagation

Ordinary backpropagation through hard top- treats the chosen index set as fixed for this forward pass. A small logit change that leaves the ranking unchanged does not change which expert runs. At a ranking boundary, the selection can jump; the ordinary derivative of that choice is not available. This is not, by itself, a straight-through estimator or another surrogate for the selection decision.

Let be the gradient arriving from the task loss, and let denote expert 's parameters. For a selected expert, the chain rule gives

An unselected expert did not compute an output for this token, so it has no task-loss gradient from this token. A different token in the batch may select it, however; its batch gradient is the sum of contributions from tokens that used it.

For the router, changing a continuous gate can change the weighted mixture without changing the selected indices. Define

Here measures how the task loss responds locally to increasing expert 's coefficient. The router gradient depends critically on how the coefficients were defined.



Full-softmax gates versus renormalized gates

Suppose the router first computes a softmax over all experts,

and selects the top probabilities. The two common gate conventions have different backward behavior.

Full-softmax gates: Set for , without rescaling. Then . Because every logit appears in the softmax denominator, changing an unselected logit changes the selected probabilities even though it does not run that unselected expert. Holding fixed,

For , the first term vanishes, but the second generally does not. Thus an unselected router logit can receive a task-loss gradient through the shared denominator. That gradient provides no evaluation of how good the unselected expert's output would have been.

Renormalized selected gates: Instead set

The second expression shows an exact cancellation: once the selected probabilities are renormalized, their weights depend only on selected logits. The gradient is

In this convention, unselected router rows receive no task-loss gradient from this token either. Selected router gradients can still happen to be zero—for example, if changing their relative weights would not change the loss locally—but they have a differentiable path.

A logit gradient reaches router parameters by the familiar linear-layer chain rule. Since ,

Therefore, when discussing whether a router “receives gradients,” specify both the gate convention and which row of its weight matrix you mean.



A numerical gradient trace—and the top-1 exception

Consider three logits . Their full-softmax probabilities are , so top-2 selects experts 0 and 1. For a scalar illustration, let their outputs be and , and let the arriving task-loss gradient be .

With full-softmax gates, . Applying the formula gives router-logit gradients

Expert 2 never ran, yet its router logit has a gradient. Its expert parameters do not.

With renormalized gates, the selected weights are , and . Now the router-logit gradients are

The unselected logit has disappeared from the computation of the output.

Now reduce the selection to top-1, still renormalizing over selected experts. If expert wins, its gate is

While remains the winner, that gate is constant. The task loss supplies zero gradient to every router logit through this MoE output, including the winning logit. The selected expert still receives its ordinary task-loss gradient. The router can alter future choices when its parameters change for some other reason, but ordinary backpropagation through this renormalized top-1 output gives it no task-loss update.

This result is specific to the gate convention. With a full-softmax top-1 gate, , so can change and router logits can receive gradients—even though only expert runs. Likewise, an explicitly implemented surrogate for the discrete selection would be a different backward rule; hard top- does not imply such a rule automatically.



Keep the accounting precise

For a given token, the useful distinctions are:

Forward gate Task-loss gradient to expert parameters Task-loss gradient to router logits
Full-softmax top- Selected experts only Potentially selected and unselected logits
Renormalized top-, Selected experts only Selected logits only
Renormalized top-1 Winning expert only None through the MoE output

These are direct paths from this token's task loss, not promises that every listed gradient is numerically nonzero. Shared transformer parameters can also receive gradients through the executed expert and residual computation. Conversely, an optimizer update, weight decay, or an auxiliary router loss should not be mistaken for a task-loss gradient through hard selection. Load-balancing losses, introduced later, are one way to provide an additional router signal when task-loss routing signals are inadequate.

The central takeaway is that sparse execution always withholds this token's task-loss gradient from unselected expert networks, but it does not by itself tell you which router parameters receive gradients. Inspect the gate normalization. In particular, renormalized top-1 makes the winning gate exactly one, leaving the router without a task-loss gradient through that output. The next lesson moves from this mathematical trace to the tensor shapes and indexing operations in a minimal PyTorch router-and-expert scaffold.