Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 14

Auxiliary Load-Balancing Loss with Router Probabilities and Assignment Statistics

Recommended resources

Welcome back. Last lesson separated two views of routing: , the fraction of actual top- assignments received by expert , and , its mean probability under the full router softmax. You will now use both quantities to implement a loss that gives the router a training signal when assignments concentrate on a few experts.


From routing measurements to a training signal

Without an incentive to spread work, early routing preferences can reinforce themselves. Frequently selected experts receive more task-loss updates, while rarely selected experts receive fewer. An auxiliary balancing loss counters that tendency; it does not replace the language-modeling objective or guarantee that every expert will learn a useful specialty.

Watch the implementation segment of Mixture of Experts: Explained & Implemented by NeuralNine. It computes mean router probabilities and selected-expert fractions inside a sparse layer. Pay attention to which quantity comes from softmax and which comes from discrete selections.

Mixture of Experts: Explained & Implemented

NeuralNine builds the balancing term alongside a top-k router, making the connection to your existing sparse-layer implementation concrete.

Watch the loss implementation. In particular, notice the one-hot selected indices averaged across tokens and selected slots. We will express the same count with torch.bincount below.

For the convention established last lesson, let a batch contain included tokens, experts, and exactly distinct selections per token. Using full-softmax probabilities and selected-expert sets , define

The unweighted auxiliary loss is

During training, a coefficient sets its strength relative to the task loss:

If both and are uniform, the unweighted balancing loss is , regardless of . Keeping outside the helper function makes that baseline and the training coefficient easier to inspect separately.

The supplied diagram shows the original top-1 form: its counts argmax selections. For top-, retain its product-and-sum structure but divide assignment counts by , as above.

The top-1 balancing loss multiplies each expert’s hard assignment fraction by its mean full-softmax probability; only the probability term provides a gradient through the router.

Read the formulation in Switch Transformers by Fedus, Zoph, and Shazeer. It gives the original top-1 definition and explains why the probability average provides a differentiable route for the balancing signal. Our top- normalization extends the assignment-count convention from the previous lesson.

arXiv:2101.03961v3 [cs.LG] 16 Jun 2022

Read the original formulation to connect the count-and-probability calculation to a training objective.

On printed p. 7, in “A Differentiable Load Balancing Loss,” start at “For each Switch layer, this auxiliary loss is added to the total model loss during training.” Read the formulation, following the definitions of the two vectors and the expert-count scaling. Continue through the coefficient discussion at the end of the section, including the coefficient range. Treat uniform routing as the intended operating point, not as a guarantee that every nonuniform batch has a loss above 1.



Implement it without breaking the gradient path

Suppose your router already returns probabilities of shape [T, E] and top- indices of shape [T, k]. The indices can be counted, but they cannot supply a useful derivative through the hard selection. The probabilities must remain attached to the computation graph.

import torch

def load_balance_loss(
    router_probs: torch.Tensor,   # [T, E], softmax over all experts
    topk_indices: torch.Tensor,   # [T, k], distinct selected experts
) -> torch.Tensor:
    if router_probs.ndim != 2 or topk_indices.ndim != 2:
        raise ValueError("Expected probabilities [T, E] and indices [T, k]")

    T, E = router_probs.shape
    if T == 0 or topk_indices.shape[0] != T:
        raise ValueError("Token dimensions must match and be nonempty")

    k = topk_indices.shape[1]
    if not 1 <= k <= E:
        raise ValueError("Require 1 <= k <= number of experts")

    # Hard assignment statistics: T*k selected token-expert pairs.
    counts = torch.bincount(
        topk_indices.detach().reshape(-1), minlength=E
    )
    P = router_probs.float().mean(dim=0)       # [E], differentiable
    f = counts.to(dtype=P.dtype) / (T * k)      # [E], constant in backward

    return E * torch.dot(f, P)                  # scalar

This function assumes valid, distinct expert indices produced by topk. Count intended selections before capacity drops; dropped contributions are a separate workload measure. If your batch contains padding positions, filter those positions from both inputs before calling it.

Two details matter more than the choice between bincount and one-hot encoding:

  • P uses probabilities for all experts, including unselected ones—not the gates renormalized over selected experts.
  • Only the count path is detached. Detaching router_probs, converting the loss to a Python number with .item(), or constructing from detached probabilities would remove the auxiliary router gradient.

Using the four-token top-2 batch from last lesson, and . The implementation should return

Here is a small executable check, including backpropagation to the router logits:

given_probs = torch.tensor([
    [0.60, 0.30, 0.10],
    [0.20, 0.50, 0.30],
    [0.40, 0.35, 0.25],
    [0.45, 0.40, 0.15],
])

logits = given_probs.log().requires_grad_()
router_probs = logits.softmax(dim=-1)
topk_indices = logits.topk(k=2, dim=-1).indices

aux = load_balance_loss(router_probs, topk_indices)
assert torch.allclose(aux.detach(), torch.tensor(1.1203125), atol=1e-6)

aux.backward()
assert logits.grad is not None
assert logits.grad.abs().sum() > 0


What the gradient does—and does not do

Within one backward pass, the selected indices and values are fixed. The differentiable part is , the mean of the softmax probabilities. Thus, heavily assigned experts receive a larger weight in the sum, and the auxiliary gradient generally favors reducing their probabilities relative to less-assigned experts.

For a precise view, let be token ’s router logit for expert . Holding the hard counts fixed gives

The gradient reaches router parameters through the softmax. This auxiliary term alone does not update expert network weights: it does not evaluate expert outputs. The task loss still trains whichever experts were selected. In particular, if your selected-only top-1 output gate renormalizes to , its task-loss path may provide no router-logit gradient through gate magnitude; this auxiliary path can still provide one.

Do not treat the scalar as a complete utilization diagnostic. If for every expert, then for any , and this term’s router gradient is zero for that batch. Conversely, is not an unconditional lower bound: for two experts, three tokens with probabilities and one with give , , and a loss of . Always inspect assignment counts alongside the loss.

To integrate the helper into your existing layer, compute it from the same full-softmax probabilities and indices used for dispatch, then return it alongside the layer output. At the training step, add alpha * aux to the task loss before calling backward(). Log detached copies of the loss and ; keep the training copy attached. If you later sum auxiliary losses from several MoE layers, specify whether multiplies their sum or their mean, since that changes the effective strength.



Takeaways

You now have a top- balancing loss whose hard assignment fractions sum to one, whose probability averages remain differentiable, and whose coefficient is applied explicitly at the training step. Its purpose is to influence routing, not to certify balanced work or useful specialization on its own.

The next lesson adds router z-loss. Unlike the balancing term, it targets router logit scale rather than the distribution of expert assignments.