Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 34

How Microbatch Size Affects Load-Balancing Measurements

Recommended resources

Welcome back. A load-balancing number is meaningful only after you specify which tokens were counted together. In this lesson, you will keep the router and token stream fixed while changing only the microbatch boundaries. That isolates a subtle effect: the same routing decisions can produce different averaged microbatch losses, even though the full-stream expert counts have not changed.

This matters both for interpreting training traces and for deciding whether a balancing loss is encouraging useful expert specialization or forcing every small batch to use every expert.


What a balancing measurement counts

Consider one MoE layer with experts and top- routing. For a set containing tokens, define two quantities for expert :

where is token 's full-softmax router probability for expert , before top- masking. Both vectors sum to one under these conventions. The hard-assignment fraction describes where work was directed; the mean probability describes the router's scores, including for experts that were not selected.

A commonly used auxiliary balance loss is

Uniform hard assignments and uniform mean probabilities give . The loss is a training surrogate, however, not a complete report of utilization: you should still inspect counts and peak expert loads.

Global-batch load balance almost free lunch to improve your MoE LLM training | Qwen

Read Qwen's explanation of why the token population used for a balance loss matters, followed by its proposed global-batch calculation.

In “Load Balancing loss,” read the setup, ending with the example of a code-only microbatch. Then, in “From micro-batch balance to global-batch balance,” read from the paragraph beginning “This situation is even more common” through the three-step calculation that synchronizes expert selection frequency and aggregates the loss. Focus on what is made global: the assignment frequency, rather than the expert computations themselves.

Here, balance batch size means the token population over which those statistics are formed. It need not equal a dispatch microbatch, a GPU's local batch, or the number of tokens in an optimizer update. Those sets can differ even when people loosely call all of them “the batch.”



One stream, two partitions, different losses

Take eight tokens, two experts, and top-1 routing. Suppose the first four tokens belong to group A and each has full-softmax probabilities ; the last four belong to group B and each has probabilities . Every A token selects expert 1, and every B token selects expert 2.

First partition the stream into two homogeneous microbatches of four:

Token set Hard fractions Mean probabilities Balance loss
Four A tokens
Four B tokens
All eight together

Nothing about the router changed. Across all eight tokens, the experts each receive four assignments. Yet averaging the two microbatch losses reports , whereas computing the loss once on the eight-token set reports .

Now retain the same microbatch size but put two A and two B tokens in each microbatch. Each microbatch has , so each loss is . Thus size alone does not determine the result: which tokens share a microbatch matters. Small batches drawn from a mixed stream may be noisy; small batches organized by domain can systematically produce the homogeneous case.

The algebra explains the discrepancy. If microbatch contains tokens, let . With the same top- convention throughout, the full-stream statistics are weighted averages:

But multiplying within each microbatch before averaging differs from multiplying those averages:

The bracketed term is the weighted covariance across microbatches of that expert's assignment fraction and mean probability. In the example, expert 1 has both high and high in A batches, then both low in B batches; expert 2 behaves oppositely. That within-batch alignment raises the averaged microbatch loss. This is not a rule that smaller batches must always yield a larger loss: the sign and size depend on the joint routing statistics and partition.

The distinction also changes training, not just reporting. In the usual implementation, hard fractions are treated as non-differentiable statistics while the probabilities carry the balance-loss gradient. A homogeneous A microbatch therefore applies a local balancing signal even though the complete stream uses both experts equally. A global-frequency objective can permit group-conditional routing so long as aggregate usage remains acceptable.



Reproducing the measurement—and naming its scope

MoE balance batch size vs expert parallel size — Fanout

Read Fanout's distinction between the statistical population of a balance loss and the hardware layout used to execute experts.

In “MoE balance batch size vs expert parallel size,” read the distinction, then continue through the definitions of expert assignment fractions and mean probabilities. In “Global counts are cheaper than global token routes,” read the implementation discussion. Notice its separate cautions about the reduction group and the cadence at which counts are updated.

You can reproduce the fixed-stream comparison without training a model. This small PyTorch snippet measures intended top-1 assignments before any capacity drops, so changes in its output come solely from regrouping tokens.

import torch

prob = torch.tensor([[0.9, 0.1]] * 4 + [[0.1, 0.9]] * 4)
chosen = prob.argmax(dim=-1)
num_experts = prob.shape[1]

def stats(rows):
    rows = torch.as_tensor(rows, dtype=torch.long)
    counts = torch.bincount(chosen[rows], minlength=num_experts)
    f = counts.float() / len(rows)       # top-1: one assignment per token
    p = prob[rows].mean(dim=0)           # full-softmax probabilities
    loss = num_experts * (f * p).sum()
    return f, p, loss.item()

homogeneous = [[0, 1, 2, 3], [4, 5, 6, 7]]
mixed = [[0, 1, 4, 5], [2, 3, 6, 7]]

for label, partition in [("homogeneous", homogeneous), ("mixed", mixed)]:
    micro_mean = sum(stats(rows)[2] for rows in partition) / len(partition)
    print(label, "mean microbatch loss:", round(micro_mean, 3))

print("full-stream loss:", round(stats(range(8))[2], 3))

The printed losses are , , and , respectively. In your own traces, do not average microbatch losses equally if their token counts differ and then compare that value to a token-weighted global loss. Record the weighting convention.

For a real run, label each measurement with tokens counted, accumulation steps, ranks included, top-, and whether counts are before or after capacity drops. Expert parallelism determines which ranks own and execute experts; increasing its size does not automatically enlarge the statistical population of the balance loss. Conversely, synchronizing assignment counts does not move expert ownership.

A global-frequency implementation can synchronize an -element vector of hard counts and pair that shared frequency with local differentiable probability means, using the correct token weights. Whether counts are collected over the current full update, refreshed after each microbatch, or taken from a delayed estimate changes the objective. Do not give all three an unqualified “global” label.



What improved balance can conceal

The image below gives an empirical reason to examine both scales. Its heatmaps compare expert-selection patterns conditioned on three types of material. The microbatch-balanced panels look more diffuse within each type; the global-batch-balanced panels show stronger conditional preferences. The accompanying bars compare perplexity and benchmark scores for the configurations shown. These are results from an ablation, not proof that a preferred expert alone causes better predictions or that every training setup will benefit from global balancing.

Left: perplexity and benchmark-score comparisons for the displayed balancing configurations. Right: expert token-ratio heatmaps by layer and material type, with micro-batch balance in each pair's left panel and global-batch balance in its right panel.

Global-batch load balance almost free lunch to improve your MoE LLM training | Qwen

Return to Qwen's article for its reported specialization result and the qualification about execution speed.

In “Results: More Performant and Interpretable MoE,” read the specialization comparison and the following paragraph on changing balance batch size. Then read the paragraph beginning “Using global batch balance may result” immediately before “Conclusion,” through the reported update-step timings. Keep validation quality and step speed separate as you interpret the results.

A globally balanced update can still contain an overloaded individual microbatch. That batch may have a high peak expert workload, overflow if capacity is enforced, or a slowest rank that holds up an all-to-all phase. Conversely, a low microbatch balance loss does not establish that routing is useful for predictions. When comparing partition sizes, retain both views:

  • Full-stream utilization: total intended assignments per expert, mean router probabilities, and, where relevant, actual assignments after drops.
  • Local execution pressure: per-microbatch peak expert count, overflow, per-rank received assignments, and step time.

With fixed token probabilities and fixed intended routes, full-stream and must remain unchanged when you merely repartition the stream. If they change in that controlled comparison, inspect your normalization, reduction group, or accounting of dropped assignments. Once you train under a different partition or balance-loss scope, router parameters can change, so later differences in full-stream counts may be genuine effects of the altered objective.



Takeaway

Microbatch-averaged balance loss and full-stream balance loss multiply statistics at different scopes; they are generally unequal even for identical tokens and routing decisions. The gap reflects how assignment fractions and router probabilities vary together across microbatches, while the runtime consequences depend on the actual local expert loads. Name the scope and weighting of every metric before treating a change in its value as improved—or degraded—routing.

Next, you will use that measurement discipline in a controlled routing-hyperparameter ablation, tracking validation loss, utilization, dropped assignments, and step time together.