Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 31

Detecting Silent Dispatch and Expert Routing Errors

Recommended resources

Hello. This module is about deciding whether a small MoE implementation is trustworthy enough to scale. The first requirement is semantic correctness: tokens must reach the selected experts, and their outputs must return to the right tokens with the right weights. An incorrect permutation can leave output shapes and even training loss looking plausible.

In this lesson, you will use three deliberately awkward cases—permuted token order, a repeated expert index, and an expert that receives no tokens—to expose silent dispatch and combination errors. The tests apply to a single-process layer now and provide a foundation for checking expert-parallel implementations later.


Define what must remain correct

For token , let be its hidden state. Suppose slot selects expert with gate weight . With capacity limits and other fallback behavior disabled, the sparse branch should produce

The slot matters. It ties an expert selection to its particular gate weight, even after dispatch has rearranged assignments into expert-sized batches. The token identity matters too: expert outputs must be accumulated back into the original token row. A correct final tensor shape verifies neither property.

The following short reading describes the same integrity problem in a serving setting. Its warning applies equally to a training implementation: a dispatch or combine optimization must preserve the model’s routing decisions and numerical meaning.

Mixture-of-experts serving is becoming an online expert-placement problem | WTF Is Trending?

Read the semantic-integrity warning and its concrete token-permutation failure case. Use them as motivation for differential tests, rather than as specifications for the implementation below.

In “Semantic integrity,” under “Systems optimizations must preserve routing and numerical meaning,” read the paragraph beginning “A placement-only change” through the integrity warning. Then find item “06 Incorrect token permutation” in the numbered failure list. Read its “What you see” and “What to do” lines, from symptom to remedy. Focus on why checking only that an output exists would miss the error.

Our differential test will compare a grouped implementation against a deliberately simple reference. The two versions should implement the same equation but not share the potentially faulty grouping-and-unpermutation procedure.



Build a small, discriminating test case

Use four toy experts, each with a distinct response:

There are three one-dimensional tokens with values , , and . Each has two assignment slots:

Token value Expert indices Gate weights Expected sparse output

For the middle token, both slots name expert 1. Its two contributions are and ; both count. The weights sum to intentionally, so an accidental renormalization also changes the answer. Expert 3 receives no assignments.

Ordinary top- selection returns distinct indices. The duplicate here is an injected test input, not a claim about ordinary top- output. It tests the contract of a generic assignment-and-combine routine that accepts assignment slots. If your layer explicitly requires distinct indices, it should reject this input clearly before dispatch; it must not silently produce a partial answer.

The following standalone PyTorch code uses a token-by-token reference and an expert-grouped implementation. It runs on CPU so that test results are easy to reproduce.

import torch

def toy_expert(expert_id, values):
    return (expert_id + 1) * values + 10 * (expert_id + 1)

def reference(x, ids, gates):
    """Independent reference: no expert grouping or unpermutation."""
    token_count, slots = ids.shape
    outputs = []
    for token in range(token_count):
        total = torch.zeros_like(x[token])
        for slot in range(slots):
            expert_id = int(ids[token, slot])
            total = total + gates[token, slot] * toy_expert(
                expert_id, x[token]
            )
        outputs.append(total)
    return torch.stack(outputs)

def grouped(x, ids, gates, expert_count):
    """Group assignments by expert, then accumulate into token rows."""
    token_count, slots = ids.shape
    token_for_assignment = torch.arange(
        token_count
    ).repeat_interleave(slots)
    flat_ids = ids.reshape(-1)
    flat_gates = gates.reshape(-1)

    output = torch.zeros_like(x)
    counts = []
    for expert_id in range(expert_count):
        positions = (flat_ids == expert_id).nonzero(
            as_tuple=True
        )[0]
        counts.append(int(positions.numel()))
        if positions.numel() == 0:
            continue

        token_ids = token_for_assignment[positions]
        expert_input = x.index_select(0, token_ids)
        contributions = (
            flat_gates[positions, None]
            * toy_expert(expert_id, expert_input)
        )
        output.index_add_(0, token_ids, contributions)
    return output, counts

x = torch.tensor([[1.0], [2.0], [4.0]], dtype=torch.float64)
ids = torch.tensor([[2, 0], [1, 1], [2, 0]])
gates = torch.tensor(
    [[0.25, 0.75], [0.20, 0.30], [0.60, 0.40]],
    dtype=torch.float64,
)

expected = torch.tensor(
    [[16.5], [12.0], [30.8]], dtype=torch.float64
)
actual, counts = grouped(x, ids, gates, expert_count=4)

torch.testing.assert_close(
    reference(x, ids, gates), expected, rtol=0, atol=1e-12
)
torch.testing.assert_close(
    actual, expected, rtol=0, atol=1e-12
)
assert counts == [2, 2, 2, 0]

permutation = torch.tensor([2, 0, 1])
permuted, permuted_counts = grouped(
    x[permutation],
    ids[permutation],
    gates[permutation],
    expert_count=4,
)
torch.testing.assert_close(
    permuted, actual[permutation], rtol=0, atol=1e-12
)
torch.testing.assert_close(
    permuted,
    reference(x[permutation], ids[permutation], gates[permutation]),
    rtol=0,
    atol=1e-12,
)
assert permuted_counts == counts

Notice that index_add_ accumulates contributions when a destination token appears more than once. An overwrite operation would lose at least one contribution here. The expert counts also check dispatch independently of the final output: the two assignments to expert 1 belong to the same token, while expert 3 must receive zero.



Understand what each case can—and cannot—prove

Token permutation. Rearrange the token rows and rearrange their expert indices and gates in exactly the same way. With fixed routing, token-independent experts, and dropless deterministic dispatch, the output rows must undergo that same permutation. In compact notation, for a row permutation ,

This catches code that relies accidentally on the original row order. It does not, by itself, prove correctness: a consistently wrong implementation might satisfy the permutation property. That is why the code also compares both orders against an independent reference and known numerical outputs. Keep order-dependent capacity dropping and stochastic behavior out of this particular test; they change its assumptions.

Repeated expert index. The middle token makes two assignments to the same destination expert but with different slot weights. A combine routine that stores one result per (token, expert) pair, rather than accumulating one contribution per (token, slot) assignment, may discard one. If duplicates are invalid under your layer’s contract, replace the numerical expectation with a test for a clear validation error. Still test repeated destination token rows in the combine routine using valid, distinct top- selections.

Zero-token expert. Expert 3 must not be given another expert’s tokens through a bad offset or padding calculation. Skipping its computation is fine; contributing a fabricated output is not. The count check makes this case visible even if an erroneous contribution happens to be small. In a training version, an unused expert receives no task-loss gradient from this batch, though optimizer behavior such as weight decay is a separate consideration.

The particular token values, expert responses, and unequal gates are chosen to make identity mistakes observable. If every expert returned the same value, or every gate weight were equal, several permutations and slot swaps would become invisible.



Transfer the tests to a real sparse layer

Start by freezing the router result and testing the sparse branch before adding the transformer residual. Supply token representations, expert indices, and gates directly. For a tiny expert bank, construct a slow reference by evaluating the selected expert for each (token, slot) pair—or by evaluating all experts and masking unselected contributions—without using the production dispatch code. Compare outputs with a tolerance appropriate to the dtype and kernels.

When a comparison fails, inspect four boundaries rather than only the final loss:

  1. Assignments: Record (token ID, slot ID, expert ID, gate). A slot ID prevents two selections for the same token from being conflated.
  2. Expert inputs: Confirm that each assignment carries the hidden state belonging to its recorded token ID. Compare records by ID, not by their position inside an expert batch.
  3. Expert results: Check that the returned vector was computed by the recorded expert for that assignment.
  4. Combined output: Check that every result is multiplied by its recorded gate and accumulated into its recorded token row.

These records can be sorted by (token ID, slot ID) for comparison even if grouping—or, later, communication between ranks—changes their execution order. If assignments match but expert inputs do not, investigate dispatch. If inputs and expert results match but the sparse output differs, investigate weight association and combination. Only after that branch passes should you compare the full transformer block; its residual path can make an incorrect sparse output less obvious.

For the permutation test, disable dropout and order-dependent capacity behavior. For the zero-token test, inspect counts as well as values. For the duplicate-index test, document whether the public API accumulates repeated assignments or rejects them. An undocumented silent third behavior is the failure these tests are designed to uncover.



Key takeaways

A sparse layer’s contract is about assignment identity, not merely output shape: each selected (token, slot) must reach the intended expert, retain its gate, and contribute to the intended token. A permutation test checks row-order integrity; a repeated-index test checks accumulation or explicit rejection; a zero-token-expert test checks empty-group handling. An independent numerical reference makes these tests stronger than plausibility checks or training loss alone.

The next lesson uses per-rank token counts and timings to diagnose expert-parallel stragglers. These correctness checks come first: timing an incorrectly dispatched workload would give a precise answer to the wrong problem.