Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 9

Token-to-Expert Dispatch with Position and Slot Preservation

Recommended resources

Hello. A top- router tells you which experts each token selected, but an expert cannot process that answer directly: it needs a batch of token vectors. This lesson turns router selections into expert-grouped batches while retaining enough information to identify every token’s original position and every selection’s top- slot. We will keep dispatch dropless for now; capacity limits come later in the course.


An expert assignment has two identities

Suppose the input to an MoE feed-forward layer has been flattened from shape to , where . Flattening does not erase a token’s place in the sequence: token came from batch item , sequence position .

Let the router return:

  • expert_idx of shape : the selected expert ID in each top- slot.
  • gate_weight of shape : the corresponding combination weight.

For a top-2 router, expert_idx[5, 0] is token 5’s first selected expert; expert_idx[5, 1] is its second. The slot is part of the assignment’s identity. If you save only token ID 5, you can no longer tell which router weight belongs to which expert result.

An assignment can therefore be identified by the pair : original token position and selected-expert slot . It also has an expert ID expert_idx[t, s]. There are assignments, though only distinct input token vectors. A top-2 token appears twice in the dispatched data, once for each selected expert.

Here, top- slot means a position in a token’s ranked selections. It does not mean a fixed-capacity storage slot inside an expert. The distinction matters when we introduce capacity-limited dispatch later.



Group by expert without losing the way back

The natural input order is token-major: all selections for token 0, then all selections for token 1, and so on. Expert execution wants a different order: all assignments for expert 0 together, then expert 1, and so on. Dispatch is chiefly this change of order, accompanied by a record of where each row came from.

Watch how the following demonstrations motivate expert-first processing and distinguish copying a token into an expert batch from scaling it by a gate weight.

Code Mixture of Experts (MoE) from Scratch in Python

In “Code Mixture of Experts (MoE) from Scratch in Python,” Vizuara contrasts reasoning token by token with gathering the tokens assigned to each expert. Watch for the change in viewpoint, rather than adopting its full layer implementation.

In the section where the sparse MoE class is discussed, watch expert-first grouping. The relevant passage starts when the implementation is described as looping over experts and ends after the explanation of accumulating contributions from tokens assigned to multiple experts.

MoE Token Routing Explained: How Mixture of Experts Works (with Code)

“MoE Token Routing Explained: How Mixture of Experts Works (with Code),” from Hugging Face, visualizes how token identities and expert destinations are carried into dispatched batches. Its example also uses fixed expert slots; our implementation below omits those capacity constraints.

In the segment that builds the token, expert, and slot mapping, watch mapping components. Then skip to batch construction, where the presentation distinguishes a binary selection matrix that copies token values from weights that would scale them. For this lesson, focus on copying the values and retaining weights separately.

We will implement grouping by sorting assignments on their expert IDs. A stable sort preserves token-major order among assignments tied to the same expert, making the result easier to inspect. Stability is convenient, not essential to correctness: the saved metadata, rather than a presumed ordering, is what lets us recover each assignment.

Consider three tokens and three experts, with top-2 selections:

Original token Slot 0 Slot 1
0 expert 2 expert 0
1 expert 1 expert 2
2 expert 2 expert 1

After grouping, expert 0 receives token 0’s slot-1 assignment. Expert 1 receives token 1’s slot-0 and token 2’s slot-1 assignments. Expert 2 receives the remaining three. An expert-grouped tensor alone would not reveal those slot numbers; we must reorder token_ids, slot_ids, and gate weights alongside the token vectors.



Implement dropless dispatch in PyTorch

The function below takes the output of a top- router; it does not recompute routing. It returns a single expert-grouped tensor and offsets marking where each expert’s rows begin and end. offsets[e]:offsets[e + 1] is the slice for expert e, including when that expert receives no assignments.

from typing import NamedTuple
import torch


class Dispatch(NamedTuple):
    values: torch.Tensor       # [T * k, d], grouped by expert
    token_ids: torch.Tensor    # [T * k], original flattened positions
    slot_ids: torch.Tensor     # [T * k], positions within top-k
    expert_ids: torch.Tensor   # [T * k], destination experts
    weights: torch.Tensor      # [T * k], aligned router weights
    offsets: torch.Tensor      # [num_experts + 1]


def dispatch(
    x: torch.Tensor,             # [T, d]
    expert_idx: torch.Tensor,    # [T, k], integer expert IDs
    gate_weight: torch.Tensor,   # [T, k]
    num_experts: int,
) -> Dispatch:
    if x.ndim != 2 or expert_idx.ndim != 2:
        raise ValueError("Expected x [T, d] and expert_idx [T, k]")
    if gate_weight.shape != expert_idx.shape:
        raise ValueError("Weights must match expert_idx [T, k]")

    T, _ = x.shape
    if expert_idx.shape[0] != T:
        raise ValueError("Router output must have one row per token")

    k = expert_idx.shape[1]
    if not 1 <= k <= num_experts:
        raise ValueError("Expected 1 <= k <= num_experts")
    if expert_idx.dtype != torch.long:
        raise TypeError("expert_idx must have dtype torch.long")
    if expert_idx.device != x.device or gate_weight.device != x.device:
        raise ValueError("Inputs must be on the same device")
    if torch.any(expert_idx < 0) or torch.any(expert_idx >= num_experts):
        raise ValueError("Expert ID out of range")

    # Flatten in token-major order. Position t * k + s denotes (t, s).
    flat_expert = expert_idx.reshape(-1)
    flat_token = torch.arange(T, device=x.device).repeat_interleave(k)
    flat_slot = torch.arange(k, device=x.device).repeat(T)
    flat_weight = gate_weight.reshape(-1)

    # Apply the SAME permutation to every assignment-aligned tensor.
    order = torch.argsort(flat_expert, stable=True)
    expert_ids = flat_expert[order]
    token_ids = flat_token[order]
    slot_ids = flat_slot[order]
    weights = flat_weight[order]
    values = x.index_select(0, token_ids)

    counts = torch.bincount(expert_ids, minlength=num_experts)
    offsets = torch.cat((counts.new_zeros(1), counts.cumsum(0)))

    return Dispatch(values, token_ids, slot_ids, expert_ids, weights, offsets)

The pivotal line is values = x.index_select(0, token_ids). It copies the appropriate input vector for each assignment, so a token selected by two experts appears in two expert batches. We do not multiply values by weights here. The weights stay aligned with these rows for weighted combination after the experts run.

The same principle applies to every field: sort just expert_ids and the vectors will go to the wrong experts; sort vectors and IDs but not slot_ids or weights and the outputs will later receive the wrong gate weights. Keeping these arrays assignment-aligned is the central correctness requirement.

This implementation uses a sort for clarity, not as a claim about the fastest production dispatcher. It allocates exactly assignment rows, with no fixed per-expert capacity, padding, or dropping.



Trace the result and verify its invariants

Here is a concrete input. The first feature makes each token easy to recognize.

x = torch.tensor([
    [10.0, 0.0],
    [20.0, 0.0],
    [30.0, 0.0],
])

expert_idx = torch.tensor([
    [2, 0],
    [1, 2],
    [2, 1],
], dtype=torch.long)

gate_weight = torch.tensor([
    [0.7, 0.3],
    [0.6, 0.4],
    [0.8, 0.2],
])

d = dispatch(x, expert_idx, gate_weight, num_experts=3)

print(d.offsets.tolist())    # [0, 1, 3, 6]
print(d.expert_ids.tolist()) # [0, 1, 1, 2, 2, 2]
print(d.token_ids.tolist())  # [0, 1, 2, 0, 1, 2]
print(d.slot_ids.tolist())   # [1, 0, 1, 0, 1, 0]
print(d.values[:, 0].tolist())  # [10., 20., 30., 10., 20., 30.]

Read the first row across those arrays: expert 0 receives a vector from original token 0, specifically its top- slot 1, with weight gate_weight[0, 1] (0.3). The offsets say expert 1’s batch is d.values[1:3], while expert 2’s is d.values[3:6]. No expert evaluates the whole batch of three tokens.

The following checks express the dispatch contract directly:

assert torch.equal(d.values, x[d.token_ids])
assert torch.equal(
    d.expert_ids, expert_idx[d.token_ids, d.slot_ids]
)
assert torch.allclose(
    d.weights, gate_weight[d.token_ids, d.slot_ids]
)

# Every original (token, top-k slot) assignment appears exactly once.
pair_ids = d.token_ids * expert_idx.shape[1] + d.slot_ids
assert torch.equal(
    torch.sort(pair_ids).values,
    torch.arange(x.shape[0] * expert_idx.shape[1])
)

# An unused expert still has a valid, empty slice.
d4 = dispatch(x, expert_idx, gate_weight, num_experts=4)
assert d4.offsets.tolist() == [0, 1, 3, 6, 6]

These checks are more informative than inspecting output shape alone. In particular, a dispatcher can return the expected shape while silently swapping two tokens or attaching the wrong selected-expert slot. Notice also that an expert with zero tokens needs no fabricated input row: its two offsets are equal.



The boundary of this component

Dispatch has produced the inputs each expert needs, but it has not produced an MoE layer output. Once expert e processes its slice, each resulting row must be associated with the saved token_ids, slot_ids, and weights. The next lesson will use that metadata to accumulate weighted expert results back at the original token positions.

The key takeaway is an invariant: every dispatched row represents one original assignment, and its vector, expert ID, and gate weight must remain aligned. Grouping rows makes expert computation possible; preserving their identities makes correct combination possible.