Hello. Last lesson established how to calculate per-expert capacity and count overflowing assignments. This lesson makes the missing decision explicit: when an expert fills up, which assignments are admitted, and what happens to the rest? You will implement a small PyTorch dispatcher that drops only the overflowing expert contributions. In a pre-norm transformer block, the residual path will remain intact even if a token loses every expert contribution. Allow about 40 minutes for the short video, targeted readings, and code walkthrough.
Decide what “drop” means before writing code
For top- routing, each token has selected expert contributions. Our policy is:
- Consider every token’s first-choice assignment in token order, then every token’s second-choice assignment, and so on. This is rank-major admission order.
- Give an assignment the next free slot at its selected expert. Keep it only if that slot number is less than the expert’s capacity.
- Do not send an overflowing assignment to another expert. Omit its contribution, and do not renormalize the remaining gate weights.
This policy favors first choices, though a first choice can still overflow at a particularly popular expert. A token that loses one of two assignments can still receive its other expert’s output. A token that loses both receives no output from this expert sublayer; it has not disappeared from the transformer.
Watch the slot-allocation portion of Hugging Face’s MoE Token Routing Explained: How Mixture of Experts Works (with Code). Focus on the cumulative count for each expert and the mask that removes assignments whose slot indices reach capacity.
MoE Token Routing Explained: How Mixture of Experts Works (with Code)
Hugging Face visualizes how selected token–expert pairs acquire slots and how an overflow mask removes the pairs that do not fit.
Watch slot selection, from the introduction of expert slots through the update to the routing coefficients. The video’s example uses two slots per expert. Pay particular attention to the zero-based slot indices: slots 0 and 1 fit, while slot 2 does not.
The Switch Transformer paper states why an overflowing top-1 assignment need not erase the token representation. Read just its capacity-policy explanation; its top-1 formula is not the top- capacity convention established in the previous lesson.
[PDF] Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Read the capacity-policy passage by Fedus, Zoph, and Shazeer to connect skipped expert computation with the transformer’s residual connection.
In Section 2.2, “Efficient Sparse Routing,” under “Distributed Switch Implementation” (printed p. 5), start where the authors discuss statically determined tensor shapes. Read the capacity passage through the sentence about wasted computation and memory. Focus on what is skipped for an overflowing token and what continues to the next layer.
If the input to a pre-norm expert sublayer is , write its update as
Here and are the selected expert and its pre-capacity gate weight; is 1 if that assignment gets a slot and 0 otherwise. Capacity changes , not . When every assignment for token overflows, , so . In particular, “drop the token” is shorthand for skip its overflowing expert computation, not delete its hidden state or its earlier attention result.
Keeping the original gate weights matters. Suppose a top-2 token has gates and , but the first expert is full. Under our policy, its MoE output is times the second expert’s output—not that expert’s output with weight 1. Renormalizing after the drop would be a different policy with different outputs and gradients.
Turn assignment order into slots
The following small implementation uses the previous lesson’s convention for a flattened routing group of tokens, experts, and selections per token:
Before implementing it, inspect one PyTorch example of capacity-limited routing. Read for the data movement, not for a formula to copy wholesale: this example uses a different capacity-rounding rule and constructs padded expert batches, whereas ours will send only admitted assignments to each expert.
nanoMoE: Mixture-of-Experts (MoE) LLMs from Scratch in ...
Read the “PyTorch implementation” subsection by Cameron Wolfe to see how a top-k selection is converted into expert positions, a capacity mask, and expert inputs.
In “PyTorch implementation,” first read the component guide. Then, in the Router code, locate the comment “compute the expert capacity” and continue through the returned expert batches; the passage from mask to batches covers the central indexing operations. Notice that its cumulative count processes first-choice assignments before second-choice assignments. Its capacity calculation floors and adjusts to an even integer; retain our ceiling convention instead.
Here is a compact dispatcher suitable for the tiny single-process model. It accepts the top- indices and gate weights produced by the router you implemented earlier. The loop over experts makes slot ownership visible; it is not intended as a high-throughput distributed implementation.
import math
import torch
def capacity_limited_moe(
x, expert_idx, gate, experts, capacity_factor
):
"""
x: [T, D] expert inputs, already normalized if appropriate
expert_idx: [T, k] selected expert IDs, highest rank first
gate: [T, k] corresponding pre-capacity gate weights
experts: sequence of E modules, each mapping [N, D] to [N, D]
Returns:
delta: [T, D] combined expert contributions
keep: [T, k] admitted-assignment mask
slot: [T, k] expert slot, or -1 if not admitted
"""
T, D = x.shape
E = len(experts)
assert T > 0 and E > 0 and capacity_factor > 0
assert expert_idx.shape[0] == T
k = expert_idx.shape[1]
assert 1 <= k <= E and gate.shape == expert_idx.shape
capacity = math.ceil(capacity_factor * T * k / E)
# Rank-major order: all first choices, then all second choices.
flat_expert = expert_idx.transpose(0, 1).reshape(-1)
flat_gate = gate.transpose(0, 1).reshape(-1)
flat_token = torch.arange(T, device=x.device).repeat(k)
keep_flat = torch.zeros(T * k, dtype=torch.bool, device=x.device)
slot_flat = torch.full(
(T * k,), -1, dtype=torch.long, device=x.device
)
for e in range(E):
requests = torch.nonzero(
flat_expert == e, as_tuple=True
)[0]
admitted = requests[:capacity]
keep_flat[admitted] = True
slot_flat[admitted] = torch.arange(
admitted.numel(), device=x.device
)
delta = torch.zeros_like(x)
for e, expert in enumerate(experts):
assignments = torch.nonzero(
keep_flat & (flat_expert == e), as_tuple=True
)[0]
if assignments.numel() == 0:
continue
tokens = flat_token[assignments]
expert_input = x.index_select(0, tokens)
expert_output = expert(expert_input)
weighted = expert_output * flat_gate[assignments, None]
delta.index_add_(0, tokens, weighted)
keep = keep_flat.reshape(k, T).transpose(0, 1)
slot = slot_flat.reshape(k, T).transpose(0, 1)
return delta, keep, slot
The first loop is dispatch admission. For each expert, requests appears in rank-major order, so its first capacity entries get distinct slots. The second loop actually gathers the admitted inputs, evaluates the appropriate expert, and accumulates weighted outputs back at their original token indices. index_add_ is essential when a token retains more than one contribution: a plain indexed assignment could overwrite one expert’s result with another’s.
The code never evaluates an expert on an overflowing assignment and never multiplies that assignment’s gate into delta. It also does not silently fill an unused slot at another expert: that would require an explicit rerouting policy. Its batches have variable lengths up to ; systems that require fixed shapes can pad each expert batch to after admission without changing which assignments are kept.
Check the policy on a deliberately skewed batch
Take four tokens, three experts, top-2 routing, and . The convention above gives
Use these selections and gates, listed in preference order:
| Token | First choice | Second choice |
|---|---|---|
| 0 | expert 0, weight | expert 1, weight |
| 1 | expert 0, weight | expert 2, weight |
| 2 | expert 0, weight | expert 1, weight |
| 3 | expert 2, weight | expert 0, weight |
Expert 0 receives four requests but has only two slots. In rank-major order it admits tokens 0 and 1; token 2’s first choice and token 3’s second choice overflow. Expert 1 admits tokens 0 and 2, while expert 2 admits tokens 3 and 1. Both tokens 2 and 3 therefore still obtain one expert contribution.
A short check can verify not just the overflow count, but the exact output. Let experts 0, 1, and 2 multiply their inputs by 1, 2, and 3, respectively, and let every input be 1:
from torch import nn
class Scale(nn.Module):
def __init__(self, factor):
super().__init__()
self.factor = factor
def forward(self, x):
return self.factor * x
experts = [Scale(1.0), Scale(2.0), Scale(3.0)]
x = torch.ones(4, 1)
expert_idx = torch.tensor([
[0, 1],
[0, 2],
[0, 1],
[2, 0],
])
gate = torch.tensor([
[0.70, 0.30],
[0.60, 0.40],
[0.80, 0.20],
[0.55, 0.45],
])
delta, keep, slot = capacity_limited_moe(
x, expert_idx, gate, experts, capacity_factor=0.75
)
assert torch.equal(keep, torch.tensor([
[True, True],
[True, True],
[False, True],
[True, False],
]))
torch.testing.assert_close(
delta,
torch.tensor([[1.30], [1.80], [0.40], [1.65]])
)
assert torch.all(slot[~keep] == -1)
Token 2 is the revealing case: its heavily weighted first choice overflows, leaving only . If the implementation produced , it had renormalized the survivor to weight 1 and changed the stated policy.
The returned keep mask is also a useful training diagnostic. Count ~keep to report dropped assignments; count rows for which keep.any(dim=1) is false to report tokens with no expert contribution. Those are different measurements for top-.
Preserve the residual in the transformer block
The dispatcher deliberately returns only delta. The transformer block—not the dispatcher—adds its residual. For a pre-norm block whose attention update has already produced h, the integration looks like this:
expert_input = expert_norm(h) # [B, S, D]
flat = expert_input.reshape(-1, h.size(-1)) # [T, D]
expert_idx, gate = router(flat) # each [T, k]
delta, keep, slot = capacity_limited_moe(
flat, expert_idx, gate, experts, capacity_factor
)
h = h + delta.reshape_as(h) # residual is unchanged
Neither keep nor slot should be used to mask h. If all of a token’s assignments overflow, its row of delta remains zero and the addition leaves that token’s incoming hidden state intact.
There is a corresponding gradient consequence. An overflowing assignment does not run its expert and supplies no task-loss gradient through that expert contribution. A retained contribution remains differentiable through its expert output and gate weight. This does not mean a shared router logit can never receive a gradient when an assignment drops: softmax gates for retained assignments may depend on other logits, and auxiliary router losses provide separate gradients. The admission decision itself—expert indices and slot selection—is discrete.
Takeaways
Capacity-limited dispatch needs both a slot count and an admission order. In this implementation, rank-major admission gives each expert its first requests; later requests lose only their overflowing contributions. Original gate weights stay in place, admitted expert outputs are accumulated at their source-token positions, and the transformer adds the resulting MoE delta to an unmasked residual. The keep and slot tensors make the policy inspectable rather than implicit.
Next, you will put this capacity-limited layer beside dropless routing on the same skewed batch and compare dropped assignments with peak expert batch size.