In the previous lesson, you traced a flattened token batch through a sparse expert layer and saw why expert_idx[t, s] and gate[t, s] must stay paired. This lesson isolates the component that produces those tensors. You will implement a router that accepts token representations of shape and returns expert indices and renormalized gate weights, each of shape . Dispatch and output combination come next.
Specify the router’s contract
For each of token representations, a linear projection produces one logit for each of experts. The router selects the highest-scoring experts independently for each token. If expert_idx[t, s] names the expert in selected slot s, then gate[t, s] is that expert’s contribution weight. The gates should be non-negative and sum to one across each token’s slots.
Watch the forward-pass portion of NeuralNine’s Mixture of Experts: Explained & Implemented. It shows the flattening, projection, selection, and weighting that form this contract; stop before following the later expert-processing code.
Mixture of Experts: Explained & Implemented
Watch how a flattened token batch becomes router logits, then selected indices and weights.
Start at the routing pass. Track the last dimension at each operation: it holds D token features before the projection, N expert scores afterward, and k selected entries after topk. The video computes a full softmax before selecting and renormalizing; below, you will implement the equivalent selected-logit softmax.
The image depicts one token’s final gates when two of eight experts were selected. Its two bars sum to one; it does not imply that the six unselected experts were evaluated and then given zero-weight outputs.

Select logits, then normalize the selected pair
Let logits have shape . Applying topk along its last dimension gives two tensors: selected logit values and their expert indices. With sorted=True, slot 0 contains the highest score, slot 1 the next highest, and so on.
The PyTorch reference is short, but its return order and tie behavior are worth checking before writing the module.
torch.topk — PyTorch 2.8 documentation
Read the PyTorch torch.topk reference to confirm what its two returned tensors contain and what ordering you may rely on.
Under the main “torch.topk” heading, find the function signature and its description. Read the return contract, ending with the Note about tied elements. Focus on dim, the sorted option, and why a test with equal logits should not require one particular tied expert to win.
Why normalize after selection? A softmax over all logits assigns some probability to experts that will not run. Taking its selected probabilities without further normalization generally leaves their sum below one. For the renormalized-gate convention used here, the selected logits instead receive a softmax over the slots:
where is token ’s logit for expert , and is its selected expert index. In exact arithmetic, this equals taking a full softmax, gathering its selected probabilities, and dividing by their sum. PyTorch’s softmax handles the numerical stabilization of the selected-logit calculation.
Implement the module
The router below accepts an already flattened token batch. That keeps its interface separate from a transformer block’s choice of batch and sequence dimensions.
import torch
from torch import nn
from torch.nn import functional as F
class TopKRouter(nn.Module):
def __init__(self, d: int, n_experts: int, k: int):
super().__init__()
if d < 1 or n_experts < 1 or not 1 <= k <= n_experts:
raise ValueError("Require d >= 1 and 1 <= k <= n_experts")
self.k = k
self.proj = nn.Linear(d, n_experts, bias=False)
def forward(self, tokens: torch.Tensor):
if tokens.ndim != 2 or tokens.shape[1] != self.proj.in_features:
raise ValueError("tokens must have shape [T, d]")
logits = self.proj(tokens) # [T, N]
chosen_logits, expert_idx = torch.topk(
logits, self.k, dim=-1, sorted=True
) # both [T, k]
gate = F.softmax(chosen_logits, dim=-1) # [T, k]
return expert_idx, gate
Notice the return order. torch.topk returns (values, indices), whereas this module deliberately returns (expert_idx, gate)—the pair that the later dispatch code needs. expert_idx contains integer expert identifiers; gate contains floating-point weights that remain connected to the selected router logits for autograd. Neither tensor is moved to the CPU or converted to a Python list.
For a transformer input x of shape , the call site supplies the flattening:
B, C, D = x.shape
tokens = x.reshape(B * C, D) # [T, D], where T = B * C
expert_idx, gate = router(tokens) # both [T, k]
This module does not call experts or create their input batches. Its indices identify which experts should receive each row; its gates specify how to weight their eventual outputs.
Verify values, shapes, and the gate convention
A small deterministic check is more informative than inspecting random outputs. Set the projection to the identity so that each token’s three features become its three expert logits:
router = TopKRouter(d=3, n_experts=3, k=2)
with torch.no_grad():
router.proj.weight.copy_(torch.eye(3))
tokens = torch.tensor([
[ 2.0, 1.0, -1.0],
[-1.0, 0.0, 3.0],
[ 0.0, 3.0, 1.0],
])
expert_idx, gate = router(tokens)
expected_idx = torch.tensor([
[0, 1],
[2, 1],
[1, 2],
])
expected_gate = torch.tensor([
[0.7310586, 0.2689414],
[0.9525741, 0.0474259],
[0.8807971, 0.1192029],
])
assert expert_idx.shape == gate.shape == (3, 2)
assert expert_idx.dtype == torch.long
assert torch.equal(expert_idx, expected_idx)
torch.testing.assert_close(gate, expected_gate)
torch.testing.assert_close(gate.sum(dim=-1), torch.ones(3))
For the second token, expert 2 has logit 3 and expert 1 has logit 0, so their selected gates are approximately 0.953 and 0.047. Expert 0’s logit of −1 does not enter that selected softmax. This is why applying softmax to chosen_logits, rather than returning probabilities gathered from a full softmax unchanged, matters.
You can also check the equivalence to the previous lesson’s implementation without changing the module:
full_probs = F.softmax(router.proj(tokens), dim=-1)
selected_probs = full_probs.gather(dim=-1, index=expert_idx)
renormalized = selected_probs / selected_probs.sum(
dim=-1, keepdim=True
)
torch.testing.assert_close(gate, renormalized)
The first check establishes expected indices and numerical weights; this second check establishes that the two gate formulations agree. For k=1, the renormalized gate is always one. For k=N, the selected gates are simply the full softmax probabilities, reordered by descending logit.
One boundary matters for training: hard topk indices are not differentiable. The selected values still feed the softmax, so with , a loss depending on the gates can differentiate through those values to the router projection. A renormalized top-1 gate, being identically one, supplies no task-loss gradient through its gate value. This implementation makes neither claim that the discrete choice itself is differentiable nor that gates alone solve expert load balancing.
Conclusion
A top- router has a compact contract: project tokens to logits, select values and indices along the expert axis, and softmax the selected values to return paired indices and gates. Explicit tests catch axis mistakes, swapped return values, and missing renormalization before expert code makes them harder to locate.
The next lesson will use these paired outputs to gather each expert’s assigned tokens while preserving both the original token row and its selected-expert slot.