Hello. This lesson is a correctness checkpoint for the sparse expert layer you have been building. You will compare it with a deliberately inefficient reference that runs every expert, then masks out unselected contributions. The two implementations should agree not only on outputs but also on gradients—the part that matters when training begins.
Define the comparison precisely
Let be token ’s representation, its router logit for expert , and its top- set. For this lesson, selected gates are renormalized over the selected experts:
The sparse implementation evaluates only experts in . The reference evaluates all , but multiplies unselected outputs by zero. It therefore does more work while computing the same mathematical output. Both sides must use the same gating convention: comparing renormalized top- gates with unrenormalized full-softmax probabilities would test two different models.
For a short visual reminder of token-to-expert indexing and recombination, watch this portion of Sparse Mixture of Experts from Neural Breakdown with AVB:
Sparse Mixture of Experts - The transformer behind the most efficient LLMs (DeepSeek, Mixtral)
Watch the top-k implementation segment to reconnect the test below with the dispatch and combination operations it checks.
In top-k dispatch, follow how the expert index identifies both a token and its selected-expert slot, then how the corresponding gate scales that expert's output. The test will check whether those pairings survive recombination.
Build an independent, compute-heavy reference
The following self-contained layer provides both paths. Its reference constructs a mask over all experts; its sparse path gathers selected logits, dispatches token subsets, and uses indexed addition to restore token order. Using different combination mechanisms makes the comparison more useful than comparing two copies of the same dispatch code.
import copy
import math
import torch
from torch import nn
from torch.autograd import gradcheck
class TinyMoE(nn.Module):
def __init__(self, d=4, hidden=6, n_experts=4, k=2):
super().__init__()
self.k = k
self.router = nn.Linear(d, n_experts, bias=False)
self.experts = nn.ModuleList([
nn.Sequential(
nn.Linear(d, hidden),
nn.GELU(),
nn.Linear(hidden, d),
)
for _ in range(n_experts)
])
def forward(self, x, reference=False):
# x: [tokens, d]; logits: [tokens, experts]
logits = self.router(x)
indices = logits.topk(self.k, dim=1).indices # [tokens, k]
if reference:
probabilities = logits.softmax(dim=1)
mask = torch.zeros_like(probabilities)
mask = mask.scatter(1, indices, 1.0)
weights = probabilities * mask
weights = weights / weights.sum(dim=1, keepdim=True)
# Deliberately evaluate every expert on every token.
all_outputs = torch.stack(
[expert(x) for expert in self.experts], dim=1
) # [tokens, experts, d]
return (weights.unsqueeze(-1) * all_outputs).sum(dim=1)
selected_logits = logits.gather(1, indices)
gates = selected_logits.softmax(dim=1) # [tokens, k]
output = torch.zeros_like(x)
for expert_id, expert in enumerate(self.experts):
token_ids, slots = (indices == expert_id).nonzero(as_tuple=True)
if token_ids.numel() == 0:
continue
expert_output = expert(x.index_select(0, token_ids))
contribution = (
expert_output * gates[token_ids, slots].unsqueeze(-1)
)
output = output.index_add(0, token_ids, contribution)
return output
The reference’s masked full-softmax probabilities equal the sparse path’s softmax over selected logits: their common full-softmax denominator cancels during renormalization. The reference does not save computation; that is why it is a convenient oracle for a small test.
Compare outputs and task-loss gradients
Use separate, identically initialized copies so that each backward pass has its own gradients. The chosen router weights and inputs give distinct top- choices without a near-tie at the selection boundary. A nonuniform output probe makes it less likely that an ordering mistake disappears when outputs are reduced to a scalar.
torch.manual_seed(7)
base = TinyMoE().double()
with torch.no_grad():
base.router.weight.copy_(torch.eye(4, dtype=torch.double))
sparse = copy.deepcopy(base)
reference = copy.deepcopy(base)
x_data = torch.tensor([
[ 3., 2., 0., -1.],
[-1., 3., 2., 0.],
[ 0., -1., 3., 2.],
[ 2., 0., -1., 3.],
[ 3., 0., 2., -1.],
], dtype=torch.double)
probe = torch.tensor([
[1.0, -0.5, 0.7, 0.3],
[0.2, 1.3, -0.4, 0.8],
[0.9, 0.1, 1.1, -0.6],
[-0.3, 0.5, 0.2, 1.4],
[0.6, -0.8, 0.4, 1.2],
], dtype=torch.double)
x_sparse = x_data.clone().requires_grad_()
x_reference = x_data.clone().requires_grad_()
y_sparse = sparse(x_sparse)
y_reference = reference(x_reference, reference=True)
torch.testing.assert_close(y_sparse, y_reference, rtol=1e-10, atol=1e-10)
def test_loss(y):
return (y * probe).sum() + 0.1 * y.square().sum()
test_loss(y_sparse).backward()
test_loss(y_reference).backward()
torch.testing.assert_close(
x_sparse.grad, x_reference.grad, rtol=1e-9, atol=1e-9
)
for (name_s, parameter_s), (name_r, parameter_r) in zip(
sparse.named_parameters(), reference.named_parameters()
):
assert name_s == name_r
# An unused sparse expert may have grad=None; the masked reference
# may instead produce an explicit zero gradient.
grad_s = (torch.zeros_like(parameter_s) if parameter_s.grad is None
else parameter_s.grad)
grad_r = (torch.zeros_like(parameter_r) if parameter_r.grad is None
else parameter_r.grad)
torch.testing.assert_close(grad_s, grad_r, rtol=1e-9, atol=1e-9)
print("Outputs, input gradients, and parameter gradients match.")
This checks three different properties. Output agreement tests the forward pairing of token, expert, slot, and gate. Input-gradient agreement tests paths through both selected experts and their gates. Parameter-gradient agreement tests whether the router and expert weights receive the same task-loss signal.
A zero gradient for an expert that receives no tokens is expected, not a failure. Conversely, matching outputs alone is insufficient: detaching a gate or an expert contribution could preserve the forward value while changing its gradient.
Add a numerical check—and know its limit
The reference comparison can miss a mistake shared by both implementations. PyTorch’s gradcheck offers a different check: it compares autograd derivatives with finite-difference estimates. Read the definition and its precision and differentiability cautions before using it:
torch.autograd.gradcheck.gradcheck — PyTorch 2.13 documentation
Read PyTorch's gradcheck reference to understand what an independent numerical gradient check establishes, and when it can fail for reasons other than a broken implementation.
On the torch.autograd.gradcheck.gradcheck page, in the gradcheck() API entry, begin with the sentence about small finite differences. Read the definition and cautions through the note on non-differentiable points. Focus on its recommendation to use double precision and its warning about points where numerical and analytical derivatives may disagree.
Check the sparse layer’s derivative with respect to its input:
x_check = x_data.clone().requires_grad_()
assert gradcheck(
lambda input_tensor: sparse(input_tensor),
(x_check,),
eps=1e-6,
atol=1e-5,
rtol=1e-3,
)
print("Input gradcheck passed.")
This call checks input gradients, not every parameter gradient. The preceding reference comparison covers parameter gradients, but because both paths use the same PyTorch operations for each expert, an additional spot-check of parameter derivatives is worthwhile:
checks = [
(sparse.router.weight, (0, 0)),
(sparse.experts[0][0].weight, (0, 0)),
]
step = 1e-6
for parameter, position in checks:
analytical = parameter.grad[position].item()
with torch.no_grad():
parameter[position] += step
plus = test_loss(sparse(x_data)).item()
parameter[position] -= 2 * step
minus = test_loss(sparse(x_data)).item()
parameter[position] += step # Restore the original value.
numerical = (plus - minus) / (2 * step)
assert math.isclose(
analytical, numerical, rel_tol=1e-3, abs_tol=1e-5
), (analytical, numerical)
print("Sample router and expert parameter derivatives match.")
Finite differences assume that the local routing decision stays fixed. At a top- tie—or so close to one that a small perturbation changes the selected set—the function has a selection boundary and gradcheck may fail without establishing an autograd bug. The test inputs above avoid that ambiguity. They also use double precision, as the documentation recommends.
Interpret a failure before changing code
If outputs differ, inspect the selected indices and the (token_ids, slots) pairs first. A correct expert output added to the wrong token, or multiplied by another slot’s gate, is enough to fail the comparison.
If outputs agree but gradients differ, inspect for .detach(), conversion to Python numbers, in-place updates that interfere with autograd, and whether the two paths use the same gate normalization. Remember that hard top- indices do not carry task-loss gradients through the act of selection; gradients flow through the selected gates and evaluated expert computations. With the renormalized gates used here, unselected logits do not affect those gate weights while the selected set is fixed.
For contrast, if you choose unrenormalized full-softmax gates in another implementation, unselected router logits can still receive gradients because they affect the full-softmax denominator. Change the reference’s weighting rule as well before comparing that variant. For renormalized top-, the sole selected gate is always one, so the task loss gives no gradient to the router through that gate while the choice remains fixed.
Key takeaways
A dense, masked reference is a strong correctness tool precisely because it computes the same selected-expert sum without relying on sparse dispatch. Compare outputs, input gradients, and every parameter gradient under one explicit gate convention; then use finite differences as a separate local check away from routing ties. With the sparse layer validated, the next step is to place it into a tiny decoder-only transformer and check that the surrounding model still behaves as intended.