Welcome back. In the previous lesson, you compared capacity-limited and dropless dispatch on fixed routes. Now the routes will change as the model learns. This lesson sets up and runs a short, reproducible next-token training experiment with a tiny MoE decoder and a synthetic dataset whose sequences belong to identifiable groups. The goal is to establish a trustworthy training run—not yet to claim that any expert has specialized. Allow about 45 minutes, including the run.
Decide what this experiment can show
A next-token model receives tokens up to the current position and predicts the following token. Here, every sequence begins with a group marker. Subsequent tokens follow a transition rule that depends on that marker, with a little randomness added. The model therefore has a reason to use context: the same content token can have a different likely successor in different groups.
We will use four experts and top- routing in one transformer block. Dispatch is dropless, so every selected assignment executes. Gate weights are taken from a softmax over all experts and are not renormalized after top- selection. Training uses float32, no router noise, and no auxiliary losses. Holding these choices fixed gives you a baseline whose behavior you can examine in later lessons.
Before running it, read a short account of why a falling language-model loss is not the only concern in MoE training.
nanoMoE: Mixture-of-Experts (MoE) LLMs from Scratch in ...
Cameron R. Wolfe’s nanoMoE article distinguishes routing collapse from numerical instability. Read this to frame what a successful first run does—and does not—establish.
In “Best Practices for Training MoEs,” find the paragraph beginning “There are two main issues that occur when training an MoE.” Read the failure modes, stopping at the sentence about restarting from a checkpoint. Focus on the distinction between repeatedly selecting a few experts and a loss that becomes numerically unstable; later lessons will measure and address them separately.
Supply grouped next-token data
The complete experiment below runs in a PyTorch notebook or script. Run the code blocks in order, in the same session. It uses CPU by default so that the small run does not require a GPU.
The vocabulary has 16 IDs: 0–3 are group markers, and 4–15 are content tokens. A sequence starts with its marker and a random content token. Most later transitions advance the content state by a group-dependent amount; a minority advance it by one more. For example, group 0 usually advances by one state, while group 2 usually advances by three. The content state wraps within its 12-token range.
import random
import torch
from torch import nn
from torch.nn import functional as F
SEED = 2027
VOCAB = 16
CONTEXT = 16
random.seed(SEED)
torch.manual_seed(SEED)
torch.use_deterministic_algorithms(True)
torch.set_num_threads(2)
device = torch.device("cpu")
def make_sequences(n, seed):
assert n % 4 == 0
rng = torch.Generator().manual_seed(seed)
# Equal numbers of the four groups, in a shuffled row order.
groups = torch.arange(n) % 4
groups = groups[torch.randperm(n, generator=rng)]
seq = torch.empty((n, CONTEXT + 1), dtype=torch.long)
seq[:, 0] = groups
seq[:, 1] = 4 + torch.randint(12, (n,), generator=rng)
for position in range(2, CONTEXT + 1):
usual = torch.rand(n, generator=rng) < 0.85
step = groups + 1 + (~usual).long()
seq[:, position] = 4 + (seq[:, position - 1] - 4 + step) % 12
# At every position, predict the token immediately to its right.
return seq[:, :-1], seq[:, 1:], groups
train_x, train_y, train_groups = make_sequences(2048, seed=1101)
val_x, val_y, val_groups = make_sequences(512, seed=2202)
assert train_x.shape == (2048, CONTEXT)
assert val_x.shape == (512, CONTEXT)
assert torch.equal(train_x[:, 0], train_groups)
assert torch.equal(val_x[:, 0], val_groups)
assert train_x.min() >= 0 and train_x.max() < VOCAB
assert train_y.min() >= 0 and train_y.max() < VOCAB
print("train:", train_x.shape, "validation:", val_x.shape)
print("group counts:", torch.bincount(train_groups, minlength=4).tolist())
print("first input: ", train_x[0].tolist())
print("first target:", train_y[0].tolist())
The target row is the input row shifted one position left. In particular, the model predicts the random first content token from the marker alone, then predicts later content tokens from the marker and preceding content. Causal attention must prevent it from reading the target token at the position being predicted.
The training and validation sets use different data-generation seeds and fixed, pre-generated examples. Some short patterns may naturally recur in both sets; they are samples from the same rule, not a promise of unique sequences. Keep the validation set out of optimizer updates and any hyperparameter adjustments for this initial run.
Use the tiny MoE decoder
If you retained the tested tiny decoder from Module 2, you can substitute it here, provided it accepts an integer tensor of shape [batch, context] and returns logits of shape [batch, context, vocabulary]. The following self-contained version makes the experiment runnable even when that earlier scaffold is not at hand. Its MoE branch follows the same dispatch-and-indexed-combination pattern you tested previously.
class SparseFFN(nn.Module):
def __init__(self, width=48, hidden=96, experts=4, top_k=2):
super().__init__()
self.top_k = top_k
self.router = nn.Linear(width, experts, bias=False)
self.experts = nn.ModuleList([
nn.Sequential(
nn.Linear(width, hidden),
nn.GELU(),
nn.Linear(hidden, width),
)
for _ in range(experts)
])
def forward(self, x):
batch, length, width = x.shape
flat = x.reshape(batch * length, width)
probabilities = self.router(flat).softmax(dim=-1)
weights, choices = probabilities.topk(self.top_k, dim=-1)
combined = torch.zeros_like(flat)
for expert_id, expert in enumerate(self.experts):
# Each row is (flattened token position, selected-expert slot).
selected = (choices == expert_id).nonzero(as_tuple=False)
if selected.numel() == 0:
continue
token_positions, slots = selected.unbind(dim=-1)
expert_input = flat.index_select(0, token_positions)
contribution = expert(expert_input)
contribution = contribution * weights[
token_positions, slots
].unsqueeze(-1)
combined = combined.index_add(
0, token_positions, contribution
)
return combined.reshape(batch, length, width)
class TinyMoETransformer(nn.Module):
def __init__(self):
super().__init__()
width = 48
self.tokens = nn.Embedding(VOCAB, width)
self.positions = nn.Embedding(CONTEXT, width)
self.attn_norm = nn.LayerNorm(width)
self.attn = nn.MultiheadAttention(
width, num_heads=4, dropout=0.0, batch_first=True
)
self.ffn_norm = nn.LayerNorm(width)
self.moe = SparseFFN(width=width)
self.output_norm = nn.LayerNorm(width)
self.output = nn.Linear(width, VOCAB)
def forward(self, token_ids):
batch, length = token_ids.shape
positions = torch.arange(length, device=token_ids.device)
x = self.tokens(token_ids) + self.positions(positions)
attended_input = self.attn_norm(x)
# True entries are future positions that attention must not read.
future_mask = torch.ones(
length, length, dtype=torch.bool, device=x.device
).triu(1)
attended, _ = self.attn(
attended_input, attended_input, attended_input,
attn_mask=future_mask, need_weights=False
)
x = x + attended
x = x + self.moe(self.ffn_norm(x))
return self.output(self.output_norm(x))
model = TinyMoETransformer().to(device)
print("total parameters:", sum(p.numel() for p in model.parameters()))
Notice where the earlier capacity comparison applies. selected can have a different number of rows for each expert, but this implementation sets no fixed capacity and drops no assignment. It is deliberately straightforward rather than a fast expert-parallel kernel.
The router’s probabilities depend on the contextual representation after attention. Thus the marker is available in the causal context of later tokens, but a group marker does not force a particular expert to handle its group. Whether routing develops a useful group pattern is an empirical question for a subsequent lesson.
Train, evaluate, and preserve the run
The measured quantity is mean next-token cross-entropy per target token. Each optimizer step samples a batch of rows from the fixed training set. Evaluation traverses fixed subsets without updating parameters. The separate batch-sampling generator ensures that model initialization and batch selection have recorded, independent random seeds.
A short reproducibility demonstration is useful here:
Pytorch Quick Tip: Reproducible Results and Deterministic Behavior
Watch “Pytorch Quick Tip: Reproducible Results and Deterministic Behavior” by Aladdin Persson for the basic reason to seed each random-number source that an experiment actually uses.
Watch seed setup. Notice the distinction between PyTorch’s seed and the additional seeds needed if code also uses NumPy or Python’s random module. This experiment generates data and batches with explicit PyTorch generators; it does not use NumPy.
Run the loop below without changing settings midway through. On a different PyTorch release or hardware configuration, exact floating-point values need not match another run; the fixed data, seeds, settings, and software version make your run inspectable and repeatable.
optimizer = torch.optim.AdamW(model.parameters(), lr=0.003)
batch_rng = torch.Generator().manual_seed(3303)
@torch.no_grad()
def mean_token_loss(inputs, targets):
model.eval()
loss_sum = 0.0
target_count = 0
for start in range(0, len(inputs), 128):
xb = inputs[start:start + 128].to(device)
yb = targets[start:start + 128].to(device)
logits = model(xb)
loss_sum += F.cross_entropy(
logits.reshape(-1, VOCAB),
yb.reshape(-1),
reduction="sum",
).item()
target_count += yb.numel()
model.train()
return loss_sum / target_count
history = []
for step in range(201):
if step % 50 == 0:
train_loss = mean_token_loss(train_x[:256], train_y[:256])
val_loss = mean_token_loss(val_x, val_y)
history.append((step, train_loss, val_loss))
print(
f"step={step:3d} train={train_loss:.4f}"
f" validation={val_loss:.4f}"
)
if step == 200:
break
row_ids = torch.randint(
len(train_x), (64,), generator=batch_rng
)
xb = train_x[row_ids].to(device)
yb = train_y[row_ids].to(device)
optimizer.zero_grad(set_to_none=True)
logits = model(xb)
loss = F.cross_entropy(
logits.reshape(-1, VOCAB), yb.reshape(-1)
)
if not torch.isfinite(loss):
raise RuntimeError(f"Non-finite training loss at step {step}")
loss.backward()
optimizer.step()
torch.save(
{
"model": model.state_dict(),
"optimizer": optimizer.state_dict(),
"history": history,
"torch_version": torch.__version__,
"settings": {
"model_seed": SEED,
"train_data_seed": 1101,
"validation_data_seed": 2202,
"batch_seed": 3303,
"steps": 200,
"batch_size": 64,
"learning_rate": 0.003,
"device": str(device),
"experts": 4,
"top_k": 2,
"dispatch": "dropless",
"selected_gate_renormalization": False,
},
},
"tiny_moe_grouped_run.pt",
)
At step zero, a model producing nearly uniform probabilities over 16 tokens would have a loss near , about ; random initialization need not produce exactly that value. Examine the recorded curve rather than assuming a particular final loss. A lower validation loss than at step zero is evidence that the model learned something about the generated sequences. A small train–validation gap is reassuring for this fixed split, but neither observation establishes which experts, if any, contributed distinctive functions.
If the run raises a non-finite-loss error, preserve the step number and settings rather than silently changing several controls at once. If it runs but makes little progress, first verify the shifted targets, the causal mask, and that the validation calculation does not update weights. Those checks are more informative than immediately increasing model size.
What to retain for the next lesson
You now have fixed grouped data, a tiny top- MoE decoder, a validation trajectory, and a checkpoint containing the settings needed to inspect the run again. Reproducible training means the data construction, initialization, batch selection, model convention, and evaluation procedure are specified—not that a loss curve alone explains the router.
Next, you will compare this MoE run with an approximately matched-active-compute dense model. After that, held-out group-conditioned routes and expert ablations will help distinguish useful specialization from merely uneven expert usage.