Hello. This lesson turns the expert-parallel layout from the previous lessons into a single-process, two-rank simulation. You will keep the router and experts in ordinary PyTorch, but make token dispatch, rank-local expert computation, and output return explicit. The final check compares the simulation with an unsharded sparse layer using the same weights.
The aim is correctness, not distributed speed: no process group, GPU-to-GPU transfer, or capacity limit is needed. Allow about 40 minutes, including reading and running the code.
What the simulation must preserve
Suppose there are four experts and two simulated ranks. Rank 0 owns experts 0 and 1; rank 1 owns experts 2 and 3. Each rank starts with some token representations and runs the same router on its own tokens. With top-2 routing, one token produces two assignments. Those assignments may go to different ranks, even though both originated from the same token.

For an assignment to survive the trip, its representation alone is insufficient. We also need its destination expert and a return address: source rank, token position within that source, and top- slot. The slot matters because the two selected experts can have different gate weights.
The following reading connects those bookkeeping requirements to an actual all-to-all layout.
Expert Parallelism: Distributed Computing for MoE Models - Interactive
Read Michael Brenndoerfer’s discussion of variable-length communication and token return addresses. It supplies the distributed motivation for the small simulation below.
In “Handling Variable-Length Communication,” read the explanation from the opening paragraph through the paragraph about the return address. Then inspect the prepare_all_to_all_buffers function immediately below it: focus on expansion into token-expert assignments, sorting by destination, send counts, and the inverse permutation; you need not run the notebook example. In “Worked Example: Tracing a Token Through Expert Parallelism,” read the five-step trace, from its opening through the final paragraph. Track where each of the two expert results must return.
The reading groups assignments by destination and uses a permutation to recover their positions. We will use explicit return-address fields instead. Both solve the same ordering problem; neither means an expert result can be combined merely because it arrived on the right rank.
Build an independent unsharded reference
We first need a conventional sparse layer as an oracle. Its forward pass selects experts, batches tokens for each expert, and accumulates weighted contributions at the original token positions. The simulation will share its router and expert objects, so a mismatch cannot be explained by different parameter initialization.
This example uses four-dimensional token representations, four experts, and gate weights renormalized over the selected experts. It is dropless: every selected assignment contributes.
import torch
from torch import nn
class TinyMoE(nn.Module):
def __init__(self, width=4, num_experts=4, k=2):
super().__init__()
self.k = k
self.gate = nn.Linear(width, num_experts, bias=False)
self.experts = nn.ModuleList(
nn.Sequential(
nn.Linear(width, 8),
nn.GELU(),
nn.Linear(8, width),
)
for _ in range(num_experts)
)
def route(self, x):
probs = torch.softmax(self.gate(x), dim=-1)
selected_probs, expert_ids = torch.topk(probs, self.k, dim=-1)
weights = selected_probs / selected_probs.sum(dim=-1, keepdim=True)
return expert_ids, weights
def forward(self, x):
expert_ids, weights = self.route(x)
output = torch.zeros_like(x)
for expert_id, expert in enumerate(self.experts):
token_ids, slots = torch.where(expert_ids == expert_id)
if token_ids.numel() == 0:
continue
contributions = (
weights[token_ids, slots].unsqueeze(-1)
* expert(x[token_ids])
)
output.index_add_(0, token_ids, contributions)
return output
If token selects experts and , this reference computes
The reference groups work by expert, not by destination rank. That makes it a useful independent check of the simulated dispatch and return ordering.
Simulate dispatch, local computation, and return
The next function splits the input at split: earlier tokens originate on rank 0, later tokens on rank 1. Its send[source][destination] lists stand in for the first all-to-all exchange. Each packet contains (source, local_token, slot, expert_id, representation).
At the destination, only the two experts owned by that simulated rank are accessible through its shard. Results go into returned[destination][source], standing in for the reverse exchange. Gate weights stay at the source and are applied after results come back.
def simulate_two_ranks(model, x, split):
assert x.ndim == 2
assert 0 <= split <= x.shape[0]
assert len(model.experts) == 4
assert model.k in (1, 2)
local_x = [x[:split], x[split:]]
shards = [
tuple(model.experts[0:2]), # simulated rank 0
tuple(model.experts[2:4]), # simulated rank 1
]
# send[source][destination]: packets from the first all-to-all
send = [[[] for _ in range(2)] for _ in range(2)]
local_weights = []
for source in range(2):
expert_ids, weights = model.route(local_x[source])
local_weights.append(weights)
for token in range(local_x[source].shape[0]):
for slot in range(model.k):
expert_id = int(expert_ids[token, slot])
destination = expert_id // 2
packet = (
source,
token,
slot,
expert_id,
local_x[source][token],
)
send[source][destination].append(packet)
send_counts = [
[len(send[source][destination]) for destination in range(2)]
for source in range(2)
]
assert sum(map(sum, send_counts)) == x.shape[0] * model.k
# returned[destination][source]: processed assignments going home
returned = [[[] for _ in range(2)] for _ in range(2)]
for destination in range(2):
inbox = [
packet
for source in range(2)
for packet in send[source][destination]
]
for local_expert_id, expert in enumerate(shards[destination]):
global_expert_id = 2 * destination + local_expert_id
group = [
packet for packet in inbox
if packet[3] == global_expert_id
]
if not group:
continue
expert_input = torch.stack([packet[4] for packet in group])
expert_output = expert(expert_input)
for packet, value in zip(group, expert_output.unbind(0)):
source, token, slot, _, _ = packet
returned[destination][source].append(
(token, slot, value)
)
return_counts = [
[len(returned[destination][source]) for source in range(2)]
for destination in range(2)
]
assert all(
return_counts[destination][source]
== send_counts[source][destination]
for source in range(2)
for destination in range(2)
)
# Combine at each assignment's source token, not at its arrival position.
local_outputs = []
for source in range(2):
output = torch.zeros_like(local_x[source])
packets = [
packet
for destination in range(2)
for packet in returned[destination][source]
]
if packets:
tokens = torch.tensor(
[packet[0] for packet in packets],
dtype=torch.long,
device=x.device,
)
slots = torch.tensor(
[packet[1] for packet in packets],
dtype=torch.long,
device=x.device,
)
values = torch.stack([packet[2] for packet in packets])
contributions = (
local_weights[source][tokens, slots].unsqueeze(-1)
* values
)
output.index_add_(0, tokens, contributions)
local_outputs.append(output)
return torch.cat(local_outputs, dim=0), send_counts
Notice the two different uses of order. The destination may batch packets in expert order for computation. The source must nevertheless accumulate each returned value at its original local token index, using the original selected-expert slot to retrieve its weight. index_add_ permits two contributions to land on the same token row.
This is a simulation of communication, not communication itself. Both shards and all packet lists live in one Python process; the shard tuples refer to the reference model’s existing expert objects. In a real two-rank run, those experts would be on different devices, and the counts would help size variable-length receive buffers.
Check equivalence, including awkward splits
Use deliberately distinct token values so that both local and remote assignments occur. Setting the gate to a scaled identity also makes the selected experts easier to inspect: large coordinates favor the corresponding expert IDs.
torch.manual_seed(7)
model = TinyMoE()
with torch.no_grad():
model.gate.weight.copy_(4.0 * torch.eye(4))
x = torch.tensor([
[3., 2., 0., 0.],
[0., 0., 3., 2.],
[3., 0., 2., 0.],
[0., 3., 0., 2.],
[2., 0., 3., 0.],
[0., 0., 2., 3.],
[0., 2., 0., 3.],
])
for k in (1, 2):
model.k = k
for split in (0, 3, len(x)):
reference = model(x)
simulated, counts = simulate_two_ranks(model, x, split)
torch.testing.assert_close(
simulated, reference, rtol=1e-5, atol=1e-6
)
print(f"k={k}, split={split}, send counts={counts}: match")
All six checks should print match. For k=2, split=3, the send counts are [[3, 3], [3, 5]]: row 0 describes assignments originating on rank 0, while row 1 describes those originating on rank 1; columns are destination ranks. Counts need not be uniform. When split is 0 or the full input length, one source has zero tokens, which checks that empty sends and empty outputs are handled.
An output match establishes forward equivalence for these inputs and routing choices. It does not establish gradient equivalence, network performance, or correct behavior under capacity limits. In particular, removing the (token, slot) return address might still preserve packet counts while silently combining results with the wrong tokens or gate weights. That is why comparing only tensor shapes and send totals is weaker than comparing the complete output against an independently organized reference.
Key takeaways
Expert parallelism changes where selected expert work runs, not the sparse layer’s intended mathematical output. A correct two-rank simulation must create one packet per selected token-expert pair, send it only to the expert’s owner, and return enough information to weight and accumulate its result at the source token. The unsharded comparison gives you a concrete correctness boundary before introducing real distributed operations.
Next, you will use per-rank timing traces to distinguish expert computation, communication, and rank imbalance as throughput limits.