Hello. In an expert-parallel MoE layer, a token’s selected experts may live on a different GPU from the token itself. The router still decides which experts run; expert parallelism determines where their computations happen. This lesson follows one small top-2 batch through dispatch, expert computation, and the return of outputs to the tokens’ original ranks.
What the two exchanges accomplish
Here, a rank is a participating GPU process. Each rank holds a subset of the experts. Rather than copying expert weights to wherever tokens happen to be, ranks send token representations to the ranks holding the selected experts. After computation, they send the resulting vectors back.
Training MoEs at Scale with PyTorch – PyTorch
Read the PyTorch article’s explanation of expert parallelism to establish why tokens move instead of expert weights and why the forward pass needs a return exchange.
Find the “Expert Parallelism” subsection, immediately after Figure 2. Read the explanation, stopping just before Figure 3. Focus on the distinct purposes of the dispatch and return all-to-all operations.
The following diagram shows the same pattern across four GPUs. Its first exchange sends selected token representations to expert owners; its second sends expert results back. We will reduce it to two ranks so that every assignment can be accounted for.

An all-to-all means every participating rank can provide a different portion of its data to every other rank. It does not mean every token must visit every rank. A portion destined for its own rank can remain local, although it is still part of the collective’s logical layout.
A two-rank, top-2 batch
Suppose rank 0 owns experts and starts with tokens . Rank 1 owns experts and starts with tokens . Each token selects two distinct experts. For this example, the two retained gate weights have been normalized to sum to one for each token.
| Origin | Token | Selected expert: slot 0 | Selected expert: slot 1 |
|---|---|---|---|
| Rank 0 | , weight | , weight | |
| Rank 0 | , weight | , weight | |
| Rank 1 | , weight | , weight | |
| Rank 1 | , weight | , weight |
There are four tokens but eight expert assignments. In particular, needs two computations on rank 1, so its representation must be present in two assignment records. Sending each token only once to a destination rank would lose information about which selected experts should process it.
Think of each assignment as having an origin rank, token position, selected-expert slot, destination expert, and gate weight. An implementation need not transmit all of that as a separate label attached to every vector: ordered buffers and saved permutation indices can carry the bookkeeping. But it must preserve enough information to reverse the routing later.
The first all-to-all groups assignments by destination rank:
| Source rank | Sent to rank 0 | Sent to rank 1 |
|---|---|---|
| Rank 0 | , slot 0, for | , slot 1, for ; , slot 0, for ; , slot 1, for |
| Rank 1 | Nothing | , slots 0 and 1; , slots 0 and 1, each for its selected expert |
Thus rank 0’s destination counts are 1 for rank 0 and 3 for rank 1. Rank 1’s are 0 for rank 0 and 4 for rank 1. After dispatch, rank 0 has one assignment to compute; rank 1 has seven. Only the three assignments sent from rank 0 to rank 1 cross ranks. The local portions can be rearranged without a network transfer.
On receipt, each rank groups its assignments by local expert, not merely by source rank. Rank 0 supplies to , while receives no tokens. Rank 1 supplies to and to . The two appearances of are intentional: different experts produce different vectors from the same input representation.
Returning outputs without mixing up tokens
The expert computations produce eight output vectors, one per assignment. An output from for belongs to on rank 0; it does not become a rank-1 token merely because ran on rank 1.
Before the return exchange, each expert-owning rank arranges its output records by origin rank. It also preserves the association between each result and its original token and selected-expert slot. The second all-to-all then has these counts:
| Result source | Returned to rank 0 | Returned to rank 1 |
|---|---|---|
| Rank 0’s experts | 1 | 0 |
| Rank 1’s experts | 3 | 4 |
These are the dispatch counts viewed in reverse: rank 1 returns three remotely computed results to rank 0. Results for stay on rank 1. This return exchange moves expert outputs, not expert weights or newly routed tokens.
Finally, each origin rank restores its token order and combines the two selected contributions using its gate weights. Rank 0 obtains
Rank 1 obtains
Here is token ’s input representation at this MoE layer, and is its combined expert contribution. It is not necessarily the final output of the surrounding transformer block. The same distinction applies to the other tokens.
Notice why the slot bookkeeping matters even when two assignments go to the same rank: ’s slot 0 uses with weight , while slot 1 uses with weight . Returning two unlabelled vectors for would not say which weight belongs to which result.
The trace as an implementation contract
A correct dispatch-and-return implementation must satisfy a few invariants:
- Assignment count: four top-2 tokens create eight expert inputs and eight expert outputs, including assignments that remain on their origin rank.
- Expert ownership: each expert computation runs on the rank holding that expert’s weights. An expert with zero assignments, such as here, contributes no output.
- Reversible ordering: after the return exchange, every output is associated with the same origin token and selected-expert slot it had before dispatch.
- Correct combination: gate weighting is applied to the matching expert outputs; results are accumulated into the original token positions.
The first and second all-to-all operations solve different ordering problems. Dispatch groups records by where computation must run. Return groups results by where combination must occur. Local permutations around those exchanges arrange records for efficient expert computation and then undo that arrangement. The collective communication alone does not perform the weighted sum.
This batch also exposes a performance issue without changing its mathematical result: rank 1 computes seven assignments while rank 0 computes one. Correct routing and balanced work are separate concerns.
Conclusion
In expert parallelism, top-2 selection produces assignment records, not just a destination for each token. The first all-to-all delivers those records to expert-owning ranks; the second returns their output vectors to the originating ranks. Saved ordering information makes it possible to restore token positions and apply the right gate weight to each expert result.
The next lesson uses this trace to estimate how much data the forward and backward exchanges move.