Hello. Last lesson traced eight expert assignments from four top-2 tokens through dispatch and return. Three assignments crossed ranks; five stayed local. That distinction is the key to estimating communication: an all-to-all describes the exchange pattern, but only cross-rank assignments consume inter-rank bandwidth.
This lesson turns that trace into byte counts for the forward and backward passes. You will estimate both the size of the data arranged for the exchanges and the smaller amount actually sent between ranks.
Count assignments before counting bytes
Let be the total number of tokens across the participating ranks at one MoE layer, the number of selected experts per token, and the number of elements in each token representation. Assume every selected assignment runs, each expert returns a -element vector, and inputs and outputs use bytes per element.
The layer creates token–expert assignments. One vector per assignment therefore contains
elements, or bytes. This is the size of the logical assignment payload for one exchange: it includes records whose expert is on the token’s own rank. It is not yet a network-traffic figure.
The following reading develops the two-exchange forward estimate and discusses how rank placement affects actual transfers.
Expert Parallelism: Distributed Computing for MoE Models - Interactive
Read “Expert Parallelism: Distributed Computing for MoE Models” at mbrenndoerfer.com for its assignment-based communication estimate. Keep the distinction between logical payload and cross-rank traffic in mind.
In “Communication Volume,” begin where the author says “The first operation dispatches tokens from their originating devices” and read the forward estimate through the paragraph explaining how each factor scales the volume. The reading writes token count as batch size times sequence length; here we call that total T. Then, in “Communication vs Computation Trade-off,” skip the opening discussion of arithmetic intensity. Begin at “As we scale to more devices” and read the placement discussion. Treat its uniform-routing fraction as an assumption for an estimate, not a guarantee for every batch.
The first forward exchange dispatches one input vector per assignment to its expert. The second returns one expert-output vector per assignment to its token’s origin. With equal input and output widths, the logical forward payload, summed over both exchanges, is
If batch size is sequences of length , then . Thus the reading’s counts elements; multiplying by converts it to bytes. It is a forward-pass figure, not a forward-plus-backward figure.
Separate logical payload from network traffic
Define as the fraction of assignments whose expert owner differs from the token’s origin rank. Count assignments, not distinct tokens: a top-2 token can have two local assignments, one local and one remote, or two remote assignments. The two remote assignments might even target different experts on the same destination rank; both still require expert computations and output records.
Under our equal-vector-size assumptions, each remote assignment sends an input vector outward and an output vector back. Summed across all ranks, the estimated forward network payload is
A local assignment may require buffer rearrangement, but it does not transmit its vector between ranks. Accordingly, is useful for sizing the complete dispatch and return payloads, while estimates the cross-rank bytes within them.
Here “summed across all ranks” counts each one-way transmission once. Counting the sender’s bytes and then counting the receiver’s copy of those same bytes as another transmission would double the estimate.
For a rough planning estimate with ranks, equally placed experts, and routing independent of the token’s origin, approximately of assignments stay local. That gives . Actual router choices can produce a very different fraction, so a recorded cross-rank assignment count is preferable when available.
Revisit the two-rank trace
The previous batch had tokens, , and eight assignments. Three assignments originated on rank 0 and ran on rank 1; none crossed in the other direction. Thus
Suppose now that and each element occupies bytes. One assignment vector occupies bytes. The dispatch exchange arranges eight such vectors, or 64 logical bytes, but sends only three of them across ranks: 24 network bytes. Return has the same counts. The forward pass therefore accounts for 128 logical bytes and 48 cross-rank bytes.
The unequal direction of transfer matters for performance. All three remote inputs travel from rank 0 to rank 1; their results travel back from rank 1 to rank 0. The aggregate byte formula is correct, but it does not describe those directional loads or the fact that rank 1 performs most of the expert work.
Backward communication mirrors the assignments
Training needs gradients through the experts, not just their forward outputs. Assume the origin rank applies the gate weights and combines returned expert results, as in the previous lesson. During backward propagation:
- The origin rank sends the gradient for each expert-output contribution to the rank that ran that expert. For a remote assignment, this transfers a -element vector.
- The expert-owning rank computes its expert parameter gradients and an input gradient for that assignment. It sends that input-gradient vector back to the token’s origin, where gradients from selected experts can be accumulated.
These are two further exchanges. If gradient vectors have the same width and bytes per element as the forward vectors, their payloads match the two forward exchanges:
Consequently, for one MoE layer, one training step, and this communication scheme,
For the small two-rank trace, backward adds 48 cross-rank bytes, making 96 cross-rank bytes for forward and backward together. The four exchanges are dispatch and return in forward, followed by an output-gradient exchange and an input-gradient return in backward.
This estimate concerns token-sized activation and gradient payloads. Expert weights remain on their owner ranks; their gradients are computed there. Any separate synchronization of replicated shared parameters is outside this count.
Use the estimate at a more realistic width
Consider one dropless MoE layer with tokens across all ranks, , , and two bytes per element. Suppose measured routing shows that of assignments cross ranks.
One exchange has elements, occupying 8 MiB of logical payload. Three-quarters of that payload, 6 MiB, crosses ranks. The four exchanges together give:
| Scope | Logical payload | Cross-rank payload |
|---|---|---|
| Forward: dispatch and return | 16 MiB | 12 MiB |
| Backward: two gradient exchanges | 16 MiB | 12 MiB |
| Forward and backward | 32 MiB | 24 MiB |
These are aggregate amounts for the layer, not automatically per-rank amounts. Equal aggregate traffic can conceal a rank that sends much more than another, just as the earlier trace concealed a seven-to-one expert-compute imbalance.
The calculation is a payload estimate, not a step-time prediction. Message setup, network topology, contention, local packing, overlap with computation, and the slowest rank all affect elapsed time. It also excludes routing metadata and possible padding in fixed-size communication buffers. If capacity limits drop assignments before dispatch, use the number actually dispatched rather than blindly substituting .
Conclusion
Start with token–expert assignments: assignments carrying elements of bytes each. Two exchanges make the forward logical payload ; two more make the equal-sized backward payload. Multiplying by the observed cross-rank fraction gives the estimated network payload, for forward and backward together, under the stated assumptions.
The next lesson will apply the same care about ownership and replication to estimate per-rank parameter, gradient, and optimizer-state memory.