Hello. In the previous lesson, you simulated two expert-parallel ranks and checked that dispatching tokens to sharded experts produced the same output as an unsharded sparse layer. That established correctness, but a single-process simulation cannot tell you what limits throughput when tokens actually move between devices.
This lesson uses phase-timing traces to distinguish three possible limits: expert computation, communication, and uneven work across ranks. The key is to follow the step’s elapsed time, not just find its longest-looking kernel. Allow about 40 minutes, including the reading and the worked traces.
Read a trace as a timeline
A useful expert-parallel trace shows when each rank dispatches assignments, computes its local experts, and returns results. It also shows gaps: a rank may finish its experts early yet be unable to proceed until another rank finishes. That gap affects throughput even though no useful expert work occurs on the waiting rank.
PyTorch’s trace-analysis tutorial provides the vocabulary for separating compute, non-compute, and idle time. Read the indicated parts for the measurement concepts; you do not need to run its analysis code for this lesson.
Introduction to Holistic Trace Analysis — PyTorch Tutorials 2.15.0+cu132 documentation
Read this PyTorch tutorial to learn what a GPU time breakdown does—and does not—tell you about a distributed step.
In “Temporal Breakdown,” read the category definitions. In “Idle Time Breakdown,” read the idle categories; note that a gap need not be caused by the network. In “Kernel Breakdown,” read the kernel categories. Finally, in “Communication Computation Overlap,” read the overlap explanation. Focus on why elapsed step time cannot always be obtained by adding category totals.
Two distinctions matter throughout the lesson. First, non-compute time is not synonymous with network transfer: it can include memory operations. Second, time attributed to a communication operation can include waiting for a peer. To identify a bottleneck, align events across ranks and look for the work that holds up completion of the step.
A simple critical-path model
Start with a deliberately simplified MoE layer in which phases do not overlap. Both ranks finish dispatch before expert computation begins; return transfer completes after both ranks have finished their experts. Let be elapsed dispatch time, and the two ranks’ expert-compute times, and elapsed return-transfer time. The layer’s elapsed time is then
The maximum, rather than the average, is crucial. If rank 0 computes for 10 ms while rank 1 computes for 3 ms, the expert phase takes 10 ms. Rank 1’s remaining 7 ms is waiting time; making its 3 ms of computation faster would not shorten this layer.
This model is a diagnostic starting point, not a formula for every implementation. Real traces may overlap transfers with other computation, contain token-packing kernels, or include CPU launch gaps. When work overlaps, summing kernel durations can double-count elapsed time. Use actual start and end times to find which events delay the step’s completion.
For now, keep the model’s assumptions and examine three traces. Times are in milliseconds; “assignments received” counts selected token–expert pairs processed on each rank, not necessarily distinct tokens. The two ranks own different experts as in the previous lesson.
| Trace | Dispatch elapsed | Expert compute: rank 0 / rank 1 | Return transfer elapsed | Assignments received: rank 0 / rank 1 |
|---|---|---|---|---|
| A | 1 | 12 / 11 | 1 | 101 / 99 |
| B | 7 | 2 / 2 | 6 | 100 / 100 |
| C | 2 | 10 / 3 | 2 | 150 / 50 |
Trace A: expert computation dominates. Its elapsed time is ms. Work is nearly even across ranks, and the two transfers together take only 2 ms. Most of the critical path is expert computation. This diagnosis says where time is spent; it does not yet say whether the expert kernels are efficient. Many tiny expert batches, for example, could make computation slow despite low GPU utilization.
Trace B: communication phases dominate. Its elapsed time is ms, with 13 ms in dispatch and return and no evident expert-load imbalance. Reducing expert compute alone offers little headroom. Investigate the collectives and adjacent token-packing work. The trace establishes that the communication phases limit this serial step; it does not, by itself, prove that raw network bandwidth is the cause. Latency, packing, and synchronization can contribute.
Trace C: rank imbalance dominates the expert phase. Its elapsed time is ms. Both transfers are short, but rank 1 finishes expert work 7 ms before rank 0. The received-assignment counts, 150 versus 50, provide a plausible explanation for the compute difference. Rank 0’s experts are on the critical path. The immediate diagnostic is not simply “expert compute is large”: uneven placement of expert work makes one rank a straggler.
These categories can coexist. Trace C spends most of its time in an expert phase and is imbalanced. Calling out imbalance is more informative than labeling it merely “compute-bound,” because speeding up the already-fast rank will not help.
Why a collective can appear slow on the fast rank
Trace C illustrates a common trap. Suppose dispatch finishes at 2 ms. Rank 1 finishes its experts at 5 ms and enters the return collective. Rank 0 finishes at 12 ms and enters the same collective; the return completes at 14 ms.
A timer around the collective call could report 9 ms on rank 1 but 2 ms on rank 0. It would be tempting to conclude that rank 1 has a communication problem. Yet rank 1 entered 7 ms earlier. Much of its apparent collective time is consistent with waiting for rank 0’s expert work, not transferring its own results slowly. The shared return window in the simplified trace is 12–14 ms; finer profiling would be needed to partition that window into actual transfer and other overhead.
To make a defensible diagnosis from a supplied trace, proceed in this order:
- Establish elapsed step and phase boundaries. Compare aligned timestamps across ranks rather than summing per-rank durations.
- Compare expert work and assignment counts. A long expert phase on every rank with similar loads suggests a computation limit. One slow rank paired with an early-finishing rank suggests imbalance; counts help explain why, though expert costs need not scale perfectly with count.
- Inspect dispatch and return on both ranks. Long collective intervals with ranks arriving at similar times support a communication-phase diagnosis. A long interval only on an early-arriving rank may chiefly reflect waiting for its peer.
- Account for gaps and overlap. CPU launch delays, memory operations, and overlapping communication can invalidate a diagnosis based only on a “compute versus communication” pie chart.
If the trace lacks aligned timestamps or assignment counts, say what remains uncertain. A single large collective-duration number is not enough to separate transfer cost from straggler-induced waiting.
Key takeaways
Throughput is limited by work on the critical path, not by the sum of work reported across ranks. In a serial expert-parallel trace, similar long expert times point toward expert computation; long dispatch and return phases with synchronized ranks point toward communication overhead; and a slow expert rank while its peer waits points toward rank imbalance. Always check whether a collective’s reported time includes waiting for another rank.
The next lesson begins the final module by testing for silent dispatch and combination errors—failures that a timing trace might not reveal even when performance appears reasonable.