Hello. This lesson focuses on a specific failure mode in dropless mixture-of-experts training: every selected expert contribution is processed, but an uneven routing pattern can put too many activations on one device. You will build an explicit storage estimate, apply it to a skewed batch, and decide whether the busiest device exceeds an activation-memory budget. Allow about 40 minutes, including the short video, readings, and worked calculation.
Dropless does not mean memory-free
For a batch of tokens using top- routing, a dropless MoE layer has expert assignments, assuming each token selects distinct experts. An assignment is one token’s contribution to one expert; with top-2 routing, a token contributes two assignments. Dropless dispatch keeps both, even if many tokens select the same expert.
Capacity-limited dispatch makes a different trade-off: it sets a maximum number of assignments per expert and may omit contributions that exceed that limit. The following video contrasts that fixed-slot approach with processing unequal expert batches without padding.
Mixture of Experts (MoE), Visually Explained
Watch “Mixture of Experts (MoE), Visually Explained” by Jia-Bin Huang for the distinction between overflow, padding, and dropless expert computation.
In expert capacity, follow why uneven routing can overflow fixed slots and why extra slots can waste storage. Then watch dropless computation, focusing on how experts can process unequal numbers of tokens without allocating padded work for every expert.
Avoid a tempting inference from the video: not padding every expert does not mean activations cannot run out of memory. Unequal expert batches still have to be stored and processed somewhere. With expert parallelism, experts live on different devices, so the distribution across devices matters as well as the total number of assignments.
The Megatron Core documentation describes dropless dispatch as its default and notes that severe early-training load imbalance can cause out-of-memory failures. Read only the two passages relevant to this estimate.
Mixture of Experts package — Megatron-LM - NVIDIA Documentation
Read NVIDIA’s Megatron Core MoE documentation to connect the dropless-versus-capacity distinction to a reported training failure.
Under “User Guide” → “Usage” → “Quick Start,” find the paragraph introducing the figure on dropping strategies. Read the three strategies, noting that a capacity factor belongs to a token-dropping or padding policy, not to the dropless count used below. Under “End-to-End Training Practice,” find “OOM Caused by Token Distribution Imbalance when Training From Scratch.” Read the imbalance warning. The subsequent remedies are not needed for this estimate.
Specify what the budget counts
A memory estimate is meaningful only when its peak and its accounting boundary are stated. An activation-memory budget is not automatically the full GPU-memory capacity. Parameters, gradients, optimizer states, communication buffers, and allocator overhead may also occupy memory. If the input instead gives total device capacity, those other costs must be accounted for before treating the remainder as an activation budget.
Memory can peak at different phases of a training step. The reading below establishes why checking only one phase is insufficient for an overall device-memory decision.
Visualize and understand GPU memory in PyTorch
Read the “Estimating Memory Requirements” section of this Hugging Face article for its distinction between a forward-pass peak and an optimizer-step peak.
In “Estimating Memory Requirements,” start with the sentence “From the above section, estimating GPU memory requirements seems simple.” Read the peak comparison. Focus on the change in the largest peak when batch size changes. For this lesson, the supplied budget and storage model will isolate a particular activation peak; they do not by themselves certify that the entire training step fits on a device.
We will estimate the instant when an MoE layer’s expert activations are live. Let be the number of assignments received by expert , and let be the experts hosted on rank . A useful supplied storage model is
Here is other activation storage already live at that instant. The coefficient is storage retained per assignment across the rank’s experts. The optional coefficient represents a temporary workspace sized for the largest local expert batch; set when the stated model has no such workspace. Each term is included only if its storage is live at the same peak. Otherwise, adding it would overstate that peak.
This formula separates two effects that are easy to conflate. The sum measures how much work lands on a rank. The maximum measures how large one expert’s batch becomes. A storage model may include either effect or both. Expert parameter counts cannot replace : expert weights occupy memory whether that expert receives zero tokens or thousands, whereas the assignment-dependent activations change with routing.

A two-rank estimate
Consider one top-2 MoE layer with four experts. Two ranks receive a combined batch of tokens before expert dispatch, so the layer creates
expert assignments. Rank 0 hosts experts 0 and 1; rank 1 hosts experts 2 and 3. After dispatch, the counts are:
| Rank | Expert | Assignments |
|---|---|---|
| 0 | 0 | 3072 |
| 0 | 1 | 2048 |
| 1 | 2 | 1536 |
| 1 | 3 | 1536 |
These counts pass two basic checks. They sum to , so no assignment has been dropped. No expert receives more than assignments, consistent with distinct top-2 choices from tokens. The ranks need not receive equal numbers of assignments after dispatch, even if they began with equal numbers of source tokens.
Now supply a storage model, rather than trying to infer one from parameter count:
- Expert input copies retain elements per assignment.
- Two expert intermediate tensors each retain elements per assignment.
- Expert output staging retains another elements per assignment.
- Each element uses 2 bytes. This model treats all four tensors as simultaneously live.
- On each rank, other activations already live at this instant occupy MiB. This includes any other buffers we intend to count; there is no additional assignment-dependent workspace, so .
- The activation-only budget is MiB per rank. One MiB is bytes.
The per-assignment storage is
Rank 0 receives assignments. Rank 1 receives . Substituting into the model gives
Decision under the supplied model: rank 0 exceeds its -MiB activation budget by MiB. Rank 1 is MiB below budget. The batch fails the per-rank check even though the average rank looks safe.
That average is instructive. If all four experts had received assignments, each rank would have held assignments. Its estimated peak would have been MiB, just below budget. Routing skew did not change the total assignments; it concentrated more of their storage on rank 0.
The -MiB figure is an estimate, not a measurement guarantee. It follows exactly from the stated tensor lifetimes, sizes, and byte widths. A real implementation might reuse one of those buffers, retain another not listed here, or reserve more memory through its allocator. Those differences require a revised model or a measured peak—not an adjustment to the routing counts.
Match the calculation to the allocation strategy
The worked estimate assumes storage proportional to actual, dropless assignments. A static buffer has a different rule. Suppose an implementation reserves assignment slots for each local expert, regardless of actual routing, and each slot carries the same -KiB modeled storage. Each rank hosts two experts, so its reserved expert storage would be
With the same -MiB fixed amount, that model estimates MiB on both ranks. It describes reserved allocation, not the -MiB and -MiB live-assignment estimates. Do not substitute actual into a fixed-reservation model, or fixed slot counts into an allocation that grows with actual .
The scope also changes the result. If all experts were on one device and the only variable term were , that sum would remain for every dropless routing pattern. Skew alone would not change that model’s variable total. It could still change a largest-expert workspace, temporary tensor lifetimes, or reserved-buffer requirements. Under expert parallelism, even the simple sum changes per rank, which is why the two-rank example fails.
For multiple MoE layers, apply the model at each candidate peak with the activations actually live there. Earlier layers’ retained activations belong in that peak’s , or in another explicitly stated term. Do not add several layer-peak estimates together if each already includes the same retained storage. Likewise, do not take only the busiest expert’s count when the model says that all local expert activations remain live.
A short calculation can make the units and per-rank comparison auditable:
counts = (3072, 2048, 1536, 1536)
local_experts = ((0, 1), (2, 3))
N, k = 4096, 2
d, f, bytes_per_element = 2048, 8192, 2
assert sum(counts) == N * k
assert max(counts) <= N # distinct top-2 choices
bytes_per_assignment = (2 * d + 2 * f) * bytes_per_element
fixed_bytes = 600 * 2**20
budget_bytes = 770 * 2**20
for rank, expert_ids in enumerate(local_experts):
local_assignments = sum(counts[e] for e in expert_ids)
estimated_bytes = fixed_bytes + local_assignments * bytes_per_assignment
print(
rank,
local_assignments,
estimated_bytes / 2**20,
estimated_bytes > budget_bytes,
)
The outputs are rank 0: assignments, MiB, over budget; and rank 1: assignments, MiB, under budget. The assertions check the accounting convention; they do not prove that a real kernel retains precisely the tensors in the supplied storage model.
Takeaway
A dropless memory decision requires four stated inputs: assignment counts, expert ownership, a storage-and-lifetime model, and a per-device budget. Count all selected contributions, estimate each rank’s peak under the specified allocation strategy, and compare the largest per-rank result with its budget. In the worked batch, the skewed rank reaches MiB against MiB, so the batch exceeds the stated activation budget.
The next lesson examines how partitioning the same token stream into different microbatch sizes changes load-balancing measurements. Those partitions can also change which assignment counts are present at a single memory peak, making the accounting boundary established here important there as well.