Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 28

Per-Rank Memory Estimation for Sharded Expert and Replicated Shared Weights

Recommended resources

Last lesson estimated the bytes that move when token assignments cross expert-parallel ranks. Those exchanges carry activations and gradients of activations; they do not move the expert weights on every forward pass. This lesson accounts for what stays in each rank’s memory during training: model weights, parameter gradients, and Adam’s update state.

The central distinction is between parameters a token uses and parameters a rank stores. Top- routing limits the former. Expert sharding limits the latter.


Start with ownership, not routing

Consider an expert-parallel group of ranks, with no tensor parallelism. Let be the number of shared parameters: attention, embeddings, layer norms, routers, and any other non-expert weights included in the model being counted. Every rank holds a copy of these parameters. Let be the number of expert parameters owned by rank . Its resident parameter count is

If there are equal-sized experts, each with parameters, divided evenly across the ranks, then

This assumes divides evenly. Otherwise, count the experts assigned to each rank separately.

Recall the per-token parameter convention from earlier in the course: a top- token uses shared parameters plus selected experts, giving under the simplified equal-expert assumption. That is not a storage formula. A rank must retain its owned experts even when the current batch routes no tokens to one of them. Likewise, sending a token to a remote expert does not create a second copy of that expert’s weights on the token’s origin rank.

Shared parameters have a different ownership rule. Each rank computes with its local shared-weight copy, and training must keep those copies consistent, typically by synchronizing their gradients or updates. An expert gradient is accumulated where that expert lives from all the assignments it receives; distinct expert owners do not synchronize gradients for different experts. If there are data-parallel copies of the same expert, those copies do need synchronization.



Turn resident parameters into training bytes

A useful first accounting convention is the one illustrated in this short video.

Ultimate Guide To Scaling ML Models - Megatron-LM | ZeRO | DeepSpeed | Mixed Precision

“Ultimate Guide To Scaling ML Models” by Aleksa Gordić breaks down why storing 16-bit weights alone substantially understates training memory.

Watch the Adam breakdown. Identify the separate 16-bit weight and gradient arrays, the 32-bit master weights, and Adam’s two 32-bit moment arrays. Treat its resulting 16 bytes per parameter as a consequence of those particular storage assumptions, not as a universal constant.

For our main estimate, use a different, explicitly stated convention: BF16 model weights, an FP32 gradient buffer, separate FP32 master weights for updates, and FP32 first and second moments for Adam. Assume each resident parameter has allocated storage in every category. The inventory is:

Storage on a rank Bytes per resident parameter
BF16 model weight 2
FP32 parameter gradient 4
FP32 master weight 4
Adam first and second moments, FP32 each 8
Total 18

The master weight is an additional parameter copy used for updates, not one of Adam’s two moments. Keeping these categories separate makes both the arithmetic and later comparisons easier to audit. For rank ,

The counts on the right are parameters; the results are bytes. Together they give

Here states means the four persistent categories in the table. It does not include saved activations, token-dispatch buffers, routing metadata, temporary update buffers, or allocator overhead. Thus is a planning estimate for model-state memory, not a prediction of peak GPU memory.



Work through a two-rank model

Suppose the whole model has million shared parameters and four experts of million parameters each. Two ranks own two experts apiece. There are million distinct model parameters, but each rank stores million shared parameters and million expert parameters: million.

Using the 18-byte convention, the per-rank breakdown is:

Storage Shared copy Owned experts Rank total
BF16 model weights 40 MB 40 MB 80 MB
FP32 gradients 80 MB 80 MB 160 MB
FP32 master weights 80 MB 80 MB 160 MB
Adam moments 160 MB 160 MB 320 MB
Total 360 MB 360 MB 720 MB

These figures use decimal MB, where one MB is one million bytes. Without expert sharding, a rank storing all million parameters under the same convention would need MB of model states. Expert sharding saves MB per rank here. It does not divide the entire MB by two, because the million shared parameters remain replicated.

Nor should you add the two ranks’ MB totals and call the result the model’s parameter count: that sum includes two copies of the shared states. The distinct-parameter count and the cluster-wide allocation answer different questions.

Even expert placement matters before a token is routed. If rank 0 owns three of these experts and rank 1 owns one, their resident parameter counts become million and million. Their model-state estimates are consequently MB and MB. A capacity check must use the largest per-rank estimate, not the average.



Why published bytes-per-parameter figures differ

The Megatron Core documentation tabulates several concrete precision and optimizer arrangements. Read its table as a set of storage contracts: identify which arrays remain replicated and which may be partitioned before applying a coefficient to an MoE layout.

Distributed Optimizer — Megatron Core

Read Megatron Core’s “Distributed Optimizer” documentation to compare the memory convention above with implementation-specific byte counts.

Under “Distributed Optimizer,” begin with the motivation. Continue through the precision assumptions and the three-row table immediately below, ending at the “fp32 parameters, fp32 gradients” row. Focus on why the BF16-parameter, FP32-gradient row lists 18 bytes without a distributed optimizer, while the FP16-parameter, FP16-gradient row lists 20.

Our 18-byte convention matches the documentation’s non-distributed BF16-parameter, FP32-gradient row. By contrast, the video’s 16-byte example counts 2-byte gradients alongside 2-byte weights, an FP32 master copy, and two FP32 moments. The documentation’s 20-byte FP16-parameter, FP16-gradient case additionally accounts for an FP32 main-gradient buffer in its implementation. These totals differ because the retained arrays differ; a precision label by itself does not determine training memory. An FP32-weight, FP32-gradient arrangement can also total 16 bytes when it has no separate master-weight copy.

Before borrowing any “bytes per parameter” number, write down whether it includes a master copy, a main-gradient buffer, and both Adam moments. Then multiply that number by the parameters resident on the rank, rather than by the global MoE parameter count or by the per-token active count.



Keep optimizer sharding separate from expert sharding

So far, expert parallelism has assigned different experts to different ranks. A distributed optimizer addresses a different redundancy: multiple ranks holding copies of the same parameters may partition their optimizer-owned states across those replicas.

The documentation’s “Data flow” description shows how an update can leave complete model-weight copies ready for the next forward pass while distributing the update work and its associated state.

Distributed Optimizer — Megatron Core

Continue the same documentation to see what is partitioned across data-parallel copies, rather than across owners of different experts.

In “Data flow,” read the paragraph before “Sharding scheme,” beginning at the buffer distinction. Then, under “Key steps,” follow the update trace: locate the reduce-scatter, the optimizer step, and the all-gather. Notice that this is a different kind of coordination from the token all-to-all examined in earlier lessons.

For the documentation’s BF16-weight, FP32-gradient arrangement, suppose the optimizer partitions the FP32 master weights and two FP32 moments over data-parallel copies of the same resident parameter set, while BF16 model weights and FP32 gradients remain allocated on each copy. The theoretical steady-state model-state estimate becomes

The 6 bytes are replicated BF16 weights and FP32 gradients; the 12 bytes are the partitioned master weight and two moments. Setting recovers 18 bytes per resident parameter.

Do not set merely because there are expert-parallel ranks. Ranks owning different experts are not automatically data-parallel copies of those experts. Shared parameters and expert parameters can also have different replica groups. If their optimizer partitioning degrees are and , respectively, the corresponding estimate is

This is an optional refinement to the earlier 18-byte calculation, not a replacement for specifying who owns each expert. Optimizer partitioning introduces its own synchronization and temporary-buffer considerations; it does not change the expert assignment traffic counted last lesson.



What this estimate can—and cannot—decide

A sensible memory worksheet records, for each rank, its shared parameter count, owned-expert parameter count, bytes for each weight and gradient copy, optimizer-state arrays, and any optimizer partitioning group sizes. Calculate every rank separately if expert placement is uneven. State whether the result uses MB or MiB.

The resulting model-state number is a baseline for a fit check, not the final answer. During training, saved activations, skewed expert batches, all-to-all buffers, temporary gradient conversions, and the optimizer step can raise the peak above it. Conversely, an expert that has not yet received a gradient may not have every optimizer array allocated in a particular implementation. Our worked estimate deliberately assumes a steady state in which the listed arrays are present for all resident parameters.

The useful separation is now precise: top- determines which experts compute for a token; expert ownership determines which expert states a rank retains; precision and optimizer design determine bytes per resident parameter. Next lesson, you will simulate two-rank dispatch in a single process and check that sharding expert ownership leaves the sparse layer’s combined output unchanged.