Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 25

Expert vs. Data Parallelism in Sharded Model Layouts

Recommended resources

Hello. This module moves from a sparse expert layer on one device to the same logical model spread across devices. Earlier modules treated routing and expert computation as local operations; here the question is where each parameter lives. By the end of this lesson, you should be able to look at a simple layout and distinguish expert weights that are sharded from shared weights that are replicated, without confusing expert parallelism with data parallelism.


Start with the data-parallel baseline

In ordinary data-parallel training, each device holds a copy of the entire model. Devices process different portions of a batch, so their copies produce different gradients. Those gradients must be synchronized before the copies take matching optimizer steps.

Ultimate Guide To Scaling ML Models - Megatron-LM | ZeRO | DeepSpeed | Mixed Precision

Watch the data-parallelism explanation in “Ultimate Guide To Scaling ML Models” by Aleksa Gordić - The AI Epiphany. It establishes what “replicated” means during training, not just during inference.

Watch model replication, beginning when the presenter introduces data parallelism and ending after the explanation of averaging gradients from different batch portions. Focus on the distinction between copying weights and splitting input data.

Apply that picture to a four-expert MoE model. If two devices use pure data parallelism, both store the shared transformer components, router, and all four experts. Each device handles different training examples, but either could run the complete model alone. Synchronizing gradients keeps their copies of expert 0 consistent, their copies of expert 1 consistent, and so forth.

This improves how much data can be processed concurrently. It does not, by itself, reduce the number of model parameters each device must store. Expert parallelism makes a different placement choice.



Shard experts, keep the shared path replicated

Consider one transformer with attention and a sparse feed-forward sublayer containing experts 0–3. Place it on two devices, which we will call ranks. In the simplest expert-parallel (EP) group, each rank holds the shared components but only a subset of the experts:

Component Rank 0 Rank 1
Attention, normalization, and other shared weights Full copy Full copy
Router, which scores all four experts Full copy Full copy
Expert feed-forward weights Experts 0 and 1 Experts 2 and 3

The router on rank 0 still needs to be able to select expert 2, even though rank 0 does not store that expert. Selection is a decision about the logical model; placement determines which rank performs the selected computation. Likewise, rank 1’s router scores all four experts, not just its local two.

Parallelisms Guide — Megatron Bridge - NVIDIA Documentation

Read NVIDIA’s “Parallelisms Guide” for its concise definition of expert parallelism and a concrete experts-per-device example.

In “Expert Parallelism and Mixture of Experts (MoE),” read the definition, starting with “Expert Parallelism (EP) is a type of model parallelism” and ending with the sentence about the rest of the layers. Then go to “Basic MoE Configuration,” under “Enable Expert Parallelism,” and read the eight-expert example. Focus on which parameters are divided among devices, rather than on configuration syntax.

“EP applies only to expert layers” does not mean the rest of the transformer has no placement strategy. It means that expert sharding does not prescribe how non-expert weights are placed. In our simple layout, we choose to replicate them. A larger system might instead split attention weights using tensor parallelism. Keep the simple layout in view for now: whole experts are assigned to ranks; each rank has the same shared weights.

For four experts across two ranks, each rank stores two experts. The logical model still has four distinct experts. There have not been two experts removed from the model—only from each rank’s local storage.



What happens when a token selects a remote expert?

Suppose a token starts on rank 0. Rank 0 computes its shared transformer operations and router scores. If top-1 routing selects expert 1, rank 0 can run that expert locally. If it selects expert 2, rank 0 must send the token representation to rank 1, where expert 2 lives. The expert result must then reach rank 0 so the token can continue through its original sequence and the remaining shared computation.

For top-2 routing, one token might select experts 1 and 2. Those selections name two distinct experts even though the token began on just one rank: rank 0 runs expert 1, rank 1 runs expert 2, and their weighted outputs are combined for that token. The next lesson will trace the exchanges and bookkeeping precisely. For this lesson, the important observation is that activations travel to sharded expert weights; the system does not need to copy every expert’s weights to the token’s starting rank for each forward pass.

Placement also determines gradient ownership during training. Rank 1 accumulates expert 2’s weight gradients from tokens routed to it, including tokens that originated on rank 0. Rank 0 needs the backward signal for its token so it can train its earlier shared operations and router. Within this EP group, it would be wrong to average expert 0’s gradient with expert 2’s gradient as though they were replicas of one parameter: they are different experts. By contrast, the replicated shared weights must stay consistent across ranks processing different data.

This gives you a practical identification rule. If ranks hold different expert identities, those experts are expert-sharded. If ranks hold copies of the same parameter and train on different data, those copies require data-parallel synchronization. Both patterns can occur in one MoE layer.



An EP group can itself be replicated

The two-rank layout can be duplicated across four ranks:

Rank EP group Locally stored experts
0 A 0 and 1
1 A 2 and 3
2 B 0 and 1
3 B 2 and 3

Groups A and B each contain a complete distributed copy of the logical model. They can process different portions of the training data. Within either group, experts are sharded; across groups, corresponding experts are replicated. Expert 0 on rank 0 and expert 0 on rank 2 are copies that must receive consistent updates. Expert 0 on rank 0 and expert 2 on rank 1 are not copies and must not be synchronized as one weight.

The shared components in this example are present on all four ranks. Their gradients also need appropriate synchronization so those copies remain equal. This is why “the model is data parallel” and “its experts are expert parallel” can both be true, provided you specify which ranks form each group and which parameters you mean.

It is also why parameters stored per rank should not be mistaken for parameters active per token. Suppose the shared model has 40 million parameters and each of four experts has 25 million. The logical model has 140 million distinct parameters. In the two-rank EP layout, each rank stores 90 million: the shared 40 million plus two local experts. With top-1 routing, a token uses 65 million parameter values along its path: the shared 40 million plus one selected expert. Sharding changed local storage; top-1 selection determined active expert computation. Neither operation changed the count of distinct parameters in the logical model. These figures exclude optimizer states and activations, which we will account for later.



Recognizing a mixed layout

The supplied diagram shows why it helps to name the parallelism per component, rather than assign one label to an entire training run.

The diagram places eight experts across an expert-parallel arrangement, with each expert additionally split into two slices. Its non-expert parameters use four-way tensor slicing, repeated across four data-parallel groups. It is a mixed layout, not the simple two-rank EP layout used above.

Look first at the upper-right portion: expert parallelism distributes the eight expert identities. The boxes for expert slices on GPU 0 and GPU 8 illustrate another choice layered on top: those two GPUs hold different slices of one expert, rather than two whole experts. Now look at the lower-right portion: non-expert parameters, such as attention, are split into four tensor slices, and that arrangement is replicated across data-parallel groups.

You do not need to trace the diagram’s collectives yet. Its immediate lesson is a boundary on the simple rule: EP says how expert identities are distributed; it does not guarantee that every non-expert layer is fully replicated on every GPU. In the introductory two-rank layout it is; in this 16-GPU diagram, non-expert weights also have a separate tensor-parallel arrangement.



Takeaway

Data parallelism replicates model parameters while splitting training data. Expert parallelism distributes distinct experts among ranks, so a rank can participate in a larger logical MoE model without storing every expert. In the simplest layout, attention and the router remain replicated, while expert weights are sharded. More elaborate layouts can combine EP with replication or other kinds of sharding, so always identify the parameter, its owner ranks, and which copies represent the same weight.

Next, you will follow selected tokens to their expert-owning ranks and back through two-rank all-to-all exchanges.