Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 5

Estimating Expert-Layer Arithmetic from Width and Expert Selection

Recommended resources

Hello. Last lesson counted two kinds of parameters: every expert stored in the model, and the selected experts on one token’s path. This lesson asks a different question: how much arithmetic does that path perform? We will estimate the forward-pass work of an expert layer from its width and the number of experts selected, then compare it with a dense feed-forward layer. The distinction matters when you later choose a small MoE architecture to implement and train. Allow about 40 minutes.


From expert width to arithmetic

A routed expert is usually a feed-forward network. For this lesson, assume the two-projection design used in the previous parameter ledger: a token representation of width is projected to an inner width , passed through an activation, and projected back to width .

Watch the feed-forward portion of Mixture of Experts (MoE), Visually Explained by Jia-Bin Huang. Focus on the dimensions of the up and down projections; the lesson below turns those dimensions into an operation count.

Mixture of Experts (MoE), Visually Explained

Watch the feed-forward explanation in Jia-Bin Huang's Mixture of Experts (MoE), Visually Explained to connect each expert's arithmetic to two ordinary matrix products.

Watch the projections. The up projection creates the inner-width vector, and the down projection returns it to the model width. Those are the two large arithmetic terms we will count.

The up projection forms outputs, each using a dot product of length : approximately multiply-accumulate operations, or MACs. The down projection forms outputs from vectors of length : another MACs. Thus, for one token through one expert,

A MAC combines a multiplication and an addition. Under the common convention of counting those as two floating-point operations, the estimate is

State the unit whenever you report a number. A figure of “two million operations” is ambiguous if one person means MACs and another means FLOPs.

These leading terms count the weight-matrix products. Bias additions, the activation function, and any gating of the final expert output add work, but do not change the usefulness of MACs as a first estimate when the matrix products dominate. The formula must change if the expert architecture changes: a three-projection gated expert, for instance, has another substantial matrix product.



What sparsity saves—and what it still computes

Read the dense-versus-MoE comparison in ApX Machine Learning’s Analyzing Parameter vs. FLOPs Trade-offs. It supplies a larger numerical example of the same two-projection structure.

Analyzing Parameter vs. FLOPs Trade-offs - ApX Machine Learning

Read the dense and MoE feed-forward comparison from ApX Machine Learning to see how expert width and top-k selection can be chosen to match a dense layer's leading arithmetic.

Under “A Concrete Comparison,” begin at the “Dense FFN Block” subsection, where the author says “A dense FFN block typically consists of two linear layers.” Read the dense calculation. Then continue through the “MoE FFN Block” subsection, beginning “Now, let's replace this dense block with an MoE layer,” to its final FLOPs bullet; follow the sparse comparison. Focus on why two selected experts of half the dense inner width require roughly the same matrix-product arithmetic as one dense FFN.

Suppose a layer stores equally sized experts and selects of them for a token, with every selected contribution actually evaluated. The stored expert weights scale with , but the leading expert arithmetic per token scales with :

The parameter expression omits biases; the arithmetic expression counts forward matrix products. Their similar appearance does not make parameters and operations interchangeable. Stored weights are a quantity in the model. MACs are work performed on a particular input. A selected weight can be used for many tokens, while an unselected expert’s weights still occupy storage.

The router also performs arithmetic before any expert is selected. If it scores all experts with a width- linear projection, its leading matrix-product cost is approximately MACs per token. Softmax and top- selection add further work. A useful first-order layer estimate is therefore

plus activation, bias, gating, selection, and combination overhead. This is an expert-layer estimate, not a whole-transformer estimate: it excludes attention and other shared computations. Increasing with fixed leaves the leading expert term unchanged, but makes the router’s score projection larger. “Fixed compute” is consequently an approximation, not an exact property of adding experts.



Revisit the previous model with a compute ledger

The previous lesson used , eight routed experts per MoE layer, inner expert width , and top-two selection. A dense feed-forward sublayer in that same model had inner width . Using the two-projection estimate:

The token visits two experts, so the selected expert work is MACs. The router scores eight experts at about MACs. Compare that with the dense sublayer:

Forward matrix-product work for one token Approximate MACs
One width- expert 1,048,576
Two selected experts 2,097,152
Router score projection to eight experts 4,096
Dense FFN with inner width 2,097,152

The selected expert matrix products match the dense FFN’s because two experts of width collectively perform the leading work of one width- FFN. Including the router projection makes the MoE estimate MACs, before the smaller operations noted above. In the two-FLOPs-per-MAC convention, the selected expert products alone are approximately FLOPs.

Yet the stored parameter counts tell a different story. Last lesson’s eight experts contained 8,400,896 parameters, and their router contained another 4,104. The dense FFN contained 2,099,712 parameters. The MoE layer stores roughly four times as many weights in its expert bank while evaluating only two experts for this token. In this example, more stored parameters coexist with roughly matched leading forward arithmetic.

For a batch of tokens with no dropped assignments, there are expert evaluations. If the experts have equal widths, their useful mathematical work sums to approximately MACs. Those evaluations need not be distributed evenly: one popular expert may receive many tokens and become a bottleneck even when the per-token arithmetic estimate has not changed.



Choosing a fair comparison—and knowing its limits

For a dense FFN of inner width and an MoE whose experts have inner width , matching their leading forward matrix-product work calls for

With the same model width , this reduces to

That is why the worked example used half-width experts for top-two routing. It is also why simply replacing a dense FFN with two full-width experts would not preserve its expert arithmetic: it would approximately double that part of the forward work. Conversely, increasing the number of available experts while keeping their width and fixed mainly increases stored weights, not the number of expert matrix products performed for one token.

The approximation is deliberately narrow. Read the caveats near the end of the ApX article, while keeping the distinction between arithmetic count and elapsed time in view.

Analyzing Parameter vs. FLOPs Trade-offs - ApX Machine Learning

Return to the ApX Machine Learning article for its practical caveats: matched arithmetic does not ensure matched runtime.

In “Important Caveats,” start at “While the parameter-FLOPs trade-off is powerful, it is not a free lunch.” Read the three caveats through the end of “Load Imbalance.” Focus on communication, weight storage, and uneven expert workloads. Treat the discussion of balance as a runtime warning: in a dropless implementation, skew need not change the total useful expert MACs, though it can make the step slower.

A FLOP estimate cannot by itself predict training speed. Dispatching token representations, moving data between devices, and combining outputs consume time; implementations may also spend work on padded expert slots. If a capacity policy drops an expert contribution, the token performs less expert arithmetic than the no-drop formula assumed. These cases will be measured more precisely in later modules.

Finally, our equations estimate forward matrix products. Training also computes backward-pass derivatives and updates weights, so its work exceeds the forward figure. As a rough guide for the large linear products, the backward pass includes products for input and weight gradients in addition to the forward product; this is not an exact step-time multiplier once routing, memory movement, and other operations are included.



Takeaway

For a two-projection expert of model width and inner width , estimate MACs per token per selected expert, or approximately FLOPs under the two-FLOPs-per-MAC convention. Top- routing multiplies that leading expert work by , while the bank’s stored weights scale with the available expert count . Add router work separately, and do not mistake matched FLOPs for matched training time.

Next, we will examine which router and expert weights receive task-loss gradients through hard top- selection—a further distinction between weights stored, computations performed, and weights updated by a token.