Skip to main content

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 4

Calculating Total and Per-Token Active Parameters in MoE Transformers

Recommended resources

Hello. A dense transformer applies the same feed-forward weights to every token. In a sparse mixture-of-experts (MoE) transformer, each token uses only a few feed-forward experts, but the model still stores all of them. The resulting parameter counts answer two different questions: How many weights does the model contain? and How many weights lie on one token’s routed path?

This lesson builds on the routing and top- selection concepts in this module. You will set a counting convention, calculate both counts for a transformer with several MoE layers, and see why neither count alone tells you how fast the model runs. Allow about 40 minutes.


Two counts, one model

Watch the short parameter-count segment of A Visual Guide to Mixture of Experts (MoE) in LLMs by Maarten Grootendorst. Its breakdown is useful for spotting the components that remain shared when feed-forward layers become sparse.

A Visual Guide to Mixture of Experts (MoE) in LLMs

Watch the distinction between stored and active weights, then the breakdown of a model into embeddings, attention, routers, experts, and an output head.

Watch stored versus active for the basic distinction. Then watch the model breakdown, focusing on which components are counted once and which expert weights are selected per token. Treat the model name as a label, not as an instruction to multiply eight by seven billion: the segment separates expert weights from shared weights.

For a particular token, an expert that was not selected is inactive for that token, not absent from the model. Its weights still belong to the checkpoint. Another token in the same batch might select it. Conversely, attention weights and other shared components do not disappear from the active count merely because the feed-forward path is sparse.

That gives us the central distinction:

  • Total parameters count the model’s unique stored trainable weights.
  • Per-token active parameters count the weights assigned to that token’s computation under a stated architectural convention.

The phrase under a stated convention matters. For example, an embedding lookup accesses one row, yet architectural active-parameter reports commonly include the embedding matrix as a shared component. Active parameters are therefore a useful model-size summary, not a literal count of every scalar read during one token’s execution.



Establish a counting convention

Read the small example in Sebastian Raschka’s Why MoE models have fewer active parameters. It introduces shared weights explicitly, then extends the idea to multiple MoE layers and shared experts.

Why MoE models have fewer active parameters

Read Sebastian Raschka’s parameter-count example to anchor the distinction between shared and routed weights before applying it to an entire transformer.

In “A small parameter-count example,” read the example, from the stated shared-parameter subtotal through the explanation of why the model-wide ratio differs from the expert-only ratio. Continue through the paragraphs beginning “The same accounting applies across all MoE layers,” ending with the statement that shared experts belong in both counts: multiple layers. Under “Active parameters are a proxy for compute,” read the compute caveat. Focus on what is counted, rather than trying to estimate arithmetic yet.

Here is the convention we will use throughout the lesson:

  1. Count each unique stored weight once in the total. If an embedding and output head tie the same weights, do not count that matrix twice.
  2. Put all weights shared across tokens’ routed paths into : attention, embeddings, normalization, dense feed-forward layers, output head, all MoE routers, and any experts that run for every token. Count those same weights in the per-token active subtotal. This is an architectural convention; it does not claim every embedding row is multiplied for each token.
  3. Assume each MoE layer has its own, untied expert bank. For layer , let be its number of routed experts, the number of distinct experts successfully selected per token, and the parameters in each equally sized expert.
  4. Ignore dropped expert assignments for now: every token reaches exactly experts in layer . Capacity-limited routing will require us to revisit that assumption later.

Under this convention, the whole-transformer counts are

The router is in because it scores the expert choices for every token; selecting two experts does not mean that only two columns of the router exist or are counted. Softmax, top- selection, and multiplying by gate values add operations, but no additional learned weights beyond the router and experts already counted.

For a single MoE layer with eight equal experts and top-two routing, the expert-bank contribution is total and active per token. Its expert-only total-to-active ratio is four. The whole model’s ratio is instead , which is smaller whenever is positive.



Work through a transformer-wide ledger

Consider a six-block transformer with four MoE feed-forward sublayers and two ordinary dense feed-forward sublayers. Suppose an audit of its other shared components—embeddings, attention, normalization, output head, and any remaining shared weights—gives 20,000,000 parameters. We will count the two dense sublayers and four routers separately, then add them to . This avoids hiding their contributions in an unexplained subtotal.

All blocks have model width . Each ordinary dense feed-forward sublayer has inner width , using an up projection and a down projection, both with biases. Its parameter count is

Each routed expert uses the same two-projection structure but has inner width :

Each MoE layer has experts and selects per token. Its router maps a width- token representation to eight scores. With a bias for each score, one router contains

parameters. Notice that all eight score outputs belong to the router even when only two experts are subsequently evaluated.

Component Total parameters Active parameters per token
Other shared components 20,000,000 20,000,000
Two ordinary dense sublayers 4,199,424 4,199,424
Four routers 16,416 16,416
Four banks of eight routed experts 33,603,584 8,400,896
Whole transformer 57,819,424 32,616,736

The first three rows form . The expert rows differ because the checkpoint contains separate experts, whereas a token follows expert paths—two in each MoE layer. Experts in different layers have different weights, so you add their counts even if a token happens to select the same expert index in successive layers.

Only about 56.4% of this model’s parameters are active per token under our convention. That percentage is not : the latter describes only the routed expert banks. Most of the difference comes from shared components that remain active on every token’s path.

The expert dimensions matter as much as the number of experts. Our calculation used two biased projections. An expert built with three learned projections, as in some gated feed-forward designs, needs a different . Before comparing published counts, identify the actual expert architecture rather than assuming every feed-forward network has the same shape.



What changes when the architecture changes?

Suppose we keep the worked example’s expert widths and top-two selection but increase each MoE layer from eight experts to sixteen. Each additional expert is stored, so the total rises substantially. The active expert subtotal stays at , because each token still visits two experts per layer. Active parameters are not quite constant, however: each router now has sixteen score outputs, so its weight matrix and bias grow. Holding everything else fixed, the new counts would be 91,439,424 total and 32,633,152 active per token.

If instead we keep eight experts but change from top-two to top-one routing, the total count stays 57,819,424. The active count falls to 28,416,288, because each MoE layer now evaluates one fewer expert per token. These comparisons show the purpose of stating both and : one governs the size of the stored expert bank; the other governs how much of that bank appears on a token’s path.

A further convention arises when an MoE layer includes a shared expert that runs for every token in addition to its top- routed experts. Count that expert once in the total and once in the active count; under our notation, its weights belong in , not in .

To see why reported model sizes need such qualifications, read the brief naming discussion in the same resource.

Why MoE models have fewer active parameters

Return to Raschka’s discussion of reported MoE sizes to see which counting choices must be checked before comparing two model labels.

Under “How to read reported MoE sizes,” start with “MoE model names often report total and active parameters together.” Read reported size conventions through the checklist of shared experts, dense layers, embeddings, and output heads. Focus on the questions you would ask about a claimed active count, not on memorizing particular model sizes.

Finally, neither number is a speed measurement. Total parameters matter for storing or distributing the weights. Active parameters give a first approximation of the size of one token’s weight-dependent path, but they do not capture attention arithmetic, router scoring, token dispatch, memory movement, or communication between devices. A batch may collectively use every expert even though each token uses only two. Those distinctions become important when estimating computation and, later, training an expert-parallel system.



Takeaway

To calculate MoE transformer parameters, first declare what counts as shared and count tied weights only once. Then add every routed expert to the total, but only each token’s selected experts to the per-token active count. Keep routers, dense layers, and always-run shared experts in both subtotals. The gap between total and active parameters describes sparsity; it does not, by itself, determine runtime.

The next lesson turns this ledger into an estimate of expert-layer arithmetic per token, making the difference between parameter count and computation more precise.