MoE Parameters: Total vs. Active Per Token

1. What Does "8x7B" Actually Mean? The Core Distinction

The naming convention for MoE models, like "Mixtral 8x7B," can be a source of confusion. It hints at the core architectural principle but doesn't tell the whole story. The central idea of MoE is to increase the model's capacity (its knowledge) without proportionally increasing the work needed for any single task. This leads to two different ways of measuring model "size."

As the article explains, the two key terms are:

1. Total Parameters: This is the full count of all weights in the model. It includes the shared components (like attention layers and embeddings) and the weights of all experts in every MoE layer, regardless of whether they are used for a particular token.

   • Impact: Total parameters determine the model's storage size and memory footprint. To run a model, you must have enough VRAM to load all of its parameters.

2. Active Parameters: This is the number of parameters that are actually used to process a single token during a forward pass. It includes the shared components plus the weights of only the activated experts (e.g., the top-K experts chosen by the router).

   • Impact: Active parameters determine the model's computational cost (FLOPs). The speed of training and inference is proportional to the number of active parameters, not the total.

This separation is the "magic" of MoE. You get the benefit of a vast repository of knowledge (total parameters) with the inference speed of a much smaller model (active parameters).

2. A Concrete Example: Deconstructing Mixtral 8x7B

Let's make this tangible by examining one of the most famous MoE models, Mixtral 8x7B.

The name suggests 8 experts of 7 billion parameters each, but the reality is more nuanced. The following video provides a clear visual breakdown of how total and active parameters are counted in this architecture.

Now, let's look at a static diagram that provides the specific numbers for the Mixtral 8x7B decoder block.

Let's analyze the numbers from the diagram:

• Expert Layers (MoE):

  • The model has 8 experts, each with about 5.6B parameters. The total for all experts is $8\times 5.6B\approx 45B$.
  • However, the router only activates 2 of these experts per token.
  • So, the active expert parameters are $2\times 5.6B=11.2B$.

• Shared Layers (Always Active):

  • Attention Layers: 1.2B
  • Embeddings & LM Head: ~0.26B
  • Router: A negligible amount.
  • Total Shared: $\approx 1.5B$

Putting it together:

• Total Parameters: $45B_{\text{experts}}+1.5B_{\text{shared}}\approx 47B$. This is the size of the model on disk and the memory required to load it.
• Active Parameters: $11.2B_{\text{active experts}}+1.5B_{\text{shared}}\approx 13B$. This is the effective "size" of the model in terms of computational cost for inference.

So, Mixtral 8x7B provides the knowledge capacity of a ~47B parameter model but runs at the speed of a ~13B parameter dense model.

3. Why the Distinction Matters: Compute vs. Memory

The difference between total and active parameters directly translates to hardware performance, creating a trade-off between memory capacity and computational throughput.

The key insight is:

• Memory cost scales with the parameters you need to load from high-bandwidth memory (HBM) to on-chip SRAM. For MoE models, this involves the shared parameters and the parameters of the active experts for the current batch of tokens.
• Compute cost scales with the FLOPs required for the matrix multiplications, which again, only involves the shared parameters and the active experts.

A model is memory-bound when the time spent moving weights from memory is greater than the time spent doing computations. As the Cerebras article shows, many MoE deployments on GPUs are memory-bound because, while the compute is low (due to few active params), you still have to fetch the weights for the activated experts from a massive pool of total parameters.

This trade-off is precisely why MoE models are so compelling. They allow you to push the boundaries of model knowledge (total params) far beyond what would be computationally feasible if all parameters were active for every token.

4. A Comparative Look at Modern MoE Models

Let's solidify our understanding by comparing the parameter counts of several state-of-the-art MoE models.

The table above gives us a great at-a-glance comparison. Here is a summary including other models discussed in the provided resources:

As you can see, all these models operate with a small fraction of their total parameters active at any given time. DeepSeek-V3 is particularly aggressive, using only about 5.5% of its total parameters per token, allowing it to pack an enormous 671 billion parameters of knowledge into a computationally efficient framework.

Conclusion

In this lesson, we've demystified the parameter counts of Mixture-of-Experts models.

Key Takeaways:

• Total Parameters represent the entire set of weights in a model, including all experts. This metric dictates the model's memory footprint and storage requirements.
• Active Parameters represent the subset of weights (shared layers + activated experts) used to process a single token. This metric dictates the model's computational cost (FLOPs) and thus its inference speed.
• The core value proposition of MoE is to dramatically increase the total parameter count without a proportional increase in active parameters, leading to models that are both knowledgeable and computationally efficient.

Preview of the Next Module:

We have now concluded our foundational exploration of modern Transformer architectures. You understand how they are constructed (RoPE, SwiGLU), made efficient (MLA), and scaled to immense sizes (MoE).

However, a pre-trained model is just a powerful next-token predictor. To make it a useful assistant that can follow instructions, answer questions, or control a robot, it must be aligned. The dominant paradigm for alignment is Reinforcement Learning (RL).

In our next module, Reinforcement Learning Foundations, we will start from the very beginning. We will build the conceptual framework needed to understand advanced alignment techniques like PPO and GRPO, beginning with the fundamental building block of all RL: the Markov Decision Process.