DeepSeek-V3 MoE: Shared and Routed Experts

1. The Redundancy Problem in Standard MoE

First, let's consider why an innovation like DeepSeekMoE was necessary. Standard MoE architectures, while powerful, suffer from an inherent inefficiency.

As the article explains, when only a small number of large experts are activated for a given token, those experts are forced to handle all aspects of that token's processing. This includes fundamental, "common knowledge" tasks like understanding basic syntax or grammar, which are relevant to nearly all tokens.

This creates a significant problem: a large portion of each expert's capacity is wasted re-learning and storing the same redundant information. This leaves less capacity for the true goal of MoE: developing a diverse set of specialized experts.

2. DeepSeek's Solution: Shared and Routed Experts

DeepSeek's primary innovation is to solve this redundancy problem by partitioning the experts into two distinct categories. This new architecture is called DeepSeekMoE.

2.1. Shared Experts

As the name implies, shared experts are feed-forward networks that are always activated for every token.

• Role: Their job is to learn the broad, common knowledge required by all inputs. Think of this as the model's "core curriculum"—syntax, high-level semantics, and other fundamental patterns of language.
• Benefit: By dedicating a specific set of parameters to this common knowledge, the model frees up the other experts to specialize without having to waste their capacity on redundant information.

2.2. Routed Experts

These are the specialist networks that we are familiar with from the basic MoE architecture. They are conditionally activated based on the decisions of the router network.

• Role: Since the shared experts handle the fundamentals, these routed experts can focus entirely on learning niche, domain-specific knowledge. One expert might specialize in Python syntax, another in medical terminology, and another in historical facts.
• Benefit: This division of labor promotes much greater parameter efficiency and allows the model to develop a far more diverse and specialized set of skills.

The combination of these two expert types is the cornerstone of the DeepSeekMoE architecture. Let's look at the mathematical formulation from the technical report.

The output of a DeepSeekMoE layer, ${\mathbf{h}}_t^l$, for a given token $t$ at layer $l$ is a combination of three components:

Where:

• ${\mathbf{u}}_t^l$ is the input to the MoE layer (from the attention sub-layer).
• The first sum is over the $N_s$ shared experts, which are all applied to the input.
• The second sum is over the $N_r$ routed experts, where $g_{i,t}$ is the gating value from the router, which is non-zero only for the top-K selected experts.

The following video provides a clear walkthrough of this combined architecture.

3. Further Innovations in DeepSeekMoE

Beyond the shared/routed expert split, the DeepSeek architecture incorporates two other key ideas to maximize efficiency and performance.

3.1. Fine-Grained Experts

Instead of using a few very large experts, DeepSeek opts for a much larger number of smaller, "fine-grained" experts.

This "free lunch," as the article calls it, dramatically increases the model's flexibility and potential for specialization on a token-by-token basis, all while maintaining the same computational budget.

3.2. Auxiliary-Loss-Free Load Balancing

In our last lesson, we saw how auxiliary losses are used to prevent routing collapse. While effective, this adds another term to the loss function that the model must optimize, creating a trade-off between task performance and load balance.

DeepSeek pioneers a clever strategy to achieve load balance without a dedicated loss term.

This dynamic bias adjustment gently "nudges" the router towards a balanced state without interfering with the main optimization objective. As shown in the technical paper's ablation studies, this method allows for greater expert specialization compared to a rigid auxiliary loss.

4. The DeepSeek-V3 MoE Configuration

Let's make these concepts concrete by looking at the actual hyperparameters used in the DeepSeek-V3 model, as detailed in its technical report.

For most of its MoE layers, DeepSeek-V3 uses:

• Total Experts: 257 per layer.
• Shared Experts: 1
• Routed Experts: 256
• Activated Experts per Token (K): 8

This configuration highlights their design philosophy: one "generalist" shared expert handles the common workload, while a large pool of 256 fine-grained specialists are available for the router to choose from, with 8 being activated for any given token.

Conclusion

In this lesson, we deconstructed the highly effective DeepSeekMoE architecture.

Key Takeaways:

• Shared vs. Routed Experts: DeepSeek solves the expert redundancy problem by splitting FFNs into a shared expert (always on, for common knowledge) and a pool of routed experts (conditionally on, for specialized knowledge).
• Fine-Grained Specialization: It uses a large number of smaller experts, which massively increases the combinatorial space of expert selections for each token without increasing computational cost.
• Auxiliary-Loss-Free Balancing: Instead of a fixed auxiliary loss, DeepSeek dynamically adjusts a bias term for each expert's routing score, achieving better balance and allowing for more natural specialization.

Preview of the Next Lesson:

The design of MoE models, particularly the use of many experts with only a few activated per token, leads to a crucial distinction: the difference between a model's total parameters and its active parameters. In our next lesson, we will formalize this concept, quantify these two numbers for models like DeepSeek-V3, and understand why this distinction is the key to scaling models to trillions of parameters while keeping them computationally feasible.