SwiGLU: Gating for Enhanced FFN Capacity

1. Beyond ReLU: The Need for Better Activations

As a brief recap, a standard Feed-Forward Network (FFN) in a Transformer block consists of two linear layers with a non-linear activation function in between. For a long time, the go-to activation function was the Rectified Linear Unit (ReLU), defined as $\text{ReLU}(x)=\max (0,x)$.

While simple and effective, ReLU has limitations. Its gradient is not smooth—it's 0 for all negative inputs and 1 for all positive inputs, with a sharp "corner" at zero. This can sometimes hinder the training process.

Modern architectures have largely adopted a smoother, more nuanced activation called Swish (also known as SiLU, or Sigmoid Linear Unit).

The formula for Swish is elegantly simple:
$\text{Swish}(x)=x\cdot \sigma (x)$
where $\sigma (x)$ is the sigmoid function. This formula introduces a fascinating concept: the input x is multiplied by a "gated" version of itself ($\sigma (x)$), which squashes values to the range (0, 1). This is a form of self-gating.

This gating idea is the key to understanding SwiGLU.

2. The Gating Mechanism

Before we get to SwiGLU specifically, it's crucial to understand the general concept of a Gated Linear Unit (GLU). A standard FFN transforms its input with a single pathway. A GLU, however, splits the input into two parallel pathways. One pathway computes the "content" or "value" to be transformed, while the second pathway acts as a gate, dynamically controlling how much of the content from the first pathway is allowed to pass through.

The output of the gate pathway (typically squashed between 0 and 1) is multiplied element-wise with the output of the value pathway.

To build a solid intuition for this, the following article provides an excellent conceptual overview.

As the reading explains, this multiplicative interaction is fundamentally more expressive than the purely additive interactions in a standard FFN. By allowing the network to learn not just what to transform but also how much of that transformation to use for any given input, we give it a more powerful tool for learning complex patterns.

3. SwiGLU: Combining Swish and Gating

Now we can combine the two concepts we've introduced: the Swish activation and the GLU architecture. The result is SwiGLU, which has become the de facto standard in modern LLMs.

The architecture is best understood visually.

The formula for SwiGLU is:
$\text{SwiGLU}(x,W,V)=\text{Swish}(xW)\otimes (xV)$
Here, the input $x$ is projected through two separate weight matrices, $W$ and $V$. The first projection, $xW$, is passed through the Swish activation. The second projection, $xV$, remains linear. The two results are then multiplied together element-wise ($\otimes$).

To get a complete understanding, including a step-by-step numerical example that satisfies your preference for practical applications, please continue with the same article.

For a supplementary verbal explanation, the following video segment is also excellent.

4. How Gating Increases FFN Capacity

You've now seen what SwiGLU is, but the learning outcome also asks how it increases the capacity of the FFN.

1. Increased Expressivity: The multiplicative interaction between the two pathways allows the network to model more complex, second-order relationships between features that a standard additive FFN cannot.
2. Parameter Efficiency: At first glance, SwiGLU seems more expensive because it uses three weight matrices ($W$, $V$, and the output projection $W_2$) instead of two. However, as noted in the "Transformer Design Guide" resource, models compensate for this by reducing the FFN's hidden dimension ($d_{ff}$). For example, instead of the standard $d_{ff}=4\times d_{model}$, they use $d_{ff}=\frac{2}{3}\times (4\times d_{model})$. This keeps the total parameter count roughly the same while delivering superior performance, demonstrating a more efficient use of parameters. The gating mechanism is so powerful that the FFN can be "thinner" and still outperform a "wider" FFN with a simpler activation function.

Conclusion

In this lesson, we've explored the SwiGLU activation function, a key component that distinguishes modern Transformers from their predecessors.

Key Takeaways:

• SwiGLU replaces older activation functions like ReLU in the Feed-Forward Network (FFN) sub-layer.
• It is a type of Gated Linear Unit (GLU), which uses a gating mechanism to dynamically control information flow.
• The input is processed through two parallel linear projections. One is passed through the Swish activation ($x\cdot \sigma (x)$), and the other remains linear. The results are multiplied element-wise.
• This multiplicative interaction increases the FFN's capacity and expressivity, allowing it to learn more complex feature relationships.
• Despite using an extra weight matrix, SwiGLU allows for a smaller hidden dimension, resulting in similar or better performance for a comparable number of parameters.

Preview of the Next Lesson:
We have now covered two crucial pieces of the modern Transformer block: RMSNorm and SwiGLU. In the next lesson, we will examine the third and final piece from this foundational module: Rotary Positional Embeddings (RoPE). You will learn how this clever technique encodes positional information by rotating feature vectors in the complex plane, a method that is fundamental to the long-context capabilities of today's LLMs.