Residual Connections and Vanishing Gradients

Introduction

In our previous lesson, we refined the Transformer's feed-forward network by replacing the simple ReLU with the more sophisticated GELU activation function. We now have the two core computational sub-layers of a Transformer block: multi-head attention and the position-wise FFN. However, as your experience with DevOps and system design might suggest, simply connecting components in a series isn't always a scalable or stable design. In deep learning, stacking dozens or even hundreds of these layers creates a network so deep that it becomes practically untrainable.

This lesson addresses this fundamental challenge by introducing residual connections, a surprisingly simple yet profoundly effective architectural pattern. We will explore why training very deep neural networks is so difficult, focusing on the mathematical roots of the infamous vanishing gradient problem. You will then learn how residual connections reformulate what each layer learns, creating "gradient highways" that allow training signals to flow unimpeded through the network. By the end of this lesson, you will be able to explain precisely how this mechanism mitigates vanishing gradients and enables the construction of the massive models that define the LLM landscape.

The Problem with Depth: Degradation and Vanishing Gradients

In theory, a deeper neural network should always be at least as powerful as a shallower one. A 50-layer network, for example, could simply learn to have its last 30 layers act as identity functions, perfectly replicating the behavior of a 20-layer network. In practice, however, early deep learning researchers observed a counter-intuitive phenomenon called the degradation problem: as networks got deeper, their performance on both the training and test sets would plateau and then get worse.

This wasn't an issue of overfitting, but of optimization. The deep networks were simply too difficult to train. The core reason for this lies in the mechanics of backpropagation.

As you'll recall from your calculus background, backpropagation relies on the chain rule to compute the gradient of the loss function with respect to the parameters of each layer. For a deep network with $L$ layers, the gradient for an early layer's parameters involves a long product of Jacobian matrices from all subsequent layers:

\frac{\partial L}{\partial θ _{l}} = \frac{\partial L}{\partial h _{L}} (k = l \prod L - 1 \frac{\partial h _{k + 1}}{\partial h _{k}}) \frac{\partial h _{l}}{\partial θ _{l}}

The term in the middle, $\prod \frac{\partial h _{k + 1}}{\partial h _{k}}$ , is the source of the problem. If the singular values of these Jacobian matrices are, on average, slightly less than 1, their product will shrink exponentially toward zero as $L$ increases. This is the vanishing gradient problem, where the training signal effectively disappears before it reaches the early layers. Conversely, if they are greater than 1, the product explodes, causing training instability.

To gain a more solid understanding of this issue, the following article provides an excellent mathematical explanation and visualization.

Residual Connections: The Gradient Highways Enabling Deep ...

This article provides a clear, mathematically-grounded explanation of the challenges faced when training deep networks, which directly motivates the need for residual connections.

Please read the section titled The Deep Network Problem. Pay close attention to the formula involving the product of Jacobian matrices and the accompanying simulation plot that shows how quickly gradients can vanish or explode. This will solidify your understanding of why depth is a problem.

The Solution: From Learning Transformations to Learning Adjustments

The insight behind residual connections, first introduced in ResNets, is to reframe what each layer learns. Instead of forcing a layer $F$ to learn a complete transformation from an input $x$ to an output $y$ , we ask it to learn only the residual (the difference or adjustment) needed to get from $x$ to $y$ .

The architecture is modified as follows:

A standard layer learns: $y = F (x)$
A residual layer learns: $y = F (x) + x$

The + x term is a direct connection that bypasses the layer's transformation. This is known as a skip connection or shortcut.

The fundamental building block of a residual network. The input `x` flows through the main transformation path `F(x)` (the weight layers) but is also added directly to the output via an "identity" skip connection.

This simple addition has a profound implication. Consider the degradation problem: if we add extra layers to a well-performing network, the ideal behavior for these new layers would be to act as identity mappings ( $y = x$ ).

For a standard layer to learn $y = x$ , it must adjust its weights to perfectly replicate the input. This is a surprisingly difficult task for a complex, non-linear function to learn.
For a residual block to learn $y = x$ , it only needs to learn $F (x) = 0$ . This is trivial! The network can easily achieve this by driving the weights of the layer towards zero.

The skip connection handles the identity mapping for free, leaving the main layers to focus only on learning the useful adjustments.

The Gradient Highway

The true power of residual connections is revealed during backpropagation. They create what are often called "gradient highways"—unobstructed paths for the gradient to flow back to early layers.

Let's look at the math. A deep network of residual blocks has an additive structure. If $x_{l}$ is the output of block $l$ , we have $x_{l + 1} = F_{l} (x_{l}) + x_{l}$ . If we unroll this for a deep network starting from input $x_{0}$ , the output of a much later layer $x_{L}$ is:

x_{L} = x_{0} + i = 0 \sum L - 1 F_{i} (x_{i})

The output of any deep layer is simply the original input plus the sum of all intermediate residual transformations. Now, let's use the chain rule to find the gradient of the loss $L$ with respect to an earlier layer's output, $x_{l}$ :

\frac{\partial L}{\partial x _{l}} = \frac{\partial L}{\partial x _{L}} \frac{\partial x _{L}}{\partial x _{l}} = \frac{\partial L}{\partial x _{L}} (\frac{\partial}{\partial x _{l}} (x_{l} + i = l \sum L - 1 F_{i} (x_{i})))

Applying the sum rule for differentiation gives:

\frac{\partial L}{\partial x _{l}} = \frac{\partial L}{\partial x _{L}} (1 + \frac{\partial}{\partial x _{l}} i = l \sum L - 1 F_{i} (x_{i}))

This equation is the key. The gradient signal $\frac{\partial L}{\partial x _{L}}$ is split into two paths:

The Direct Path: The + 1 term comes from the derivative of $x_{L}$ with respect to $x_{l}$ through the chain of skip connections. This path is a "highway" that passes the gradient back directly, without being attenuated by any weight layers.
The Residual Path: The second term, involving the sum of residuals, is the gradient that flows back through the traditional weighted layers. This path can still suffer from vanishing or exploding gradients.

Crucially, even if the residual path gradient vanishes entirely (i.e., the second term becomes zero), the + 1 from the direct path ensures that the gradient never becomes zero. The training signal always has a clean, direct route to the early layers.

The following resources will walk you through this concept, first with a short, animated explanation and then with a detailed article that formalizes the mathematics.

How Attention Residuals Work

This short video from Jia-Bin Huang offers a concise, animated walkthrough of the recursive expansion of residual connections and the resulting gradient flow.

Watch the entire video. The first part (initial explanation) shows the basic formula. The second part (gradient flow) demonstrates how the additive structure creates a "clean highway for gradients," which is the central idea of this lesson.

Now, let's dive deeper into the mathematical derivation and its implications with the same article from before.

Residual Connections: The Gradient Highways Enabling Deep ...

This reading will connect the high-level idea of residual blocks to the specific mathematical derivation of the gradient highway.

First, read the section The Residual Connection Solution to solidify your understanding of the y = F(x) + x formulation and the identity mapping advantage. Then, jump to the section titled The Gradient Highway Interpretation. This is the most critical part of the lesson. Focus on the "Unrolling the Residual Network" and "The Gradient Decomposition" subsections to see the derivation of the 1 + ... term in the gradient.

Residual Connections in the Transformer

The Transformer architecture heavily relies on this principle. Within each Transformer block, there are two main sub-layers: the multi-head self-attention mechanism and the position-wise feed-forward network. Each of these sub-layers is wrapped with its own residual connection, followed by layer normalization (which we will cover in the next lesson).

In the Transformer block (left for encoder, right for decoder), you can see the "Add & Norm" steps. The "Add" represents the residual connection where the input to the sub-layer is added to its output. This is done after both the Multi-Head Attention and the Feed Forward sub-layers.

The computation flow for a single Transformer block looks like this:

x_intermediate = Attention(x) + x (First residual connection)
x_output = FFN(x_intermediate) + x_intermediate (Second residual connection)

This design ensures that gradient highways exist through every single sub-layer of the entire model, allowing for the stable training of models with dozens or even hundreds of these blocks stacked on top of each other.

Conclusion

In this lesson, we've explored how a simple additive connection can solve one of the most significant barriers to building deep neural networks. You now understand that residual connections are not just an incidental feature but a foundational component that makes deep models like Transformers trainable.

Here are the key takeaways:

The Problem: Deep networks are difficult to optimize due to the degradation problem and the vanishing/exploding gradient problem, where the training signal is exponentially attenuated or amplified as it propagates through many layers.
The Solution: Residual connections reformulate layers to learn an adjustment ( $F (x)$ ) to the input, rather than a full transformation. The output becomes $y = F (x) + x$ .
The Mechanism: This creates a "gradient highway" during backpropagation. The gradient splits into two paths: one through the skip connection that contributes a gradient of exactly 1, and one through the weight layers.
The Result: Even if the gradient through the weight layers vanishes, the direct path guarantees that a non-zero gradient signal reaches the earliest layers, enabling stable training of extremely deep networks.
In Transformers: Residual connections are applied around both the multi-head attention and the feed-forward sub-layers in every block, ensuring robust gradient flow throughout the model.

We have now assembled the core computational logic of the Transformer block. However, we've repeatedly seen the "Add & Norm" label. While residual connections ensure gradients flow, they don't prevent the scale or distribution of the activations from fluctuating wildly between layers, which can cause its own instabilities. In our next lesson, we will investigate the "Norm" part of this duo: Layer Normalization.