Gradients: Exploding and Vanishing

Recommended resources

The freely available resources this lesson will direct you to.

Introduction

In our previous lesson, we derived the complete backpropagation algorithm, culminating in the recursive equation for propagating the error signal $δ^{(l)}$ backward through the network:

δ^{(l)} = ((W^{(l + 1)})^{T} δ^{(l + 1)}) ⊙ g^{'} (z^{(l)})

As we briefly noted, this process involves a long chain of matrix multiplications. The gradient for a parameter in an early layer, say layer 1, is influenced by the product of all the transposed weight matrices that come after it: $(W^{(L)})^{T} (W^{(L - 1)})^{T} \dots (W^{(2)})^{T}$ .

This lesson dives into the profound and often problematic consequences of this multiplicative chain. We will investigate what happens when the components of these matrices and the activation derivatives are consistently small or large. This leads directly to two of the most fundamental obstacles in training deep neural networks: the vanishing and exploding gradient problems. Understanding these issues is not just theoretical; it's a crucial diagnostic skill for any practitioner and a primary motivator for many of the architectural innovations we will study later.

The Vanishing Gradient Problem

Let's begin with the more common and insidious of the two problems: vanishing gradients. The core issue arises from the repeated multiplication of numbers that are less than one.

Consider a deep network. The gradient of the loss $L$ with respect to a weight in an early layer depends on a product of many Jacobian matrices, each involving a weight matrix and the derivative of an activation function.

To build intuition, we'll watch a short, clear explanation that focuses on the sigmoid activation function.

The Fundamental Problem with Neural Networks - Vanishing Gradients

The video "The Fundamental Problem with Neural Networks" by ritvikmath provides a great first-principles explanation of this phenomenon.

Please watch from the breakdown of the chain rule, where the presenter isolates the derivatives of the activation function. Then, continue through the explanation of why the product of these small numbers leads to a vanishing gradient. Focus on why the derivative of the sigmoid function is always less than or equal to 1/4.

As the video explains, the derivative of the sigmoid function, $σ^{'} (z) = σ (z) (1 - σ (z))$ , has a maximum value of 0.25. This means every time we backpropagate through a sigmoid activation, we multiply the gradient by a number no larger than 0.25. In a network with many layers, the gradient in the early layers will be the result of multiplying many such small numbers together.

Question: If you multiply 0.25 by itself $k$ times, the result is $(0.25)^{k}$ . What happens to this value as $k$ (the number of layers the gradient has to pass through) becomes large? What does this imply for the magnitude of the gradients in the earliest layers of a deep network?

Answer

The value $(0.25)^k$ decays exponentially towards zero as $k$ increases. This implies that for deep networks, the gradients reaching the early layers can become infinitesimally small. If the gradients are effectively zero, the weight updates for those layers will also be zero, and the layers will stop learning.

This is the essence of the vanishing gradient problem. The error signal from the loss function shrinks exponentially as it propagates backward, rendering the early layers untrainable.

Saturation: The Practical Cause

The mathematical cause is the product of small derivatives. The practical cause, which you can directly observe during training, is often neuron saturation. An activation function like sigmoid or tanh is said to be "saturated" when its input is a large positive or negative number, causing the output to be very close to its asymptotic values (1 or 0 for sigmoid; 1 or -1 for tanh).

The sigmoid function $\sigma(x)$ (magenta) and its derivative $\sigma'(x)$ (black). In the "saturating" regions where the sigmoid is flat (for large positive or negative x), its derivative is close to zero.

When a neuron is in a saturated state, its local gradient is nearly zero. This effectively "closes the gate" for the gradient to flow through during backpropagation. If many neurons in a layer are saturated, the gradient signal for the entire network can be severely attenuated or "killed".

Andrej Karpathy gives an excellent demonstration of this effect. He visualizes the distribution of activations in a hidden layer and shows how improper initialization can lead to widespread saturation.

Building makemore Part 3: Activations & Gradients, BatchNorm

This segment from Andrej Karpathy's "Building makemore Part 3" provides a fantastic practical look at why you should be "sweating" if you see saturated activations.

Watch from his analysis of the hidden state. Pay attention to the histogram of the tanh activations, how he connects the saturated values of -1 and 1 to the local gradient 1 - t^2 becoming zero, and his analogy of the gradient being "killed" or "destroyed".

Even the popular ReLU activation function, $g (z) = max (0, z)$ , is not entirely immune. While its gradient is 1 for positive inputs (avoiding the shrinking problem), it is exactly 0 for all negative inputs. If a neuron's weights are updated such that its pre-activation $z$ is consistently negative for all inputs in the training set, that neuron will always output 0 and have a gradient of 0. It becomes a "dead ReLU" and effectively ceases to participate in learning. Karpathy also discusses this phenomenon in the next segment of the video (LINK, from the discussion on other activations).

The Exploding Gradient Problem

The opposite problem occurs when the repeated multiplications involve numbers significantly greater than one. This is less common but can be even more destructive.

If the weights in the network matrices $W^{(l)}$ are initialized to be large, the repeated multiplication during backpropagation can cause the gradient to grow exponentially. This leads to an exploding gradient.

A conceptual illustration of vanishing and exploding gradients. Vanishing gradients shrink to zero in the early layers (left), while exploding gradients grow uncontrollably (right).

An exploding gradient results in a massive update to the weights, often so large that it overwrites any previous learning. The parameter values can become NaN (Not a Number) or infinity, and the training process completely destabilizes. Geometrically, this phenomenon corresponds to encountering extremely steep regions, or "cliffs", in the loss landscape. A single gradient step can overshoot the optimal region by a huge margin.

Lecture 15: Exploding and Vanishing Gradients

This excerpt from lecture notes by Roger Grosse at the University of Toronto provides a good intuition for this landscape.

Read the section on "Iterated functions" starting from the qualitative perspective. Then, skim down to Figure 4 and the discussion of "cliffs" in the optimization landscape, which provides a powerful visual metaphor for what happens during gradient explosion. The text you should focus on is from this paragraph.

A Deeper Analysis: Jacobians and Eigenvalues

From a first-principles perspective, both phenomena can be analyzed rigorously by examining the product of the Jacobian matrices from the backpropagation rule. The error signal propagation is a linear transformation:

δ^{(l)} = \dots [((W^{(l + 2)})^{T} ⊙ g^{'} (z^{(l + 1)})) ((W^{(l + 1)})^{T} ⊙ g^{'} (z^{(l)}))] \dots δ^{(L)}

Let's simplify by considering a linear network where $g^{'} (z) = 1$ . The backpropagated gradient signal after $k$ layers would be proportional to the product of transposed weight matrices: $(W^{(l + k)})^{T} \dots (W^{(l + 1)})^{T}$ . The behavior of this product—whether it shrinks or grows—is determined by the eigenvalues of these matrices.

A more formal treatment, often presented in the context of Recurrent Neural Networks (RNNs) where this problem is particularly acute, proves the following:

If the largest eigenvalue (in magnitude, also called the spectral radius) of the weight matrix is consistently less than 1, the norm of the gradient will decrease exponentially as it propagates backward (vanish).
If the largest eigenvalue is consistently greater than 1, the norm of the gradient will increase exponentially (explode).

This gives us the fundamental mathematical condition governing gradient stability. The paper "On the difficulty of training Recurrent Neural Networks" is a foundational text that explores this.

[PDF] On the difficulty of training Recurrent Neural Networks - arXiv.org

This is a seminal research paper by Pascanu, Mikolov, and Bengio. Reading academic papers is a core skill, and this is a great one to start with. Don't worry about understanding every detail; focus on the high-level argument.

First, read Section 2.1, "The mechanics", which presents the mathematical argument for vanishing and exploding gradients based on the Jacobian product and introduces the condition based on the largest eigenvalue \lambda_1. Then, look at the supplementary section at the end of the paper, the analytical proof, to see how power iteration and eigendecomposition are used to formalize this intuition.

Diagnosing and Mitigating Gradient Problems

Given your interest in systems and performance, it's crucial to know how to detect these issues. Plotting the norm of the gradients for each layer during training is a direct way to see if they are shrinking or growing out of control.

Andrej Karpathy demonstrates a suite of powerful diagnostic tools for exactly this purpose.

Building makemore Part 3: Activations & Gradients, BatchNorm

Let's return to Karpathy's video to see how a practitioner debugs these issues.

First, watch from the section on diagnostic plots. He visualizes histograms of both activations and gradients layer by layer, showing how to spot distributions that are shrinking to zero or becoming too saturated. Then, watch from the 'update to data ratio' section. This is a brilliant heuristic that relates the magnitude of the parameter update to the magnitude of the parameters themselves, giving you a single number (\approx 10^{-3}) to aim for.

While we will cover the solutions in detail in the next module, it's useful to have a high-level map now:

Exploding Gradients: The most common fix is gradient clipping. If the norm of the gradient vector exceeds a certain threshold, it is scaled down. This is a simple, effective, and widely used technique.
Vanishing Gradients: This is a harder problem that has driven major architectural innovations.
- Activation Functions: Using ReLU instead of sigmoid/tanh alleviates the problem for positive inputs.
- Weight Initialization: Schemes like Xavier/Glorot and He initialization are designed to set the initial weights such that the variance of activations and gradients is preserved across layers, keeping the eigenvalues of the weight matrices close to 1.
- Architectural Changes: More advanced solutions involve changing the network structure itself. Residual Networks (ResNets) introduce "skip connections" that provide a direct path for the gradient to flow, bypassing some layers. Gated mechanisms in LSTMs and GRUs were specifically designed to combat this in sequential data.

Conclusion

You have now explored the critical challenges of vanishing and exploding gradients from multiple angles: intuitive, practical, geometric, and formal mathematical. This instability is not a minor detail but a central feature of deep learning that has shaped the field's research trajectory.

Key Takeaways:

Core Cause: The backpropagation algorithm involves a long product of Jacobian matrices. The magnitude of gradients in early layers is exponentially sensitive to the properties of these matrices.
Vanishing Gradients: Occur when derivatives (from activations like sigmoid/tanh or due to small weights) are consistently less than 1. The gradient signal shrinks to zero, and early layers stop learning. This is often caused by neuron saturation.
Exploding Gradients: Occur when weights are large, causing derivatives to be consistently greater than 1. The gradient grows uncontrollably, destabilizing training and creating "cliffs" in the loss landscape.
Formal Condition: The stability of the gradient is governed by the eigenvalues of the weight matrices. A spectral radius greater than 1 leads to explosion, while less than 1 leads to vanishing.
Mitigation: Key strategies, which we'll detail later, include gradient clipping, better activation functions (ReLU), careful weight initialization, and architectural changes (ResNets, LSTMs).

Preview of the Next Lesson:

We have now assembled all the theoretical components: the forward pass, the backward pass (backpropagation), and an understanding of the potential instabilities. In the next module, "Training Deep Neural Networks," we will put it all together. You will implement the full forward and backward passes from scratch in Python and build a complete training loop. Our first topic will be a deep dive into the very first line of defense against unstable gradients: weight initialization strategies.