Understanding Vanishing and Exploding Gradients in RNNs

How Gradients Flow in RNNs: Backpropagation Through Time

To understand why gradients can misbehave, we first need to visualize how an RNN is trained. The process is called Backpropagation Through Time (BPTT). You can think of it as "unrolling" the recurrent network for each token in the sequence. An RNN processing a 10-word sentence can be visualized as a 10-layer feed-forward network, with the crucial difference that the weights are shared across all 10 layers (timesteps).

When we calculate the loss at the end of the sequence, the error signal must be propagated backward through this entire unrolled structure to update the weights.

For an intuitive walkthrough of how this unrolling works and how information flows, the following video is an excellent starting point.

The Mathematical Origin of the Problem

Now, let's look at the mathematics behind BPTT. Your goal of deep mastery requires us to go beyond intuition and understand the mechanism precisely. The gradient signal propagating from a later timestep k to an earlier timestep k-1 depends on the derivative of the hidden state update function.

Recall the basic RNN recurrence equation:
$h_t=\tanh (W_{hh}{h}_{t-1}+W_{xh}{x}_t+b_h)$

Using the chain rule, the gradient of the loss L with respect to the weights W involves a sum of contributions from each timestep. The critical part of this calculation is how the hidden state at a late timestep T is affected by a hidden state at an early timestep t. This is expressed as a product of Jacobian matrices:

$\frac{\partial h_T}{\partial h_t}={\prod }_{k=t+1}^T\frac{\partial h_k}{\partial h_{k-1}}$

Each Jacobian matrix in this product breaks down into two key components:

$\frac{\partial h_k}{\partial h_{k-1}}=\text{diag}({\tanh }^{\prime }(z_k))\cdot W_{hh}$

Where:

• W_hh is the recurrent weight matrix.
• tanh' is the derivative of the tanh activation function, which is always between 0 and 1.
• diag(...) creates a diagonal matrix from the vector of tanh derivatives.

This product of matrices is the heart of the problem. We are repeatedly multiplying the same kind of transformation over and over. Linear algebra tells us that the behavior of repeated matrix multiplication is dominated by the matrix's eigenvalues.

The following article provides a clear and detailed mathematical derivation of this process.

The Vanishing Gradient Problem

As the article explains, the tanh derivative is always ≤ 1. This means the diag(tanh'(z_k)) matrix always shrinks the gradient. If the largest eigenvalue (in magnitude), known as the spectral radius ρ(W_hh), of the recurrent weight matrix is also less than 1, the overall effect of each multiplication is to shrink the gradient vector.

When you multiply many numbers that are less than 1 together, the result gets exponentially smaller.

• 0.9 * 0.9 = 0.81
• 0.9^10 ≈ 0.35
• 0.9^50 ≈ 0.005

The Consequence: The gradient signal from the end of a long sequence becomes virtually zero by the time it propagates back to the beginning. The model gets no information about how to adjust the weights that processed the early inputs to reduce the final error. This makes it impossible for the network to learn long-range dependencies, such as connecting a pronoun at the end of a long paragraph to the noun it refers to at the beginning.

The Exploding Gradient Problem

Now consider the opposite case. What if the spectral radius of W_hh is greater than 1?

Even if the tanh derivative shrinks the gradient slightly, the expansion from W_hh can dominate. When you multiply many numbers greater than 1 together, the result grows exponentially.

• 1.1 * 1.1 = 1.21
• 1.1^10 ≈ 2.6
• 1.1^50 ≈ 117

The Consequence: The gradient values can become astronomically large (Inf or NaN), leading to huge, unstable updates to the weights. This causes the training loss to diverge, and the model fails to learn anything.

The following video provides a more rigorous look at both scenarios from a linear algebra perspective, which will be valuable for your goal of deep mastery.

An Asymmetric Problem with a Partial Solution

At first glance, these two problems seem like two sides of the same coin. However, there's a critical asymmetry in how we can deal with them.

For exploding gradients, we have a simple and effective solution: gradient clipping. If the norm (magnitude) of the gradient vector exceeds a certain threshold, we simply scale it down to that threshold. This acts like a safety valve, preventing the huge weight updates that destabilize training. It's a standard and essential technique when training RNNs.

For vanishing gradients, there is no such easy fix. Once a gradient has decayed to near-zero, you cannot "amplify" it because the directional information is already lost. You can't make a vector longer if you don't know which way it's supposed to point. This makes vanishing gradients the more fundamental and difficult problem to solve.

Conclusion

Today we've dissected one of the most significant historical challenges in training neural networks for sequences.

Key Takeaways:

• Training RNNs involves Backpropagation Through Time (BPTT), which unrolls the network and creates a long chain of gradient calculations.
• This process involves the repeated multiplication of a Jacobian matrix that contains the recurrent weights (W_hh) and the derivative of the activation function (tanh').
• Vanishing Gradients: If the effective transformation factor is less than 1, the gradient signal shrinks exponentially as it travels back in time, preventing the model from learning long-range dependencies.
• Exploding Gradients: If the factor is greater than 1, the gradient grows exponentially, leading to unstable training.
• Exploding gradients can be managed with gradient clipping, but vanishing gradients require a more fundamental, architectural solution.

What's Next?

The inability of simple RNNs to overcome the vanishing gradient problem was a major roadblock. To solve it, researchers had to rethink the core RNN unit itself. This led to the development of gated architectures.

In our next lesson, we will explore Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU). You will see how they introduce explicit "gates" and a "cell state" to create an express lane for gradients, allowing learning signals to flow across hundreds of timesteps without vanishing. This was the key innovation that made RNNs truly effective for a wide range of complex sequence tasks.