LSTMs and GRUs: Solving RNN Gradient Problems

The Core Idea: An Uninterrupted Information Superhighway

The fundamental flaw in a simple RNN is that the hidden state, which serves as the network's only memory, is completely recomputed and passed through a non-linear activation function at every single timestep. This repeated transformation is what corrupts the gradient signal.

LSTMs introduce a radical solution: a separate, dedicated information channel called the cell state, often denoted as $C_t$. Think of it as an information superhighway or a conveyor belt that runs parallel to the main RNN processing path.

Information on this cell state conveyor belt can flow along largely undisturbed, making it the default behavior to remember things long-term. The key is that modifications to the cell state are primarily additive, not multiplicative transformations. In the backpropagation pass, this additive structure avoids the repeated matrix multiplication that causes gradients to vanish or explode.

But a memory that can't be updated is useless. The true power of the LSTM comes from its ability to carefully regulate this information flow using a set of mechanisms called gates.

Deconstructing the LSTM: The Gating Mechanism

Gates are neural networks in their own right, which learn to open and close access to the cell state. They use a sigmoid activation function, which squashes its output to be between 0 and 1. This output can be interpreted as a percentage: 0 means "let nothing through," and 1 means "let everything through."

An LSTM cell has three main gates to manage its cell state:

1. Forget Gate ($f_t$): Decides what information to throw away from the previous cell state, $C_{t-1}$.
2. Input Gate ($i_t$): Decides what new information to store in the current cell state, $C_t$.
3. Output Gate ($o_t$): Decides what information from the current cell state, $C_t$, to use for the hidden state $h_t$, which is the output of the cell for the current timestep.

To get a clear, step-by-step walkthrough of how these gates work together, the following blog post by Chris Olah is considered the canonical explanation.

For a more dynamic and numerically-grounded explanation, the following video walks through an LSTM cell with concrete values, which can build a strong intuition for how the gates operate in practice.

The Mathematics of Gradient Preservation

Now we can connect this architecture back to the vanishing gradient problem. The LSTM's cell state update equation is:
$C_t=f_t\odot C_{t-1}+i_t\odot {\tilde{C}}_t$
where $\odot$ is element-wise multiplication.

Notice the key operation is addition. When we calculate the gradient of the loss with respect to the previous cell state $C_{t-1}$, the chain rule gives us:
$\frac{\partial C_t}{\partial C_{t-1}}=f_t+\dots$
The derivative path includes the forget gate $f_t$ acting directly on $C_{t-1}$. Because the forget gate's activation is a learned function of the input, the network can dynamically set $f_t$ to be close to 1 for information it needs to preserve. This creates a pathway where the gradient can flow backwards through time without being forced to decay, as it's not being repeatedly multiplied by a fixed weight matrix whose eigenvalues are less than 1.

The network learns to control the gradient flow. This is the crucial leap from a simple RNN. For a deeper dive into the mathematics of the backward pass, the following resource is excellent.

Gated Recurrent Units (GRUs): A Simpler Cousin

The LSTM architecture is powerful but complex, with many parameters. In 2014, researchers introduced the Gated Recurrent Unit (GRU) as a simpler alternative that often achieves comparable performance.

The GRU makes two key simplifications:

1. It combines the cell state and hidden state into a single state vector, $h_t$.
2. It reduces the number of gates from three to two:
   • Update Gate ($z_t$): This single gate controls both forgetting and updating, analogous to a combination of the LSTM's forget and input gates. It decides how much of the past state to keep versus how much of the new candidate state to incorporate.
   • Reset Gate ($r_t$): This gate determines how much of the past state to forget when computing the new candidate state.

GRUs are less computationally expensive than LSTMs and can train faster. While LSTMs are sometimes better for problems requiring very long-term memory, the choice between LSTM and GRU often comes down to empirical testing for a specific task.

Conclusion

You've now seen the first major architectural innovation that made deep learning for sequences practical. By moving away from a simple, repeated transformation and toward a managed memory system, LSTMs and GRUs unlocked the ability to model complex, long-range dependencies.

Key Takeaways:

• LSTMs mitigate vanishing gradients by introducing a cell state, an "information superhighway" that allows gradients to flow backwards with minimal degradation.
• Gates (Forget, Input, Output) are learnable neural networks that regulate the flow of information into, out of, and within the cell state, giving the model dynamic control over its memory.
• The additive nature of the cell state update is the mathematical key to preventing the exponential decay of gradients.
• GRUs are a simpler, computationally cheaper alternative to LSTMs that merge the cell and hidden states and use only two gates (Update and Reset).

What's Next?

LSTMs and GRUs were a monumental step forward, dominating NLP for several years. However, they still have one fundamental limitation inherited from their RNN ancestor: they must process information sequentially, one token at a time. This makes it impossible to parallelize processing within a single sequence, creating a computational bottleneck.

In the next lesson, we will introduce the concept that broke this sequential barrier: the attention mechanism. You will see how attention allows a model to look at all parts of an input sequence simultaneously and decide which parts are most important at each step, paving the way for the Transformer architecture.