RNN Architecture and Sequential Data

1. Why Do We Need a New Architecture?

Before diving into the architecture of an RNN, let's understand why our existing tools fall short when dealing with sequences.

A standard neural network takes a fixed-size input and produces a fixed-size output. A CNN does the same, although it's specialized for image grids. But what if your input has a variable length, like a sentence?

• "I am good." (3 words)
• "I am not at all bad or sad right now." (9 words)

A fixed-size network can't handle this variability. More importantly, it doesn't have a mechanism to understand that the meaning of "sad" might be negated by "not" appearing several words earlier. It treats every input element independently.

RNNs were designed to solve exactly this problem. To see this in action, watch the beginning of the following video.

As you saw, the key innovation of an RNN is its feedback loop, which allows it to persist information across steps. Let's formalize this idea.

2. The Core Architecture of an RNN

At the heart of an RNN is a simple but powerful idea: a loop. The network processes a sequence one element at a time, and at each step, the output from the previous step is fed back as an input to the current step. This feedback loop creates a form of memory.

The "memory" is stored in a vector called the hidden state, usually denoted by $h$. At each time step $t$, the network calculates a new hidden state $h_t$ based on two things:

1. The current input element $x_t$.
2. The previous hidden state $h_{t-1}$.

This recurrence can be visualized in two ways: as a "folded" model with a loop, and as an "unrolled" model stretched out over time. The unrolled view makes it clear how information flows through the sequence.

Crucially, the same set of weights is used at every single time step. This parameter sharing is what allows the RNN to handle sequences of variable length and generalize what it learns at one position to all other positions.

The following video from IBM Technology provides a concise explanation of the hidden state and the unrolling process.

The Mathematics of a Simple RNN

The core computations within a single RNN cell at each time step $t$ are defined by two equations.

1. The Hidden State Update:

   $$h_t=\tanh (W_{hh}{h}_{t-1}+W_{xh}{x}_t+b_h)$$
   • $h_t$ is the new hidden state.
   • $h_{t-1}$ is the previous hidden state.
   • $x_t$ is the current input vector.
   • $W_{hh}$ and $W_{xh}$ are weight matrices for the recurrent and input connections, respectively.
   • $b_h$ is a bias vector.
   • tanh is a common activation function used to keep the hidden state values within a certain range (-1 to 1).

2. The Output Calculation:

   $$y_t=W_{hy}{h}_t+b_y$$
   • $y_t$ is the output at this time step.
   • $W_{hy}$ is the weight matrix for the hidden-to-output connection.
   • $b_y$ is the output bias vector.

To see these equations within a full network diagram, let's turn to a classic textbook.

Let's put this all together with a concrete example.

3. RNN Architectures and Applications

The basic RNN cell can be configured in different ways to solve different kinds of problems. These configurations are often named based on the relationship between the input and output sequence lengths (e.g., "many-to-one").

The following video gives an excellent overview of these common patterns.

Let's summarize those key architectures:

• Sequence-to-Vector (Many-to-One): Processes a whole sequence and produces a single output.
  • Application: Sentiment analysis (input: a movie review, output: positive/negative), text classification.
• Vector-to-Sequence (One-to-Many): Takes a single input and generates a sequence.
  • Application: Image captioning (input: an image vector, output: a sequence of words).
• Sequence-to-Sequence (Many-to-Many): Processes an input sequence and produces an output sequence of the same length.
  • Application: Time series forecasting (e.g., predicting stock prices for each day based on past days), video frame classification.
• Encoder-Decoder (Delayed Sequence-to-Sequence): A more advanced architecture that maps an input sequence to an output sequence of a different length. It consists of two RNNs: an "encoder" that reads the input sequence into a single context vector, and a "decoder" that generates the output sequence from that vector.
  • Application: Machine translation (e.g., English to French), question answering.

For more complex tasks, we sometimes need to look at both past and future context to make a prediction. This is where Bidirectional RNNs come in. A Bidirectional RNN is simply two separate RNNs: one that processes the sequence from start to end, and another that processes it from end to start. The outputs from both are combined at each time step.

4. A Key Challenge: Long-Term Dependencies

While powerful, the simple "vanilla" RNN we've discussed has a significant weakness. During training, gradients must be backpropagated through every time step. For long sequences, these gradients can either shrink exponentially until they disappear (vanishing gradient problem) or grow exponentially until they overflow (exploding gradient problem).

The vanishing gradient problem makes it very difficult for the network to learn dependencies between elements that are far apart in a sequence. For example, in the sentence "The man who I saw earlier this morning in the coffee shop, and who was wearing a red hat, ... is happy," the verb "is" depends on the singular noun "man" from the very beginning of the sentence. A simple RNN would struggle to carry this information over such a long distance.

To solve this problem, more advanced RNN variants were developed, most notably the Long Short-Term Memory (LSTM) network and the Gated Recurrent Unit (GRU). These architectures introduce "gates"—special mechanisms that allow the network to learn when to add, forget, and output information from its memory, making them much more effective at learning long-range dependencies. We will touch on these in future lessons.

Conclusion

In this lesson, you've learned the fundamental principles behind Recurrent Neural Networks. You have seen how their unique architecture, based on a recurrent feedback loop and shared weights, enables them to process sequential data in a way that other network types cannot.

Key Takeaways:

• RNNs are specialized for sequential data (e.g., text, time series) where order matters.
• The core of an RNN is a loop that maintains a hidden state, which acts as the network's memory of the past.
• RNNs can be visualized as an unrolled chain of identical cells, but the weights are shared across all time steps.
• Different RNN configurations exist for different tasks, including many-to-one (classification), one-to-many (generation), and encoder-decoder (translation).
• Simple RNNs struggle with long sequences due to the vanishing and exploding gradient problems, which makes learning long-term dependencies difficult.

Preview of the Next Lesson:
The sequential, step-by-step processing of RNNs, while powerful, has its own limitations—it can be slow and still struggles with extremely long-range dependencies. This led to the development of a revolutionary new architecture that processes all elements of a sequence at once using a mechanism called "attention." In the next lesson, we will explore the high-level concept of the Transformer architecture and understand why it has become the state-of-the-art for most NLP tasks today.