Understanding Vanilla RNNs: Architecture, Weight Sharing, and Unrolling

Introduction

In our previous lessons on Convolutional Neural Networks, we explored architectures designed to exploit spatial structure in data, such as images. We saw how convolutions and pooling layers build hierarchical feature representations that are equivariant or invariant to translation. We now pivot from the domain of space to the domain of time and sequence. Many real-world problems involve data where order is not just important, but defines the data's very meaning—stock prices, natural language, or video frames are prime examples.

This lesson introduces the Recurrent Neural Network (RNN), a class of neural networks specifically designed to operate on sequential data. Our goal is to understand the fundamental architecture of a "vanilla" RNN. You will learn how it uses an internal state and a recurrent loop to process sequences step-by-step, how it achieves immense parameter efficiency through weight sharing across time, and how to visualize its operation by "unrolling" the network.

The Challenge of Sequential Data

Feedforward networks, including the CNNs we've studied, typically assume that their inputs are independent and identically distributed (i.i.d.). For a task like image classification, shuffling the training images doesn't affect the outcome. However, for a sentence, shuffling the words destroys its meaning. A model processing such data needs to understand context and dependencies across time.

RNNs provide a flexible framework for handling various types of sequence processing tasks. The following video gives an excellent overview of the different sequence-to-sequence paradigms that RNNs enable.

To handle such tasks, a model needs a form of memory. It must be able to retain information from earlier parts of a sequence to inform its processing of later parts. This is the core idea behind the RNN.

The Vanilla RNN Architecture: A Recurrent Loop

An RNN processes a sequence element by element. At each time step $t$, it takes an input $x_t$ and updates its internal hidden state, $h_t$. This hidden state serves as the network's memory, summarizing the information from all previous time steps.

The defining feature of an RNN is its recurrence relation. The hidden state at the current time step, $h_t$, is a function of both the current input, $x_t$, and the previous hidden state, $h_{t-1}$.

The "vanilla" RNN, the simplest variant, implements this recurrence with a specific set of operations. The following lecture clips formally define this architecture.

As explained in the video and the image above, the state update and output are governed by two equations:

1. Hidden State Update:

   $$h_t=\tanh (W_{hh}{h}_{t-1}+W_{xh}{x}_t+b_h)$$

2. Output Calculation:

   $$y_t=W_{hy}{h}_t+b_y$$

Let's break down the components:

• $x_t$: The input vector at time step $t$.
• $h_{t-1}$: The hidden state vector from the previous time step. $h_0$ is typically initialized as a vector of zeros.
• $h_t$: The new hidden state vector, which captures information from the past up to time $t$.
• $y_t$: The output vector at time step $t$.
• $W_{xh}$, $W_{hh}$, $W_{hy}$: The weight matrices that are the learnable parameters of the model.
• $b_h$, $b_y$: The bias vectors, also learnable parameters.
• $\tanh$: The hyperbolic tangent activation function, which squashes the values of the hidden state to be between -1 and 1.

The crucial insight here is that the computation at each step depends on the result of the previous step. This temporal dependency is what allows the network to maintain and propagate information through the sequence.

Unrolling and Weight Sharing

The compact, looped diagram of the RNN cell is elegant but can be hard to reason about. To understand how the network operates over a full sequence and, critically, how it is trained, we can visualize it by unrolling the loop. Unrolling simply means creating a copy of the network for each time step in the sequence.

When unrolled, the RNN looks like a very deep feedforward neural network, where each time step is effectively a layer. However, there is a fundamental difference.

Weight Sharing Across Time

In this unrolled view, the most important concept is that the same set of weight matrices ($W_{xh}$, $W_{hh}$, $W_{hy}$) is used at every single time step. This is the principle of parameter sharing across time.

This is directly analogous to the parameter sharing you saw in CNNs.

• In a CNN, a single kernel's weights are shared across different spatial locations in the input.
• In an RNN, a single set of weights is shared across different temporal positions in the sequence.

This design has two profound consequences:

1. Efficiency: The number of parameters in the model is independent of the sequence length. Whether you process a sentence of 5 words or a document of 5000 words, the number of weights remains the same. This is a stark contrast to a standard feedforward network, which would require a unique set of input weights for each position in the sequence, leading to a parameter count that grows linearly with sequence length.
2. Generalization: The model learns a transition function that is applicable at any point in a sequence. It doesn't have to learn separate rules for the beginning, middle, and end of a sequence.

The following resources provide a formal look at the unrolling process and the concept of weight sharing.

A Concrete Example: Character-Level Language Modeling

To solidify these abstract concepts, let's consider a classic application: a character-level language model. The task is to train an RNN to read a sequence of characters and, at each step, predict the next character in the sequence.

Let's use the training sequence "hello".

1. Input: The characters 'h', 'e', 'l', 'l', 'o' are fed into the RNN one at a time. Each character is represented as a one-hot vector (a vector of zeros with a single '1' at the index corresponding to that character in the vocabulary).
2. Target: At each time step, the target for the network is the next character in the sequence.
   • When the input is 'h', the target is 'e'.
   • When the input is 'e', the target is 'l'.
   • When the input is 'l', the target is 'l'.
   • When the input is 'l', the target is 'o'.
3. Process:
   • At $t=1$, the RNN takes $x_1$ (the vector for 'h') and the initial hidden state $h_0$ (a zero vector) to compute $h_1$. From $h_1$, it computes an output $y_1$, which is a vector of scores for every character in the vocabulary.
   • At $t=2$, the RNN takes $x_2$ (the vector for 'e') and the previous hidden state $h_1$ to compute $h_2$. The output $y_2$ is then computed from $h_2$.
   • This process repeats for the entire sequence. At each step, the hidden state $h_t$ contains a summary of the characters seen so far.
4. Loss: At each step, we can use a softmax function on the output vector $y_t$ to get a probability distribution over the possible next characters. We then compare this distribution to the actual next character using a cross-entropy loss. The total loss for the sequence is the sum of the losses at each time step.

This example is brilliantly explained by Andrej Karpathy in his CS231n lecture.

Conclusion

In this lesson, we have laid the theoretical groundwork for understanding Recurrent Neural Networks. You have moved from architectures for spatial data to a new paradigm for handling sequences.

Here are the key takeaways:

• RNNs process sequential data by iterating through the sequence and maintaining a hidden state $h_t$, which acts as a memory.
• The core of an RNN is the recurrence relation, where the current hidden state is a function of the previous state and the current input: $h_t=f(h_{t-1},x_t)$.
• The vanilla RNN implements this with the equation $h_t=\tanh (W_{hh}{h}_{t-1}+W_{xh}{x}_t+b_h)$.
• Unrolling an RNN reveals a deep computational graph where the same set of weights is shared across all time steps, enabling parameter efficiency and generalization over sequence length.

While the unrolled graph looks like a standard deep network, training it is not so simple. The very mechanism that gives RNNs their power—the repeated application of the same weights—also introduces significant challenges for learning. In our next lesson, we will explore how we train these models using Backpropagation Through Time (BPTT) and confront the critical problem of vanishing and exploding gradients, which hinders the ability of vanilla RNNs to learn long-range dependencies.