The Original Transformer Architecture

From Recurrence to Attention: A Paradigm Shift

To fully appreciate the Transformer's design, it's essential to understand the limitations of the models it replaced. For years, sequence-to-sequence tasks were dominated by Recurrent Neural Networks (RNNs), such as LSTMs and GRUs. These models process sequences token by token, maintaining a hidden state that carries information forward.

While effective, this inherently sequential nature created two major problems:

1. Computational Bottleneck: Processing cannot be parallelized within a sequence. To process the 10th token, you must first process the preceding 9 tokens. This makes training on very long sequences extremely slow.
2. Long-Range Dependencies: The path for information to travel from an early token to a late token is long, passing through many intermediate steps. This makes it difficult for the model to capture relationships between distant words, a phenomenon often referred to as the "long-range dependency problem."

The Transformer was designed to solve precisely these issues by eliminating recurrence entirely and relying solely on attention mechanisms. The following video provides an excellent overview of the challenges with RNNs and introduces attention as a mechanism to shorten the information path length, setting the stage for the Transformer's architecture.

The Encoder-Decoder Architecture

The Transformer follows a high-level structure common to many sequence transduction models: the encoder-decoder architecture. You can see a high-level diagram of this structure below.

The workflow can be summarized as follows:

1. The Encoder: This component's job is to "understand" the input sequence. It reads the entire source sentence (e.g., an English sentence for translation) and produces a rich, contextualized representation for each token. This output can be thought of as a form of memory that captures the meaning and relationships within the input.
2. The Decoder: This component's job is to generate the output sequence (e.g., the French translation) one token at a time. At each step, it looks at the tokens it has already generated and, crucially, attends to the encoder's memory to decide which token to produce next. This property of generating outputs based on previous outputs is called auto-regression.

This design elegantly separates the task of understanding the input from the task of generating the output. Your DevOps experience offers a parallel: think of the encoder as a powerful data-processing service that ingests raw logs, enriches them, and indexes them into a searchable database like Elasticsearch. The decoder is then a client application that, step-by-step, builds a report by querying this database for relevant information.

Let's now take a detailed tour of these components.

A Deeper Look at the Components

Now let's break down each of the main stacks and their sub-layers.

The Encoder Stack

The encoder is not a single entity but a stack of N identical layers (in the original paper, N=6). As we covered in the previous module, each encoder layer has two main sub-layers:

1. A Multi-Head Self-Attention mechanism.
2. A Position-wise Feed-Forward Network.

Each of these sub-layers is wrapped with a residual connection and a layer normalization step.

The process for the encoder is as follows:

• An input sequence of token embeddings, with positional encodings added, is fed into the first encoder layer.
• The output of this layer becomes the input to the next, and so on, up the stack.
• Because the self-attention mechanism in each layer can attend to all tokens from the previous layer's output, the representations become progressively more contextualized as they move up the stack.
• The final output of the top encoder layer is the memory tensor, which will be used by every layer in the decoder stack.

The Decoder Stack

The decoder is also a stack of N identical layers. However, each decoder layer is slightly more complex, containing three sub-layers:

1. Masked Multi-Head Self-Attention: This layer operates on the output sequence. It allows each position in the decoder to attend to all preceding positions in the generated sequence (including itself). The "masking" is crucial; it prevents the decoder from "cheating" by looking at future tokens, thereby preserving the auto-regressive property.

2. Encoder-Decoder (Cross) Attention: This is the bridge between the encoder and the decoder. Here, the queries ($Q$) come from the output of the previous sub-layer (the masked self-attention), while the keys ($K$) and values ($V$) come from the encoder stack's final output (memory). This allows every token being generated by the decoder to "look back" at the entire input sequence and weigh the importance of each input token.

3. Position-wise Feed-Forward Network: This functions identically to the one in the encoder, providing additional non-linear transformation for each token's representation.

Let's watch a detailed explanation of the decoder's unique components.

The Full Process: Training and Inference

Having seen the static components, let's understand how they operate dynamically during training and inference.

Training (Teacher Forcing)

During training, the model is fed both the source sentence and the complete target sentence simultaneously. The target sentence fed to the decoder is shifted right by one position and prepended with a start-of-sequence (<SOS>) token. The decoder is then asked to predict the next token at every position, with the actual target sentence (the "ground truth") used as the label. For example, to predict the token at position 3, the decoder is given the first 3 tokens of the ground truth sequence as input. This method, where the model is always fed the correct previous tokens regardless of its own predictions, is called teacher forcing. It allows for massive parallelization, as the loss for all positions can be computed in a single forward pass.

Inference (Autoregressive Generation)

During inference (when generating a new translation), there is no ground truth to "force." The process is sequential:

1. The encoder processes the entire source sentence to produce the memory tensor. This is done only once.
2. The decoder is initialized with only the <SOS> token. It generates a probability distribution for the first token.
3. The most likely token (or a sampled token) is chosen and appended to the decoder's input sequence.
4. This new sequence is fed back into the decoder to generate the next token.
5. This loop continues until the model generates an end-of-sequence (<EOS>) token.

The following video segment provides an excellent visual walkthrough of this entire process, clearly contrasting the parallel nature of training with the sequential nature of inference.

The Original Paper and Code Implementation

To solidify your understanding, it's always valuable to go to the source. The original "Attention Is All You Need" paper lays out this architecture, and the annotated version from Harvard NLP provides a bridge to the code.

Now, let's see how this abstract architecture translates into a high-level code structure. Your experience with software development will make this tangible. The EncoderDecoder class acts as a master module that orchestrates the flow between its child encoder and decoder modules.

Conclusion

In this lesson, we have assembled the Transformer blocks from our previous module into the complete encoder-decoder architecture. You now have a comprehensive view of the original model that revolutionized natural language processing.

Key takeaways include:

• The Transformer's encoder-decoder structure is designed for sequence-to-sequence tasks, separating the problem of understanding the input (encoder) from generating the output (decoder).
• The encoder stack processes the entire input sequence in parallel, producing a contextualized memory representation.
• The decoder stack is auto-regressive, generating the output token by token. Its key distinction is a third sub-layer—cross-attention—which allows it to consult the encoder's memory at each step.
• Training is done in parallel using teacher forcing, while inference is a sequential, auto-regressive process.

You now understand the blueprint that started it all. However, the encoder-decoder model is just one of three major architectural families. In our upcoming lessons, we will explore how researchers adapted this design, leading to the encoder-only (e.g., BERT) and decoder-only (e.g., GPT) architectures that dominate different areas of NLP today. We'll start in the next lesson by taking a closer look at the different types of attention and where they are used.