Transformer Architecture: Encoder-Decoder & Core Components

The Transformer's Blueprint: An Encoder-Decoder Structure

At its heart, the original Transformer is a sequence-to-sequence (seq2seq) model, a concept you're familiar with from RNNs. It's designed for tasks like machine translation, where you map an input sequence (a sentence in one language) to an output sequence (the same sentence in another language).

It consists of two main parts:

1. The Encoder: Its job is to read the entire input sequence and build a rich, contextualized numerical representation for each token. It "encodes" the meaning of the input sentence.
2. The Decoder: Its job is to take the encoder's representation and generate the output sequence, one token at a time. It's autoregressive, meaning each new token it generates is based on the encoder's output and the tokens it has already produced.

This division of labor is a powerful paradigm. The encoder can focus solely on understanding the input, while the decoder can focus on generating a coherent output based on that understanding.

Let's look at a high-level diagram.

To get a quick walkthrough of this high-level data flow, watch the following short clip.

Now, let's break down what's inside each of these stacks.

Dissecting the Encoder

The encoder's goal is to create a set of vectors, one for each input token, where each vector is rich with contextual information from the entire input sentence. This process happens in a stack of identical encoder layers (the original paper used N=6).

Here's the journey of an input sentence through the encoder:

1. Input Embedding: As with previous models, we start by converting each token into a dense vector using a learned embedding matrix.
2. Positional Encoding: The self-attention mechanism, by design, is permutation-invariant—it doesn't have a built-in sense of word order. To fix this, we inject information about each token's position in the sequence. The original Transformer did this by adding a pre-calculated vector based on sine and cosine functions of different frequencies. This gives the model a way to understand the relative and absolute positions of tokens.
3. Multi-Head Self-Attention: This is the core of the encoder. For each token, this layer allows it to "look at" every other token in the input sequence and calculate attention scores. These scores determine how much influence each other token should have on its representation. "Multi-head" simply means this process is done in parallel multiple times (e.g., 8 heads), with each "head" learning to focus on different types of relationships (e.g., syntactic, semantic). The results are then combined.
4. Add & Norm (Residual Connection + Layer Normalization): The output of the attention layer is added to its input (a residual connection, just like in ResNet), and the result is normalized. Your experience with deep learning will tell you this is crucial for enabling deep networks to train effectively by preventing vanishing gradients and stabilizing layer inputs.
5. Position-wise Feed-Forward Network (FFN): This is a simple two-layer fully connected neural network that is applied independently to each token's representation. It provides additional computational depth, allowing the model to learn more complex transformations.
6. Another Add & Norm: A second residual connection and layer normalization are applied after the FFN.

The output of one encoder layer becomes the input to the next, allowing the model to build progressively more sophisticated representations.

Dissecting the Decoder

The decoder's goal is to generate the output sequence autoregressively. It also consists of a stack of identical layers (N=6), but each layer has an extra sub-layer compared to the encoder.

Here's the data flow for one generation step:

1. Output Embedding & Positional Encoding: Similar to the encoder, the decoder takes the tokens generated so far, embeds them, and adds positional encodings.
2. Masked Multi-Head Self-Attention: This layer works just like the self-attention in the encoder, but with a critical difference: it applies a "look-ahead mask." This mask prevents each position from attending to subsequent positions. Without this, the decoder could "cheat" by looking at the word it's supposed to predict. This masking enforces the autoregressive property.
3. Add & Norm: A residual connection and layer normalization are applied.
4. Encoder-Decoder Attention (Cross-Attention): This is the crucial link between the encoder and decoder. In this layer, the queries ($Q$) come from the decoder's masked self-attention layer, but the keys ($K$) and values ($V$) come from the final output of the encoder stack. This allows the decoder to look at the entire input sentence and decide which parts are most relevant for predicting the next output token. For instance, when translating a sentence, this layer helps the decoder focus on the relevant source words.
5. Add & Norm: Another residual connection and layer normalization.
6. Position-wise Feed-Forward Network: Same as in the encoder.
7. Final Add & Norm: The final residual connection and normalization in the decoder block.

After the final decoder layer, a linear transformation followed by a softmax function produces a probability distribution over the entire vocabulary. The token with the highest probability is chosen as the output for that step, and it is then fed back into the decoder as input for the next step. This process repeats until a special <end_of_sentence> token is generated.

This detailed walkthrough can be quite dense. The following video resource provides a comprehensive visual explanation of each component we've just discussed.

For a more text-based and highly visual alternative, the article "Transformers Illustrated!" is an excellent companion resource. You can refer to it to clarify any points from the video.

Conclusion

In this lesson, we've constructed the original Transformer architecture from the ground up. You now have a mental map of its components and the flow of data through the system.

Key Takeaways:

• Overall Structure: The Transformer uses an encoder-decoder architecture. The encoder processes the full input to create contextual representations, and the decoder generates the output autoregressively.
• Encoder Components: Each encoder layer contains a multi-head self-attention mechanism and a position-wise feed-forward network, glued together with residual connections and layer normalization.
• Decoder Components: Each decoder layer has three sub-layers: masked multi-head self-attention (to look at previous outputs), encoder-decoder cross-attention (to look at the input), and a feed-forward network.
• Key Enablers: Positional encodings provide word order information, while residual connections and layer normalization are critical for stable training of these deep models.

What's Next?

You now understand the "what" and "where" of the Transformer's components. In the next lesson, we will dive deep into the "how." We'll focus on the engine of the Transformer: the scaled dot-product attention mechanism. You will move from description to implementation, coding this mechanism from scratch in PyTorch to gain a practitioner's understanding of how it works.