Encoder-Decoder Synergy in Transformers

Hello! Welcome back to our series on the Transformer architecture.

In the last two lessons, we've taken a deep dive into the core building blocks of the Transformer: the encoder block, which creates a rich representation of the input sequence, and the decoder block, which uses masked self-attention and cross-attention to generate an output sequence. You learned that the cross-attention mechanism is the crucial link where the decoder "looks at" the input sentence.

Today, we'll zoom out from the individual blocks to see the full picture. Our goal is to explain how the encoder and decoder stacks work together in the full Transformer model. We will explore how information flows from the input sequence all the way to the final generated output, and we'll see how this process differs critically between training the model and using it for inference.

The Complete Architecture: A Bird's-Eye View

Let's begin by looking at the now-famous diagram from the original "Attention Is All You Need" paper.

As we've discussed, the Transformer consists of two main parts:

1. The Encoder Stack: A series of identical encoder blocks stacked on top of each other. It processes the entire input sequence (e.g., an English sentence) in one go.
2. The Decoder Stack: A series of identical decoder blocks. It generates the output sequence (e.g., the French translation) token by token.

The most important connection is the set of arrows that flow from the top of the encoder stack to the cross-attention layer in every block of the decoder stack. Let's see what information is being passed.

The Information Bridge: How the Encoder "Talks" to the Decoder

The encoder's job is to distill the meaning and context of the entire input sequence into a set of vectors. After the input sentence passes through the entire encoder stack, the output from the topmost encoder is a sequence of contextualized vectors, one for each input token.

These vectors are then transformed into a set of Key (K) and Value (V) matrices. These K and V matrices represent the "memory" of the input sentence. This memory is then broadcast to every decoder block.

Within each decoder block, the cross-attention sub-layer receives these K and V matrices from the encoder. The decoder's own internal state provides the Query (Q) vector. By performing attention with its Q and the encoder's K and V, the decoder can ask questions like, "For the next word I need to generate, which part of the original input sentence is most important right now?"

The following reading provides a great visualization and explanation of this process.

This interaction is the heart of the Transformer's sequence-to-sequence capability. The decoder isn't just working from a single context vector; at every step, it has full access to the richly encoded representation of the entire input sentence.

The Two Modes of Operation: Training vs. Inference

Understanding how the encoder and decoder stacks interact requires looking at two distinct scenarios: training the model and using it for inference (generation). The data flow is quite different in each case.

1. Training Phase: Parallel Processing with "Teacher Forcing"

During training, we have both the input sentence (e.g., "I am a student") and the target translation (e.g., "Je suis un étudiant"). To make training fast and efficient, we want to process as much as possible in parallel. The Transformer achieves this with a method called teacher forcing.

Here's how it works:

1. The encoder receives the entire input sentence ("I am a student") and processes it in a single forward pass.
2. The decoder receives the entire target sentence, but shifted right and with a start-of-sequence token (<SOS>) prepended (e.g., "<SOS> Je suis un étudiant").
3. The decoder processes this entire target sequence in parallel. The masked self-attention mechanism ensures that when predicting the word at position i, the model can only see the words from positions 0 to i, preventing it from "cheating".
4. At each position, the decoder uses cross-attention to consult the encoder's output and predicts the next word in the sequence. For the input "Je", it tries to predict "suis"; for "suis", it tries to predict "un", and so on.
5. The model's predictions are compared to the actual target sentence ("Je suis un étudiant <EOS>"), a loss is calculated, and gradients are backpropagated to update the model's weights.

The key takeaway is that during training, the entire process happens in one parallel forward pass.

This video provides a fantastic visualization of the training flow.

2. Inference Phase: Sequential, Autoregressive Generation

When we use a trained model to perform a translation, we don't have the target sentence. We must generate it one word at a time. This is an autoregressive process.

Here's the step-by-step flow:

1. The encoder processes the entire input sentence ("I am a student") and computes its memory (the K and V vectors). This happens only once.
2. The decoder input starts with only the <SOS> token.
3. The decoder processes <SOS>, uses cross-attention to look at the encoder's memory, and passes its output through a final Linear and Softmax layer to predict the first word ("Je").
4. The decoder now takes its new sequence, "<SOS> Je", as input. It processes this sequence, again using the encoder's memory, and predicts the second word ("suis").
5. This loop continues: the output from the previous step is appended to the input for the next step.
6. The process stops when the model predicts a special end-of-sequence (<EOS>) token.

Inference is inherently sequential, but notice the efficiency gain: the computationally heavy encoding of the input sentence is done just once and reused at every decoding step.

Let's watch the same video, which now explains this iterative inference process. The contrast with the training process you just saw will make the difference very clear.

For another excellent, step-by-step walkthrough of this encoder-decoder interaction during inference, you can also watch this video from StatQuest.

From Vector to Word: The Final Layers

After the decoder stack produces its final output vector for a given time step, one last step is needed to turn that vector into a predicted word.

1. Linear Layer: A fully connected layer projects the high-dimensional vector from the decoder stack into a very large vector called a logits vector. The size of this vector is equal to the size of our entire output vocabulary (e.g., 30,000 words). Each element of the logits vector corresponds to a score for a particular word.
2. Softmax Layer: A softmax function is applied to the logits vector, converting all the scores into probabilities that sum to 1. The word with the highest probability is then selected as the output for that time step.

This final projection is what connects the abstract vector representations inside the model to concrete words in a human language.

Conclusion

In this lesson, we have assembled the full picture of how the Transformer's encoder and decoder stacks work in concert to perform sequence-to-sequence tasks.

Key Takeaways:

• The encoder stack processes the entire input sequence in parallel to create a set of contextual representations (Keys and Values).
• This "memory" of the input is passed to the cross-attention mechanism in every decoder block, allowing the decoder to reference the input at each step of generation.
• During training, the model uses "teacher forcing," processing the complete source and target sequences in a single, parallel forward pass for efficiency.
• During inference, the model works autoregressively: the encoder runs once, and the decoder generates the output one token at a time, feeding its own output back as input for the next step.
• A final Linear layer and Softmax function translate the decoder's output vector into a probability distribution over the vocabulary to select the next word.

Preview of the Next Lesson

The encoder-decoder architecture is powerful, but it's not the only way to use Transformer blocks. Many state-of-the-art models use only one half of the architecture. In our next lesson, we will explore the different "flavors" of Transformer models and learn to differentiate between encoder-only (like BERT), decoder-only (like GPT), and encoder-decoder (like T5) architectures, and understand the types of tasks each is best suited for.