Self-Attention vs. Cross-Attention in Encoder-Decoder Models

Introduction

In our previous lesson, we constructed a complete picture of the original Transformer from "Attention Is All You Need," identifying its two main components: the encoder and the decoder. We noted that while both stacks use attention, the decoder has a unique "Encoder-Decoder Attention" layer that acts as a bridge to the encoder's output.

This lesson will delve into the precise nature of that bridge. We will formally distinguish between the two fundamental modes of attention used in the Transformer: self-attention and cross-attention. Your goal will be to understand their core difference—which lies in the origin of their Query, Key, and Value vectors—and to pinpoint exactly where each is deployed within the encoder-decoder architecture to fulfill its specific role.

The Source of Q, K, and V: The Fundamental Distinction

At its core, the scaled dot-product attention mechanism is a single, consistent formula:

Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}}) V

The operational mechanics remain the same regardless of where this formula is applied. The crucial difference between self-attention and cross-attention is not in the calculation but in the source of the $Q$ (Query), $K$ (Key), and $V$ (Value) matrices. Everything hinges on a simple question: are we attending within a single sequence or between two different sequences?

Let's watch a brief video that introduces this core distinction.

A Dive Into Multihead Attention, Self-Attention and Cross-Attention

The video "A Dive Into Multihead Attention, Self-Attention and Cross-Attention" from Machine Learning Studio provides a clear and concise initial explanation of our topic.

Watch the section from the introduction to understand the high-level purpose of self-attention (relationships within one sequence) versus cross-attention (relating two different sequences).

Self-Attention: A Sequence Looking at Itself

Self-attention is the mechanism that allows a model to build a contextual understanding of a single sequence. In this mode, the Query, Key, and Value vectors are all projected from the same input sequence.

Purpose: To weigh the importance of all other tokens within the same sequence relative to a specific token. It answers the question: "To understand the word 'it' in this sentence, which other words in the same sentence should I pay attention to?"
Where it's used:
1. Encoder Layers: All attention layers in the encoder are self-attention layers. They process the source sequence (e.g., the English sentence to be translated), allowing every token to attend to every other token. This creates a deeply contextualized representation of the input.
2. Decoder Layers (First Sub-layer): The first attention sub-layer in each decoder layer is a masked self-attention layer. It allows each token in the target sequence (the translation being generated) to attend to itself and all preceding tokens in that same target sequence. The mask is critical to prevent the model from seeing the future, preserving its autoregressive, token-by-token generation capability.

The diagram below provides a simple visual for this concept.

A high-level comparison. On the left, self-attention derives its Query, Key, and Value from a single source 'x'. On the right, cross-attention derives its Query from a target 'u' and its Key and Value from a source 'x'.

Cross-Attention: Bridging Two Sequences

Cross-attention is the mechanism that connects the decoder to the encoder, allowing the generation process to be conditioned on the source sequence. Here, the Query, Key, and Value vectors are projected from two different input sequences.

Purpose: To align information between two distinct sequences. Specifically, it allows the decoder to "look back" at the encoder's full output and focus on the most relevant parts of the source sequence needed to generate the next target token. It answers the question: "Given the target sequence I've generated so far (my Query), which parts of the original source sentence (the Keys/Values) are most important for predicting the next word?"
Where it's used:
- Decoder Layers (Second Sub-layer): This is the "Encoder-Decoder Attention" layer we identified in the previous lesson.
  - The Query (Q) matrix is derived from the output of the decoder's first sub-layer (the masked self-attention layer). It represents the state of the sequence being generated.
  - The Key (K) and Value (V) matrices are derived from the final output of the encoder stack (the memory tensor). They represent the fully contextualized source sequence.

Let's watch a detailed explanation of how these inputs are sourced differently.

A Dive Into Multihead Attention, Self-Attention and Cross-Attention

Continuing with the same video, this segment clearly explains how the inputs to Q, K, and V differ between self-attention and cross-attention.

Watch the section from the explanation of inputs. Pay close attention to which sequence provides Q and which provides K and V in the cross-attention scenario.

The following diagram illustrates the flow of data in cross-attention, which should resonate with your background in system design. Here, Inputs 1 (with n tokens) could be the decoder's state, and Inputs 2 (with m tokens) could be the encoder's output. The queries are projected from Inputs 1, while keys and values are projected from Inputs 2.

In this cross-attention diagram, a sequence of `n` tokens (Inputs 1) generates the Query matrix Q, while a second sequence of `m` tokens (Inputs 2) generates the Key (K) and Value (V) matrices. The attention matrix `A` has dimensions `n x m`, showing how each of the `n` query tokens attends to each of the `m` key/value tokens.

For a more technical and code-oriented perspective, Sebastian Raschka's blog post provides an excellent breakdown.

Understanding and Coding the Self-Attention Mechanism ...

This post from Sebastian Raschka's blog, "From Scratch," explains cross-attention by building on the self-attention mechanism, showing how the input sources change.

Please read the section on cross-attention, which clearly defines it and contrasts it with self-attention. Then, review the conceptual code snippet that follows, starting from this line. Notice how embedded_sentence_2 is used to generate the keys and values, while the query comes from the original sentence. This directly mirrors the encoder-decoder dynamic.

Architectural Placement: A Summary

With a clear distinction between the two attention types, we can now map them precisely onto the encoder-decoder architecture. The article "The Transformer Architecture: A Deep Dive" provides an authoritative summary.

The Transformer Architecture: A Deep Dive - Imad Dabbura

This article by Imad Dabbura provides a very clear, concise definition of both attention types and their roles.

Please read the short section Self-Attention vs. Cross-Attention. This text explicitly states which mechanism is used where, reinforcing everything we've covered.

To summarize the complete data flow:

The Encoder: Processes the source sequence. Each of its N layers uses bidirectional self-attention to mix information across the entire sequence.
The Decoder: Generates the target sequence token by token. At each step, for each of its N layers, it performs two attention operations in sequence:
- Masked Self-Attention: The decoder looks at the tokens it has already generated to build a contextual representation of the partial target sequence.
- Cross-Attention: The decoder takes its current representation (as Q) and attends to the final output of the encoder (as K and V), pulling in relevant information from the source sequence.

The Illustrated Guide to Transformers video provides a final, clear walkthrough of this two-step process within the decoder.

Illustrated Guide to Transformers Neural Network: A step by step explanation

This video by "The AI Hacker" walks through the decoder's architecture, highlighting its two distinct attention layers.

First, watch the segment explaining the decoder's first attention layer, the masked self-attention. Then, immediately following that, watch the concise explanation of the second layer, the cross-attention mechanism, which clearly states that Q comes from the decoder and K/V from the encoder.

Conclusion

In this lesson, we have dissected the attention mechanisms within the encoder-decoder Transformer, moving beyond a single "attention" concept to a more nuanced understanding of its two primary modes.

Here are the key takeaways:

The fundamental difference between self-attention and cross-attention is the source of the Q, K, and V vectors.
Self-attention derives Q, K, and V from a single sequence. Its purpose is to build internal contextual representations. It is used in all encoder layers and as the first sub-layer in all decoder layers.
Cross-attention derives Q from a target sequence and K and V from a source sequence. Its purpose is to align and transfer information between two sequences. It is used as the second sub-layer in decoder layers to condition the output on the encoder's representation.

You now understand not just that the encoder and decoder are connected, but how this connection is implemented at the level of the attention mechanism. This distinction is crucial, as the architectural patterns we will explore next are defined by their exclusive use of one or both of these mechanisms.

In our next lesson, we will broaden our view to examine the other major Transformer families: the decoder-only architecture (like GPT) and the encoder-only architecture (like BERT), and understand why their specific use of attention makes them suitable for different kinds of tasks.