Attention Mechanisms: A Comparative Study

Introduction

Welcome back! In our previous lesson, you implemented causal masking from scratch, gaining a deep, practical understanding of how autoregressive models like GPT are prevented from "cheating" by looking at future tokens. This mechanism, which we can also call masked self-attention, is a cornerstone of modern language generation.

Today, we will zoom out and place masked self-attention within the broader family of attention mechanisms. The learning outcome for this lesson is to compare self-attention, cross-attention, and masked self-attention, identifying their specific use cases. By understanding the unique role of each, you'll be able to deconstruct any Transformer-based architecture—be it an encoder, a decoder, or both—and grasp why it was designed that way.

This lesson will clarify the fundamental differences in information flow that give rise to models as distinct as BERT, GPT, and T5.

1. A Taxonomy of Attention Mechanisms

The original "Attention Is All You Need" paper introduced three distinct applications of the attention mechanism, each serving a specific purpose within the model's architecture. Let's start with a high-level overview.

Creating a Transformer From Scratch - Part One: The Attention ...

The article 'Creating a Transformer From Scratch', which we referenced in our last lesson, provides an excellent introductory section that frames the three main 'flavors' of attention. This will give you a solid mental map for our discussion.

Read the section titled 'Three Flavors of Attention'. Focus on the high-level definitions and the specific use cases cited for Bidirectional Attention (which we'll call standard self-attention), Causal Attention (masked self-attention), and Cross Attention.

To summarize and formalize the concepts from that reading, let's break down the three variants. The critical difference lies in which sequence(s) provide the Queries (Q), Keys (K), and Values (V).

Attention Type	Q, K, V Source	Information Flow	Primary Use Case	Example Architectures
Self-Attention	Q, K, and V all derive from the same input sequence.	Bidirectional: Every token attends to every other token in the sequence (past and future).	Creating rich, contextual embeddings of an entire input sequence.	Encoder blocks (BERT, ALBERT)
Masked Self-Attention	Q, K, and V all derive from the same input sequence, but a mask is applied.	Unidirectional (Causal): A token can only attend to itself and previous tokens.	Autoregressive generation of a sequence, token by token.	Decoder blocks (GPT series, LLaMA)
Cross-Attention	Q derives from a target sequence, while K and V derive from a source sequence.	Source-to-Target: Allows a target sequence to selectively focus on relevant parts of a source sequence.	Aligning or translating between two different sequences.	Encoder-Decoder bridge (T5, BART)

2. The Core Difference: Deriving Q, K, and V

Let's dive deeper into the technical distinction. As the table above shows, it all comes down to the origin of Q, K, and V.

A short video will make this distinction very clear.

A Dive Into Multihead Attention, Self-Attention and Cross-Attention

This clip from Machine Learning Studio directly contrasts self-attention and cross-attention, focusing on how the Q, K, and V matrices are derived in each case.

Watch the segment from 07:21 to 09:07. The key takeaway is identifying which input sequence (or sequences) the Q, K, and V matrices are derived from in each scenario.

This difference has direct implications for implementation. Given your experience with PyTorch, you can appreciate how this translates to code:

For Self-Attention (and Masked Self-Attention): A single input tensor x is projected to create Q, K, and V. Often, this is done with a single large linear layer Wqkv for efficiency.

# From 'Creating a Transformer From Scratch' (resource [LINK](https://benjaminwarner.dev/2023/07/01/attention-mechanism.html), section 1)
# A single input 'x' is used for Q, K, V
# x has shape (Batch, SequenceLength, HiddenSize)
self.Wqkv = nn.Linear(hidden_size, hidden_size * 3)
# ... in forward pass ...
q, k, v = self.Wqkv(x).chunk(3, dim=-1)

For Cross-Attention: Two different input tensors are required. Let's call them x (from the decoder, for the query) and y (from the encoder, for key and value). This requires separate linear layers.

# From 'Creating a Transformer From Scratch' (resource [LINK](https://benjaminwarner.dev/2023/07/01/attention-mechanism.html), section 3)
# Two inputs: 'x' (decoder) and 'y' (encoder)
self.Wq = nn.Linear(hidden_size, hidden_size)
self.Wkv = nn.Linear(hidden_size, hidden_size * 2)
# ... in forward pass ...
q = self.Wq(x)
k, v = self.Wkv(y).chunk(2, dim=-1)

This clear separation in the code structure is a direct consequence of the different roles these attention mechanisms play.

3. Assembling the Pieces: Attention in the Transformer Architecture

Now, let's see how these three mechanisms are assembled within the original Transformer architecture, which is an encoder-decoder model designed for machine translation.

Transformer Model Architecture with Attention Mechanisms — This diagram shows the complete architecture of the original Transformer. The left side is the **Encoder stack**, and the right is the **Decoder stack**. Note the three distinct attention sub-layers highlighted in purple: (1) self-attention in the encoder, (2) masked self-attention in the decoder, and (3) cross-attention connecting the two.

This diagram is the key to understanding where each attention type fits.

Encoder Self-Attention (Bottom-Left): This is standard, bidirectional self-attention. It processes the entire source sentence (e.g., "A cute teddy bear is reading"). Each word's representation is enriched with context from all other words in that same sentence.
Decoder Masked Self-Attention (Top-Right): This is the causal attention you implemented in the last lesson. It processes the target sentence being generated (e.g., "Un mignon ours en peluche..."). It ensures that when predicting the next word, the model only uses the words it has already generated.
Cross-Attention (Middle-Right): This is the bridge. The queries (Q) come from the output of the decoder's masked self-attention layer. The keys (K) and values (V) come from the final output of the entire encoder stack. This allows the decoder, at each step of generation, to "look back" at the source sentence and focus on the most relevant parts to inform its next word choice.

To see this in motion, let's watch a walkthrough of the architecture.

Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 1 - Transformer

This lecture from Stanford provides an excellent step-by-step walkthrough of the data flow in the Transformer, explaining the purpose of each attention layer in context.

Watch the segment from 01:13:50 to 01:20:46. The lecturer clearly explains: The overall encoder-decoder structure (01:13:50 - 01:16:13). The role of cross-attention, where the decoder queries the encoder's output (01:16:13 - 01:18:29). The role of masked self-attention within the decoder (01:18:29 - 01:20:46).

Test your understanding!

Imagine you are building a model for English-to-French translation. The input is "The cat sat on the mat."

Which attention mechanism is primarily responsible for understanding that "mat" is related to "sat" and "cat" within the input sentence?
As the model begins generating the French translation "Le chat...", which attention mechanism prevents it from knowing that the next word should be "s'est"?
When the model is deciding to generate the word "s'est", which attention mechanism allows it to look back at the English input and focus on the word "sat"?

Show answer

Self-Attention in the encoder. It builds a bidirectional representation of the English sentence, capturing the relationships between all words.
Masked Self-Attention in the decoder. It ensures that the generation process is autoregressive, meaning the prediction for the current token can only depend on previously generated tokens ("Le chat").
Cross-Attention. It allows the decoder (which has just generated "Le chat") to query the encoder's representation of the English sentence and identify that "sat" is the most relevant word to translate next.

Conclusion

You have now systematically compared the three fundamental attention mechanisms that form the building blocks of nearly all modern LLMs. Understanding their distinct functions is crucial for both analyzing existing models and designing new ones.

Key Takeaways:

Self-Attention is bidirectional and operates on a single sequence. It's used in encoders to build rich contextual representations (e.g., BERT).
Masked Self-Attention is unidirectional (causal) and operates on a single sequence. It's used in decoders for autoregressive tasks like text generation (e.g., GPT).
Cross-Attention connects two different sequences. It's used in encoder-decoder models to allow the decoder to consult the encoder's representation of the source input (e.g., T5, original Transformer for translation).
The fundamental difference lies in the source of the Q, K, and V matrices, which directly impacts the flow of information and the model's capabilities.

What's Next?

With this complete picture of the different attention mechanisms, our next step is to analyze their outputs. How can we "see" what the model is paying attention to? In the next lesson, we will focus on a practical skill: implementing a method to extract and visualize attention weights from a trained model to interpret its focus. This will move us from architectural theory to model interpretability.