Attention and Transformers Explained

1. The Need for a New Architecture

Before we dive into the solution, let's solidify our understanding of the problems with recurrent models like RNNs and LSTMs.

The core takeaway is that a non-sequential approach, one that can look at the entire sentence at once and draw connections between any two words, would be a massive improvement. This is exactly what the attention mechanism enables.

2. The Core Idea: Attention

The central idea of attention is to allow a model to weigh the importance of different words in an input sequence when processing a specific word. Instead of just passing a hidden state along a chain, the model can create direct connections to every other word and learn which ones are most relevant.

Let's start with a high-level, intuitive explanation of what attention does.

This ability to dynamically update a word's representation based on its context is the superpower of the Transformer. The mechanism that makes this possible is called self-attention, where a sequence "attends" to itself to build richer representations for each of its tokens.

3. Self-Attention: The Mechanics

Self-attention works by assigning a score to every other word in the sequence relative to the word we are currently processing. This score determines how much "attention" the current word should pay to each of the other words. This is accomplished using three vectors for each input word embedding:

1. Query (Q): Represents the current word's "question" about what it should pay attention to. Think of it as a probe.
2. Key (K): Represents what kind of information a word has to offer. It's like a label that the Query can match against.
3. Value (V): Represents the actual content or meaning of the word.

The process for calculating the new, context-aware representation for a single word is as follows:

1. For the current word, take its Query vector.
2. Calculate a score by taking the dot product of this Query with the Key vector of every word in the sentence (including itself). A higher score means a better match and more relevance.
3. Normalize these scores using a softmax function so they add up to 1. These are the "attention weights."
4. Calculate a weighted sum of all the Value vectors in the sentence, using the attention weights. Words with higher weights will contribute more to the final result.
5. This weighted sum becomes the new representation for the current word.

The best way to understand this is visually. The following blog post by Jay Alammar is considered a classic for its clear, step-by-step illustrations.

In practice, this is not done one vector at a time but with matrices for efficiency. The entire calculation can be expressed in a single, compact formula:

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$

Here, Q, K, and V are matrices containing the query, key, and value vectors for all words in the sequence. The $\sqrt{d_k}$ term is a scaling factor to keep the gradients stable during training.

4. Multi-Head Attention

A single self-attention mechanism might learn to focus on one type of relationship (e.g., how verbs relate to subjects). But what if we want the model to capture multiple types of relationships simultaneously? This is the motivation for multi-head attention.

Multi-head attention runs the self-attention process multiple times in parallel, each with its own set of Query, Key, and Value weight matrices (WQ, WK, WV). Each of these parallel "heads" can learn to focus on different aspects of the language.

For example, when processing the sentence "The animal didn't cross the street because it was too tired", one attention head might learn to link "it" back to "animal", while another might link "it" to the adjective "tired".

5. Assembling the Full Transformer Architecture

Now that we understand the core component—multi-head self-attention—we can assemble the full Transformer model. The original paper introduced an encoder-decoder architecture, which is excellent for sequence-to-sequence tasks like machine translation.

Let's break down the key components you see in this diagram.

Positional Encoding

Since self-attention processes all words in parallel, it has no inherent sense of word order. "The man bit the dog" and "The dog bit the man" would look identical to it! To fix this, we add a positional encoding vector to each input embedding. This vector follows a specific pattern based on the word's position, giving the model crucial information about the sequence order.

The Encoder Stack

The encoder's job is to build a rich, contextual representation of the input sentence. It consists of a stack of identical encoder layers (the original paper used 6). Each encoder layer has two main sub-layers:

1. A Multi-Head Self-Attention layer (as we've just discussed).
2. A simple, position-wise Feed-Forward Neural Network.

Each of these sub-layers also has a residual connection around it, followed by layer normalization (Add & Norm).

The Decoder Stack

The decoder's job is to generate the output sequence, one token at a time. It's similar to the encoder but has three sub-layers:

1. A Masked Multi-Head Self-Attention layer. The "masking" is crucial: it prevents the decoder from "cheating" by looking at future tokens in the output sequence it is trying to generate.
2. An Encoder-Decoder Attention layer. This is where the decoder pays attention to the output of the encoder. The queries (Q) come from the decoder layer below, while the keys (K) and values (V) come from the final output of the encoder stack. This allows the decoder to focus on relevant parts of the input sentence.
3. A position-wise Feed-Forward Neural Network.

Like the encoder, these layers also have residual connections and layer normalization.

To get a clearer picture of how the decoder uses masking, let's watch one more short clip.

Conclusion

You've just walked through one of the most important architectures in modern AI. By replacing sequential recurrence with parallel self-attention, the Transformer not only trains faster but also achieves a new state-of-the-art in understanding long-range context.

Key Takeaways:

• Attention Mechanism: A method to compute a weighted sum of value vectors, where the weights are dynamically computed based on the similarity between a query and a set of keys.
• Self-Attention: A specific form of attention where a sequence attends to itself, allowing each token to incorporate information from all other tokens in the sequence.
• Multi-Head Attention: Running self-attention in parallel allows the model to capture different types of relationships (e.g., syntactic, semantic) simultaneously.
• Transformer Architecture: An encoder-decoder model built entirely on multi-head attention and feed-forward layers. It discards recurrence, using positional encodings to maintain sequence order.
• Decoder Masking: Causal masking in the decoder's self-attention layer ensures that predictions for a position can only depend on known outputs at previous positions.

Preview of the Next Lesson:
Understanding the Transformer architecture is a major milestone. Now, we can leverage this knowledge to use powerful pre-trained models that are based on it. In the next lesson, we will explore the Hugging Face Transformers library, a toolkit that makes it incredibly easy to download and use models like BERT and GPT for practical NLP tasks such as text classification and Named Entity Recognition (NER).