Understanding Self-Attention and Scaled Dot-Product Attention

Hello! Welcome back to our module on the Transformer architecture.

In our last lesson, we identified the key limitations of Recurrent Neural Networks (RNNs). We saw that their sequential nature prevents parallel processing, making them slow, and that they struggle to capture long-range dependencies due to vanishing gradients. This set the stage for a new architecture that could overcome these challenges.

Today, we will dive into the core mechanism that powers the Transformer and solves these problems: self-attention. This lesson will explain the intuition behind self-attention and walk you through the step-by-step calculation of its most common implementation, scaled dot-product attention. By the end of this lesson, you will understand how a model can look at all words in a sentence simultaneously to build a rich, contextual understanding.

The Core Idea: Context is Everything

Remember the challenge with RNNs? Information had to flow sequentially. If you wanted to connect the first word to the last, that information had to survive a long journey through many intermediate steps.

Self-attention proposes a radical alternative: what if every word could directly interact with every other word in the sentence, regardless of their position? This would allow the model to dynamically weigh the importance of other words when interpreting a specific word.

As the video explains, the goal is to update each word's initial embedding (its vector representation) to create a new embedding that is context-aware. Self-attention is the mechanism that calculates this update.

The Mechanism: Query, Key, and Value

To achieve this, the self-attention mechanism introduces three new vectors for each input word, which are derived from the word's original embedding vector $x$. These are:

• Query (Q): A vector representing the current word's "question" or "search query." It asks, "What kind of information am I looking for to better understand myself?"
• Key (K): A vector that acts like a "label" or "keyword" for the word. It advertises, "This is the kind of information I can provide."
• Value (V): A vector containing the actual "substance" or information of the word. It represents what will be passed on if another word pays attention to it.

Think of it like a search in a digital library. Your search term is the Query. The library matches your query against the Keys (titles, tags) of all documents. The relevance score determines how much of each document's Value (the actual content) you get in your search results.

These three vectors are generated by multiplying the input embedding vector $x_i$ for each word $i$ by three distinct weight matrices ($W^Q,W^K,W^V$) that are learned during training.

Since your background includes linear algebra, you can see that these are linear projections of the input embeddings into three different subspaces. The model learns the optimal projections for the task during training.

The Calculation: Scaled Dot-Product Attention

Now that we have the Query, Key, and Value vectors for every word, we can calculate the attention output in four steps. This entire process is called Scaled Dot-Product Attention.

Let's say we are calculating the new, context-aware embedding for a single word.

Step 1: Calculate Similarity Scores
To determine how much attention our current word's Query should pay to every other word's Key, we compute the dot product between them. If the Query vector and a Key vector are similar, their dot product will be large, indicating high relevance.

For a matrix of Queries $Q$ and Keys $K$, this is a single matrix multiplication: $Q{K}^T$.

Step 2: Scale the Scores
The dot products can grow very large in magnitude, especially when the embedding dimension ($d_k$) is large. Large values can push the subsequent softmax function into regions with extremely small gradients, which hurts learning (the vanishing gradient problem you're familiar with). To counteract this, the scores are scaled down by dividing by the square root of the dimension of the key vectors, $\sqrt{d_k}$.

Step 3: Normalize with Softmax
To convert these scaled scores into a probability distribution, we apply the softmax function to each row of the score matrix. The resulting values, called attention weights, are all between 0 and 1 and sum to 1. Each weight represents the proportion of attention the current word should pay to every other word.

Step 4: Compute the Weighted Sum of Values
Finally, we compute the output vector for our current word by taking a weighted sum of all the Value vectors in the sequence. The weights are the attention weights we just calculated. A word with a high attention weight will contribute more of its Value vector to the final result.

This is done by multiplying the attention weights matrix by the Value matrix $V$.

The resulting vector is the new, context-aware representation for that word. And because all of this is just a series of matrix multiplications, the entire process can be executed in parallel for all words in the sequence, completely overcoming the sequential bottleneck of RNNs.

Conclusion

In this lesson, we demystified the self-attention mechanism, the engine of the Transformer. You learned how it enables a model to create context-aware word representations by allowing every word to attend to every other word in the sequence.

Key Takeaways:

• Self-attention's goal is to refine a word's embedding by incorporating information from all other words in the sequence.
• It uses three learnable projections for each word: the Query (what I'm looking for), the Key (what I have), and the Value (what I'll provide).
• Scaled dot-product attention is the function that calculates the output. Its formula is: $\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$.
• The calculation involves four steps:
  1. Compute similarity scores ($Q{K}^T$).
  2. Scale the scores ($/\sqrt{d_k}$).
  3. Normalize with softmax to get attention weights.
  4. Compute a weighted sum of the Value vectors.
• This entire process is based on matrix multiplications, making it highly parallelizable and solving a core limitation of RNNs.

Preview of the Next Lesson
What we've explored today is a single "attention head." But what if a word needs to focus on different kinds of relationships simultaneously? For example, in "The animal didn't cross the street because it was too tired," the word "it" needs to attend to "animal" (its antecedent) but also "tired" (its state). In the next lesson, we will explore multi-head attention, a technique that allows the model to do just that by running the self-attention mechanism multiple times in parallel.