Dot Products and Token Relevance

The Dot Product as a Similarity Metric

From your background in linear algebra, you're familiar with the dot product as an operation that takes two vectors and returns a scalar. In the context of neural networks and embeddings, this operation takes on a new significance: it becomes a powerful, learnable measure of similarity or relevance.

Before we apply this to Queries and Keys, let's build the intuition for why the dot product is a suitable choice for measuring how "alike" two vectors are.

As the video illustrates, if vectors are constructed such that their directions in space represent semantic meaning, their dot product naturally captures their relationship. Vectors pointing in similar directions yield a large positive dot product, while orthogonal vectors (representing unrelated concepts) yield a dot product near zero.

The Geometric Interpretation of Relevance

To understand why the dot product works so well as a similarity score, we can turn to its geometric definition. The dot product of two vectors $\mathbf{a}$ and $\mathbf{b}$ is defined as:

where $\Vert \mathbf{a}\Vert$ and $\Vert \mathbf{b}\Vert$ are the magnitudes (or lengths) of the vectors, and $\theta$ is the angle between them.

This image illustrates the geometric relationship between two vectors, a and b, their magnitudes, and the angle θ between them, which are the components of the dot product formula.

This formula reveals two key factors that influence the score:

1. Alignment ($\cos (\theta )$): This is the crucial part for measuring similarity.

   • When vectors $\mathbf{a}$ and $\mathbf{b}$ point in the same direction, $\theta =0^{\circ }$ and $\cos (\theta )=1$, maximizing the score.
   • When they are perpendicular (orthogonal), $\theta =90^{\circ }$ and $\cos (\theta )=0$, indicating no relationship.
   • When they point in opposite directions, $\theta =180^{\circ }$ and $\cos (\theta )=-1$, indicating they are antonyms or opposites.

2. Magnitude ($\Vert \mathbf{a}\Vert \Vert \mathbf{b}\Vert$): The lengths of the vectors also scale the score. This means that vectors with larger magnitudes can have a larger dot product, even if they aren't perfectly aligned. In the context of attention, magnitude can be interpreted as a measure of a token's importance or salience.

A very insightful way to interpret this is through the concept of projection. The term $\Vert \mathbf{b}\Vert \cos (\theta )$ is the length of the projection of vector $\mathbf{b}$ onto vector $\mathbf{a}$. Therefore, the dot product $\mathbf{a}\cdot \mathbf{b}$ measures how much of vector $\mathbf{b}$ lies in the direction of vector $\mathbf{a}$, scaled by the magnitude of $\mathbf{a}$.

Computing Relevance between Queries and Keys

Now we can apply this understanding directly to our Query ($q$) and Key ($k$) vectors.

• A Query vector $q_i$ represents token i asking a question, like "What action is being performed on me?"
• A Key vector $k_j$ represents token j advertising its properties, like "I am an action word."

The dot product, $s_{ij}=q_i\cdot k_j$, calculates the raw attention score or relevance between these two tokens. A high score means the "question" asked by token i is well-answered by the "advertisement" from token j. The learned weight matrices $W_Q$ and $W_K$ are trained specifically to project the initial embeddings into a subspace where this dot product reflects meaningful contextual relationships.

This plot visualizes how the alignment between Query (q) and Key (k) vectors determines their relevance. The pair (q_A, k_A) are closely aligned, suggesting a high dot product and high relevance. In contrast, q_A and k_C are nearly orthogonal, implying a low dot product and low relevance.

The following resources will solidify this final connection.

To see this process in motion, let's return to the 3Blue1Brown video.

For an entire sequence of $N$ tokens, this calculation is performed for every possible pair of Query and Key vectors. If we stack our vectors into matrices $Q$ and $K$ of size $[N\times d_k]$, where $d_k$ is the dimension of the key/query vectors, we can compute all relevance scores simultaneously with a single matrix multiplication:

The resulting matrix $S$ is an $[N\times N]$ matrix of raw, unbounded attention scores, where the element $S_{ij}$ represents the relevance of token $j$ (as a key) to token $i$ (as a query).

Conclusion

In this lesson, we have focused on the fundamental mechanism for determining relevance in the Transformer. We've established that the dot product is not just an arbitrary choice but a geometrically and semantically meaningful operation for this task.

Here are the essential takeaways:

• The dot product between two vectors measures their similarity or alignment. Its value is influenced by both the angle between the vectors and their magnitudes.
• In the attention mechanism, the dot product is computed between the Query vector of one token and the Key vector of another.
• A high dot product indicates that the key is highly relevant to the query, signaling that the token associated with the key holds important information for the token associated with the query.
• This pairwise computation across all tokens results in a matrix of raw attention scores ($S=Q{K}^T$), which represents the network's initial judgment about the contextual relevance of every token to every other token.

These raw scores are a crucial intermediate step, but they are not yet ready to be used as weights for combining the Value vectors. The scores are unbounded and do not form a proper distribution. In our next lesson, we will address this by introducing scaling and the softmax function to convert these raw relevance scores into the final attention weights, completing the scaled dot-product attention formula.