Understanding Sinusoidal Positional Encodings

Introduction

In our previous lesson, we examined the learnable projection matrices ($W_Q,W_K,W_V,W_O$) that empower multi-head attention. We saw how they project input embeddings into specialized subspaces, allowing each head to focus on different types of relationships.

However, the self-attention mechanism, even with these projections, has a fundamental blind spot: it is a set operation. It processes all tokens in parallel and doesn't inherently know their order. For example, to a basic self-attention layer, the sentences "man bites dog" and "dog bites man" look like the same bag of words, despite their meanings being entirely opposite. The model has no concept of "first," "second," or "third."

This lesson addresses that gap. Our learning goal is to describe sinusoidal positional encodings and why position information must be explicitly injected. We will explore:

1. Why transformers need an external mechanism for positional information.
2. How sinusoidal functions provide an elegant and effective solution.

Let's begin by solidifying our understanding of the problem.

1. The Problem: Permutation Equivariance

Self-attention is permutation equivariant, meaning if you shuffle the input tokens, the corresponding output embeddings are shuffled in the same way, but their values don't change. This is a problem because sequence order is vital for meaning.

The following short video provides a great visual intuition for why this is a critical issue that needs solving.

To solve this, we must "enrich" the input embeddings with information about their position in the sequence.

As the resources highlight, we need a method that gives each position a unique, deterministic signal, allows the model to understand relative distances, and ideally generalizes to sequences of any length.

2. The Solution: Sinusoidal Positional Encoding

The original "Attention Is All You Need" paper introduced a clever solution using sine and cosine functions. The core idea is to create a unique vector, the positional encoding (PE), for each position in the sequence. This PE vector is then added to the corresponding token's input embedding.

Building the Intuition

Instead of just adding the position number (e.g., 0, 1, 2...), which could lead to arbitrarily large and unstable values, the authors used periodic functions. Think of it like this: for each position, we generate a vector of values by sampling from a set of sine and cosine waves with different frequencies.

• The low-frequency waves change slowly, giving a rough sense of position (e.g., beginning vs. end of a sentence).
• The high-frequency waves change rapidly, providing fine-grained information to differentiate adjacent positions.

This combination of frequencies creates a unique "fingerprint" for each position.

The Formulas

This intuition is captured by the following formulas for a positional encoding vector $PE$ of dimension $d_{\text{model}}$ at position $pos$:

For even-indexed dimensions ($2i$):
$P{E}_{(pos,2i)}=\sin \left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$

For odd-indexed dimensions ($2i+1$):
$P{E}_{(pos,2i+1)}=\cos \left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$

Let's break this down:

• pos: The position of the token in the sequence (0, 1, 2, ...).
• i: The dimension index within the encoding vector (0, 1, 2, ... up to $d_{\text{model}}/2$).
• d_model: The dimension of the embeddings (e.g., 512).
• 10000: A large constant chosen to create a wide range of frequencies, from very high ($i=0$) to very low ($i\approx d_{\text{model}}/2$)).

The result is a matrix where each row is a unique positional vector for a specific position in the sequence.

3. Why Sinusoidal Encoding Works: The Power of Relative Position

We've established what sinusoidal encodings are, but why this specific formulation with paired sine and cosine functions? The answer lies in a key property: it makes it incredibly easy for the model to learn relative positions.

The attention score between two tokens at positions $m$ and $n$ depends on the dot product of their Query and Key vectors. The authors of the original Transformer paper chose this sine/cosine formulation because the positional encoding for pos + k can be expressed as a linear transformation of the encoding for pos.

This means the relationship between any two positions is consistent, regardless of where they are in the sequence. The "distance" from position 5 to 7 is encoded in the same way as the distance from position 50 to 52.

The linear transformation that achieves this is, fascinatingly, a rotation.

This property is crucial. Because the attention score is based on a dot product, and dot products are preserved under rotation, the model can learn to compute attention based on the relative rotation between two positional encodings, which directly corresponds to their relative distance in the sequence.

Let's read a concise summary that crystallizes why this method is so effective.

Conclusion

In this lesson, we tackled the fundamental problem of order in Transformers. We established that because self-attention is permutation-equivariant, we must explicitly inject position information into the model.

Key Takeaways:

• The Problem: Self-attention treats an input sequence as an unordered set of tokens, making it blind to word order.
• The Solution: We add a positional encoding vector to each token's embedding. This vector is unique to its position in the sequence.
• Sinusoidal Encoding: A specific method using sine and cosine functions of varying frequencies to generate these unique positional vectors. The use of sine/cosine pairs is not arbitrary.
• Relative Positions: This formulation allows the positional encoding of pos + k to be represented as a simple rotation of the encoding at pos. This makes it easy for the model to learn relationships based on relative distances between tokens.

Preview of the Next Lesson:
We have now examined all the key components of the vanilla attention mechanism: Q, K, V projections, multi-head attention, and positional encoding. In our next lesson, we will move to Module 3: Transformer Architecture and LLM Basics. We will assemble these pieces into a full Transformer Block, introducing the final architectural elements—residual connections and layer normalization—that make deep Transformers trainable.