Positional Encodings Explained

Hello! Welcome to the fourth lesson in our "LLM and Transformer Fundamentals" module.

In our last lesson, we explored multi-head attention, seeing how it enables a Transformer to analyze input from multiple perspectives simultaneously. However, we left a critical problem unsolved: self-attention, by its nature, is permutation-invariant. It treats a sequence as an unordered "bag" of tokens. To the attention mechanism alone, "a dog chases a cat" and "a cat chases a dog" are indistinguishable.

This lesson tackles that problem head-on. Our learning goal is to describe the role and implementation of positional encodings in capturing sequence order. We'll explore why they are necessary, how the classic sinusoidal approach works, and what makes it such an elegant solution.

1. The Problem: Permutation Invariance

Recurrent Neural Networks (RNNs) naturally process sequence order by consuming tokens one by one. Transformers, in contrast, process all tokens in parallel for speed and to capture long-range dependencies more effectively. This parallelism, however, means the model has no built-in sense of "before" or "after."

To understand why this is a fundamental issue and to see why simple solutions fall short, the following video provides a clear explanation.

As the video highlights, any viable solution must have two key properties:

1. Bounded Values: The positional values should not grow infinitely and overwhelm the token embedding values.
2. Consistency: The encoding for a certain position or relative distance should be consistent and mean the same thing regardless of the overall sequence length.

2. The Solution: Sinusoidal Positional Encodings

The original Transformer paper, "Attention is All You Need," introduced a clever solution using sine and cosine functions of varying frequencies. The idea is to create a unique positional encoding vector for each position in the sequence. This vector is then added to the corresponding token embedding to inject positional information.

The formula for the positional encoding $PE$ for a token at position pos and dimension i of the embedding vector is:

$$P{E}_{(pos,2i)}=\sin \left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$ $$P{E}_{(pos,2i+1)}=\cos \left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$

Where:

• pos is the position of the token in the sequence (0, 1, 2, ...).
• i is the index of the dimension pair within the embedding vector ($i\in [0,d_{\text{model}}/2)$).
• $d_{\text{model}}$ is the dimension of the embedding (e.g., 512).

At first glance, this looks complex. Let's break down the intuition.

3. Building Intuition: Multi-Frequency Waves

The core idea is that each dimension (or pair of dimensions) of the positional encoding corresponds to a wave of a specific frequency.

• Low dimensions (small i) have high-frequency waves (the denominator is small). These values change rapidly and are useful for encoding fine-grained, local positional differences between nearby tokens.
• High dimensions (large i) have low-frequency waves (the denominator is large, approaching 10000). These values change very slowly and are useful for encoding coarse, global positional relationships between distant tokens.

This creates a multi-resolution representation of position, which is a powerful concept. The following video uses some excellent analogies to make this clear.

Here is a visualization of the pattern. You can clearly see the high-frequency waves on the left (low dimensions) and the slow-moving, low-frequency waves on the right (high dimensions).

4. Key Properties and Implementation Details

Now, let's address two key questions an engineer would ask: "Why 10000?" and "Why use both sine and cosine?"

Why the "Magic Number" 10000?

The value 10000 is a hyperparameter that defines the range of wavelengths, from $2\pi$ to $10000\cdot 2\pi$. This was chosen by the authors as a value that worked well, providing a spectrum of frequencies that balanced sensitivity to local differences with the ability to encode long-range relationships across very long sequences.

Why Both Sine and Cosine?

This is perhaps the most elegant part of the design. Using a sin/cos pair for each frequency allows the model to easily learn relative positions. The positional encoding for pos + k can be represented as a linear transformation (specifically, a 2D rotation) of the encoding for pos.

Let $\theta =pos/10000^{2i/d_{\text{model}}}$ and $\varphi =k/10000^{2i/d_{\text{model}}}$. The encoding for pos + k is based on $\sin (\theta +\varphi )$ and $\cos (\theta +\varphi )$. Using trigonometric identities:

$$\cos (\theta +\varphi )=\cos (\theta )\cos (\varphi )-\sin (\theta )\sin (\varphi )$$ $$\sin (\theta +\varphi )=\sin (\theta )\cos (\varphi )+\cos (\theta )\sin (\varphi )$$

This shows that the new encoding is a linear combination of the original sin and cos values. This means that for any offset k, the transformation from $P{E}_{pos}$ to $P{E}_{pos+k}$ is a fixed rotation matrix that is independent of pos. The model can easily learn this transformation, making it simple to understand relative positional relationships like "the word 3 positions ahead".

The final segments of the ExplainingAI video cover these points well.

5. Implementation in Code

Understanding the formula is one thing; implementing it efficiently is another. The following resource walks through a PyTorch implementation, which is very clear and maps directly to the formulas we've discussed.

6. Alternatives and Broader Context

Sinusoidal encodings are elegant because they are fixed (not learned), parameter-free, and can extrapolate to any sequence length. However, they are not the only solution. The field has evolved, and it's useful to know where this original approach fits in.

• Learned Absolute Positional Embeddings: Instead of a fixed formula, we can create a simple embedding matrix of size (max_sequence_length, d_model) and let the model learn the positional vectors during training. This is more flexible but cannot extrapolate beyond the max_sequence_length it was trained on. This approach is used by models like BERT and the original GPT.
• Relative Positional Embeddings: More advanced methods modify the attention mechanism itself to directly incorporate the relative distance between tokens. Rotary Position Embedding (RoPE), used in many modern LLMs like Llama and Mistral, is a popular example that also uses sinusoidal functions but applies them as rotations to the query and key vectors directly.

The following resource provides a superb summary of the trade-offs.

Conclusion

In this lesson, we demystified positional encodings, a crucial component that gives Transformers their sense of sequence order. We saw how the original sinusoidal method provides a clever, parameter-free solution to the permutation-invariance of self-attention.

Key Takeaways:

• Transformers need explicit positional encodings because the self-attention mechanism is permutation-invariant.
• Sinusoidal encodings use a set of sine and cosine functions with different frequencies across the embedding dimensions.
• High-frequency waves capture local, fine-grained order, while low-frequency waves capture global, long-range order.
• The use of a sin/cos pair is critical, as it allows relative positions to be represented by a simple linear transformation (rotation), which is easy for the model to learn.
• Positional encodings are added directly to the token embeddings before they enter the first Transformer layer.
• While effective, sinusoidal encodings are just one approach. Modern models often use learned embeddings or more advanced relative methods like RoPE.

Next Lesson Preview

We have now covered the core components of a Transformer block: input embeddings, multi-head self-attention, and positional encodings. In the next lesson, we will put all these pieces together. We will diagram a full transformer decoder block, explaining the role of each component (attention, feed-forward network, layer normalization, and residual connections), to see how they jointly produce the model's output.