Positional Encodings: Understanding Sequence Order

The Problem of Order-Blindness

Recurrent Neural Networks (RNNs) naturally process sequences token-by-token, so they have a built-in sense of order. Transformers, however, process all tokens in a sequence at once to achieve massive parallelization. This speed comes at the cost of losing positional awareness.

Let's start by watching a short video that clearly articulates this problem and sets the stage for the solution.

What Makes a Good Positional Signal?

So, how can we signal the position of each word to the model? Before jumping to the solution, let's consider what properties an ideal positional signal should have.

As the video outlines, we need a method that:

1. Provides a unique encoding for each position.
2. Is deterministic (not random), so it's consistent.
3. Allows the model to easily determine the relative distance between tokens.
4. Can generalize to sequences longer than those seen during training.

Simple approaches, like using integer indices (1, 2, 3...) or normalizing them (e.g., 0.1, 0.2, 0.3...), fail on one or more of these criteria, particularly the last two. Integers can grow unboundedly, causing training instability, while simple normalization makes the distance between positions dependent on the total sentence length.

The Sinusoidal Solution

The authors of the original "Attention Is All You Need" paper proposed a brilliant solution using sine and cosine functions. Instead of a single number, each position in the sequence is mapped to a high-dimensional vector, called the positional encoding vector.

The values in this vector are generated by the following formulas:

$P{E}_{(pos,2i)}=\sin \left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$
$P{E}_{(pos,2i+1)}=\cos \left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$

Let's break down these components:

• pos: The position of the token in the sequence (e.g., 0, 1, 2, ...).
• d_model: The dimension of the embedding vectors (e.g., 512).
• i: The index of the dimension within the encoding vector, from 0 to d_model/2.

The core idea is that for each position pos, we create a vector of size d_model. Each pair of dimensions (2i and 2i+1) in this vector corresponds to a sine and cosine wave of a different frequency.

• When i is small, the wavelength is short (high frequency). The values change rapidly with pos.
• When i is large, the wavelength is long (low frequency). The values change very slowly with pos.

This approach is analogous to binary representation, where the first bit flips every number, the second every two numbers, and so on. Here, we use smooth, continuous sinusoids instead of discrete 0s and 1s, which is much more suitable for gradient-based learning.

The Power of Relative Positioning

The true genius of this method isn't just that it creates unique vectors. It's that the resulting pattern makes it easy for the model to learn about relative positions.

Because of the trigonometric identities (e.g., sin(A+B) and cos(A+B)), the positional encoding for a future position pos + k can be represented as a linear transformation of the encoding for pos. The model doesn't need to know this formula explicitly; it just needs to learn the transformation. This means the model can learn what "three steps forward" looks like in the positional encoding space, regardless of the starting position.

Let's watch a detailed explanation of this crucial property.

Integrating Positional and Semantic Information

Now that we have a vector representing the word's meaning (the word embedding) and a vector representing its position (the positional encoding), how do we combine them? The Transformer does this with simple element-wise addition.

You might wonder: won't adding these two vectors together jumble their information? This is a common point of confusion.

The reason it works is that the positional encodings have a very regular, wave-like structure that is distinct from the learned semantic patterns in the word embeddings. The model can learn to distinguish these two sources of information. Furthermore, this method is computationally efficient because it doesn't increase the size of the input vectors, unlike concatenation.

The following video segment addresses this exact question.

To reinforce this, you can also read a very clear, text-based explanation of the same concept.

Conclusion

In this lesson, we've solved the "order-blindness" problem of the self-attention mechanism. By injecting a clever, mathematically structured signal, we give the Transformer a sense of sequence order without sacrificing its parallel processing power.

Key Takeaways:

• The Problem: The parallel nature of Transformers makes them inherently unaware of sequence order, which is critical for language.
• The Solution: Positional encodings are fixed, deterministic vectors that are added to word embeddings to inject this order information.
• Sinusoidal Method: The standard approach uses a combination of sine and cosine functions of varying frequencies to generate a unique vector for each position.
• Relative Positioning: The most important property of this method is that it creates a predictable pattern, allowing the model to easily learn relative positions (e.g., "three words ahead").
• Efficient Integration: The positional vectors are added to the word embeddings, an efficient method that combines semantic and positional information without increasing the model's dimensionality.

Preview of the Next Lesson:

We have now covered all the core components of the Transformer's encoder: the input processing with embeddings and positional encodings, and the context-building power of multi-head self-attention. In our final lesson of this module, we will outline the complete architecture of a Transformer, showing how these pieces, along with feed-forward layers and residual connections, fit together to form the powerful building block of modern NLP.