Understanding Positional Encoding

Hello! Welcome to the fourth lesson in our module on the fundamentals of Transformers.

In our previous lesson, we saw how raw text is transformed into a sequence of meaningful numerical vectors through tokenization and embedding. We now have a list of vectors like [vector_for_token_1, vector_for_token_2, ...], where each vector captures the semantic meaning of a token.

However, this process has a critical side effect. A sentence like "The dog chased the cat" and "The cat chased the dog" would be represented by the same collection of embedding vectors, just in a different order. The model, as it stands, has lost the crucial information about word order. This lesson addresses that exact problem, covering the learning outcome: Describe the purpose of positional encoding in sequence models.

We will explore how Transformers, which process all input tokens in parallel, re-introduce this vital sequential information.

1. The Problem: Order Blindness in Parallel Processing

Traditional sequence models like Recurrent Neural Networks (RNNs) process text token by token, from left to right. This sequential nature inherently preserves word order. The hidden state for "chased" depends on the state from "dog," which depends on "The."

Transformers, however, were designed for massive parallelization to improve training speed and capture long-range dependencies more effectively. They process all token embeddings simultaneously. While this is highly efficient, it creates a fundamental challenge: the model has no built-in sense of which token came first, second, or last. It sees the input as an unordered "bag of vectors."

To understand this problem more deeply, let's read a short introduction from a well-explained article on the topic.

As the article highlights, without a way to represent position, the model can't distinguish between grammatically and semantically different sentences that happen to use the same words. The solution is to explicitly inject positional information into the input embeddings. This mechanism is called positional encoding.

2. The Solution: Adding a Positional Signature

The core idea is simple: for each token embedding in the sequence, we add another vector of the same dimension that encodes its position.

The crucial question is: how do we generate these positional encoding vectors? A naive approach, like adding the position index (0, 1, 2, ...), has problems: the numbers can grow indefinitely, and the model may not generalize to sequence lengths it hasn't seen during training.

The authors of the original Transformer paper, "Attention Is All You Need," devised a clever, elegant solution using sine and cosine functions. Let's watch a video that builds the intuition behind this method.

3. The Sinusoidal Formula

As the video explained, the method uses a pair of sine and cosine functions with different frequencies for each dimension of the embedding vector. This creates a unique positional signature for each position in the sequence.

The formulas for the positional encoding (PE) for a token at position pos and dimension i are:

$$PE(pos,2i)=\sin \left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$ $$PE(pos,2i+1)=\cos \left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$

Let's break this down:

• pos: The position of the token in the sequence (0, 1, 2, ...).
• i: The index for the dimension within the embedding/positional vector. It goes from 0 up to d_model / 2. We use 2i for even dimensions and 2i+1 for odd dimensions.
• d_model: The dimension of the embedding (e.g., 768).

The key is the term in the denominator. As i increases (i.e., as we move to higher dimensions of the vector), the wavelength of the sine/cosine waves gets exponentially larger.

• For small i (early dimensions), the waves have high frequencies, meaning their values change rapidly with position. This helps encode fine-grained information about relative positions.
• For large i (later dimensions), the waves have very low frequencies, so their values change slowly across the sequence. This helps encode coarser, more global position information.

This combination of frequencies creates a unique vector for each position. The resulting positional encoding matrix can be visualized as a heatmap.

4. Why This Method? Key Properties

The choice of sinusoidal functions is not arbitrary. It provides several powerful properties that are ideal for this task. The video from BrainDrain gives a concise overview of these properties.

To expand on these points, let's dive into a textual explanation of the intuition and advantages.

Let's summarize the key advantages discussed in the resources:

1. It's Deterministic and Parameter-Free: The encoding is calculated using a fixed formula. It doesn't require any training, which saves computational resources and reduces model complexity.
2. It Generalizes to Unseen Sequence Lengths: Because the encoding is a continuous function of position, the model can compute encodings for sequences longer than any it encountered during training. This is a significant advantage over learned positional embeddings.
3. It Encodes Relative Positions: This is perhaps the most powerful property. Due to trigonometric identities, the positional encoding for pos + k can be represented as a linear transformation of the encoding for pos. This means the model can easily learn to attend to relative positions (e.g., "the word 3 positions to my left"), which is often more important than absolute position.

A Note on Learned Positional Encodings

As mentioned in the article, an alternative is to have the model learn the positional embeddings, just like it learns word embeddings. This is an nn.Embedding layer in PyTorch where the input is the position index. Models like BERT and GPT-2 use this approach.

• Pro: Can adapt to the specific training data.
• Con: Adds trainable parameters, struggles to generalize to sequence lengths beyond what was seen in training, and may overfit.

The field continues to evolve, with newer methods like Rotary Positional Embeddings (RoPE), used in LLaMA models, which apply rotations to embeddings instead of adding to them. For our current scope, understanding the original sinusoidal method is the most important foundation.

Conclusion

In this lesson, we tackled the "order blindness" of the Transformer architecture. You now understand how positional encoding solves this problem by injecting a unique, position-dependent signal into each token's embedding.

Key Takeaways:

• Transformer models process tokens in parallel, which makes them efficient but unable to inherently understand word order.
• Positional encoding adds a vector to each token's embedding to provide information about its absolute and relative position in the sequence.
• The original Transformer uses a clever sinusoidal function with varying frequencies across the embedding dimensions.
• This method is deterministic, requires no training, and allows the model to generalize to different sequence lengths while easily learning relative positional relationships.

Now that our input vectors contain both semantic meaning (from the embedding) and sequential order (from the positional encoding), we are ready for the next crucial step. In our next lesson, we will dive into the heart of the Transformer: the self-attention mechanism. This is where the model uses this rich information to weigh the importance of every other token in the sequence when processing a given token.