Positional Encoding in Transformers

Hello! Welcome back to our exploration of the Transformer architecture.

In our last lesson, we delved into multi-head attention, seeing how it allows the model to capture diverse contextual relationships by running multiple attention mechanisms in parallel. We concluded with a critical observation: the self-attention mechanism is permutation-invariant. This means if you shuffle the words in a sentence, the attention scores between any two words remain the same. The model has no inherent sense of sequence order, which is a major problem for understanding language where order is paramount.

Today, we will solve this puzzle. This lesson addresses the learning outcome: Explain the role of positional encoding and why it is needed in the Transformer architecture. We will explore the ingenious method Transformers use to inject information about the position of each token directly into its embedding.

1. The Problem: A Model Without a Sense of Time or Order

Recurrent Neural Networks (RNNs) naturally process sequences token by token, so the order is baked into their very structure. Transformers, by contrast, process all tokens simultaneously to achieve massive parallelization. This speed comes at the cost of sequential awareness. Without an explicit way to encode position, a Transformer would see "the cat sat on the mat" and "mat the on sat cat the" as the same collection of words, unable to grasp their fundamentally different meanings.

To give the model a sense of order, we need to provide a unique "signature" for each position in the sequence.

As the video highlights, an ideal positional encoding scheme should meet several criteria:

• It must produce a unique encoding for each position.
• The values should be bounded to not overwhelm the token's semantic information.
• It should be deterministic and consistent across sequences of different lengths.
• The model should be able to generalize to sequences longer than those seen during training.

2. The Solution: Sinusoidal Positional Encodings

The original Transformer paper, "Attention is All You Need," introduced a clever solution using sine and cosine functions of varying frequencies. The idea is to create a $d_{\text{model}}$-dimensional vector for each position, which is then added to the corresponding token embedding.

This diagram provides a great overview of where positional encoding fits into the architecture.

The formula to calculate the positional encoding (PE) for a token at position pos and dimension i of the embedding vector is as follows:

$$P{E}_{(pos,2i)}=\sin \left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$ $$P{E}_{(pos,2i+1)}=\cos \left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$

Let's break this down:

• pos: The position of the token in the sequence (0, 1, 2, ...).
• i: The index of the dimension within the embedding vector (0, 1, ..., $d_{\text{model}}/2-1$).
• $d_{\text{model}}$: The total dimension of the embedding (e.g., 512).

Notice that for each position pos, we get a full vector of $d_{\text{model}}$ values. The formulas apply to pairs of dimensions (even indices $2i$ use sine, odd indices $2i+1$ use cosine). As i increases, the denominator gets smaller, causing the frequency of the wave to decrease.

This creates a unique positional signature where:

• Low-frequency waves (later dimensions) encode coarse, long-range positional information.
• High-frequency waves (early dimensions) encode fine-grained, local positional differences.

The combination of sine and cosine waves across different frequencies ensures that every position has a unique vector, and these vectors change smoothly from one position to the next.

Here is a visualization of what the resulting positional encoding matrix looks like for a short sentence.

3. The Power of the Sinusoidal Formula

You might be wondering: why this specific, seemingly complex formulation? The choice of sine and cosine functions provides a crucial mathematical property that makes it easy for the model to learn about relative positions.

For any fixed offset k, the positional encoding at position pos + k can be represented as a linear transformation of the positional encoding at position pos. This means the model doesn't have to learn the positional encoding for every absolute position from scratch; instead, it can learn the linear transformation that corresponds to a relative shift of k positions. This is incredibly powerful for capturing patterns like "the verb is often 3 words after the noun."

Since you have a strong mathematical background, let's examine the proof for this property.

The key insight from the proof is that for any pair of dimensions corresponding to a single frequency ${\omega }_k$, the transformation from position $t$ to $t+\varphi$ is a simple 2D rotation. The rotation matrix $M$ depends only on the offset $\varphi$ and the frequency ${\omega }_k$, not on the absolute position $t$. This makes learning relative positions extremely efficient for the model.

4. Alternatives and Integration

The sinusoidal method is a fixed, deterministic approach. An alternative, used in models like BERT and GPT, is to use learnable absolute positional embeddings. In this method, a matrix of positional embeddings is created and its values are learned during training just like other model weights. While simpler to implement, it may not generalize as well to sequence lengths not seen during training.

Finally, how is this positional information combined with the semantic information from the token embedding? As we've seen, it's a simple element-wise addition:

final_embedding = token_embedding + positional_encoding

The model learns to use this combined embedding, where part of the vector space represents semantics and another part represents position. The residual connections in the Transformer architecture ensure that this positional information is not lost as the data flows through the deeper layers of the network.

Conclusion

In this lesson, we tackled the fundamental problem of order in the Transformer architecture. We saw that the model's parallel nature makes it blind to sequence order, a limitation we overcame by explicitly injecting positional information.

Key Takeaways:

• The Transformer's self-attention mechanism is permutation-invariant, meaning it cannot distinguish word order on its own.
• Positional encoding is a vector added to each token's embedding to provide information about its absolute position in the sequence.
• The sinusoidal positional encoding method uses sine and cosine functions of varying frequencies to create a unique and bounded signature for each position.
• A key advantage of the sinusoidal approach is that the encoding for a relative position pos + k can be represented as a linear transformation of the encoding at pos, making it easy for the model to learn relative positional relationships.

Preview of the Next Lesson

We have now examined the three core components of a Transformer block: multi-head attention, the feed-forward network (from a previous lesson), and positional encoding. In our next lesson, we will assemble these pieces to construct a complete Transformer encoder block, and see how they work together with residual connections and layer normalization to process sequences.