Absolute Positional Encodings: Role & Implementation

Introduction

In our last lesson, we built the powerful multi-head attention mechanism. You now understand how the Transformer can process a sequence by simultaneously focusing on different types of relationships from multiple "perspectives" or subspaces. We have a model that can weigh the importance of tokens based on their content.

However, there's a fundamental piece missing. A core property of the self-attention mechanism is that it's permutation-invariant. It treats the input as a "bag of words" rather than an ordered sequence. If you feed it "The dog chased the cat" versus "The cat chased the dog," the attention scores would be calculated based on the vector similarities, without any inherent knowledge of which word came first.

Today's lesson addresses this critical gap. Our learning outcome is to describe the role and implementation of absolute positional encodings. We will explore how to inject information about the order of tokens directly into their representations, enabling the Transformer to understand sequences. We will focus on the clever sinusoidal method proposed in the original paper and touch upon a simpler, learnable alternative.

The Problem: A Model Without a Sense of Time or Order

The operations at the heart of self-attention—matrix multiplications of Queries, Keys, and Values—do not depend on the order of the rows (tokens) in the input matrices. This design choice enables massive parallelization, a key advantage over sequential models like RNNs. However, it comes at the cost of losing all sequence information.

To build a solid intuition for why this is a problem and how it arises from the self-attention mechanism, the following article provides an excellent explanation.

The solution is to explicitly add information about the position of each token to its representation. If position is part of the vector itself, the model can learn to use it when calculating attention.

Sinusoidal Positional Encoding: The Classic Solution

The authors of the "Attention Is All You Need" paper introduced an elegant solution: a fixed, deterministic positional encoding matrix that is added to the input embeddings. The idea is to create a unique positional vector for each position in the sequence using a combination of sine and cosine functions with different frequencies.

First, let's get a high-level overview of the desired properties of any good positional encoding scheme.

The Formula

As the video hinted, the sinusoidal method satisfies all four criteria. It does so using the following pair of equations for the positional encoding $PE$ at a given position pos and for each dimension i of the embedding vector:

Where:

• pos is the token's position in the sequence (e.g., 0, 1, 2, ...).
• i is the index of the dimension within the embedding vector. We use 2i for even dimensions and 2i+1 for odd dimensions.
• d_model is the dimension of the embeddings (e.g., 512).

This looks complex, but the intuition is straightforward:

• Alternating Functions: For each position, the encoding vector is made of pairs of sine and cosine values.
• Varying Frequencies: As you go across the embedding dimensions (as i increases), the wavelength of the sine/cosine functions increases exponentially. Dimensions at the beginning of the vector vary very quickly (high frequency), while dimensions at the end vary very slowly (low frequency).

This design is analogous to binary representation. The least significant bit flips every number (0, 1, 0, 1...), the next bit flips every two numbers (0, 0, 1, 1...), and so on. Similarly, the high-frequency sinusoids encode fine-grained positional information, while the low-frequency ones encode coarse-grained information.

For a comprehensive look at the mathematics and implementation, the "Dive into Deep Learning" book provides a canonical reference.

The ability to represent relative positions via a linear transformation is a profound property. It means that the model can easily learn to attend to tokens that are "k steps away" regardless of the absolute position i. This is because the transformation matrix for a given offset k is the same for all positions.

Implementation in PyTorch

Now let's translate this into a PyTorch nn.Module. Your experience with production code will make the following implementation feel clean and efficient. We pre-calculate the entire positional encoding matrix up to a max_len and register it as a buffer—a tensor that is part of the model's state but is not a trainable parameter.

import torch
import torch.nn as nn
import math

class PositionalEncoding(nn.Module):
    def __init__(self, d_model: int, dropout: float = 0.1, max_len: int = 5000):
        super().__init__()
        self.dropout = nn.Dropout(p=dropout)

        # Create a matrix of shape (max_len, d_model)
        position = torch.arange(max_len).unsqueeze(1) # (max_len, 1)
        div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model)) # (d_model / 2)
        
        pe = torch.zeros(max_len, 1, d_model)
        pe[:, 0, 0::2] = torch.sin(position * div_term)
        pe[:, 0, 1::2] = torch.cos(position * div_term)
        
        # Register pe as a buffer, not a parameter
        self.register_buffer('pe', pe)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """
        Args:
            x: Tensor, shape [seq_len, batch_size, d_model]
        """
        # Add positional encoding to the input tensor
        # self.pe is (max_len, 1, d_model). We slice it to the input sequence length.
        x = x + self.pe[:x.size(0)]
        return self.dropout(x)

Note: The code assumes batch_first=False, a common convention in older PyTorch NLP tutorials. For batch_first=True models ([batch_size, seq_len, d_model]), you would adjust the forward pass accordingly.

To see a detailed, line-by-line construction of this in a live-coding style, the following video is an excellent resource.

Alternative: Learned Positional Encodings

While the sinusoidal method is clever, a simpler and surprisingly effective alternative exists: learned positional encodings. This is the approach used by models like GPT.

The idea is to treat the position as just another vocabulary to be embedded. You create a standard nn.Embedding layer that maps a position index (0, 1, 2, ...) to a d_model-dimensional vector.

The implementation is very straightforward.

# In your model's __init__
self.pos_embedding = nn.Embedding(max_seq_len, d_model)

# In your model's forward pass
positions = torch.arange(0, seq_len, device=x.device).unsqueeze(0) # Shape: (1, seq_len)
positional_embeddings = self.pos_embedding(positions) # Shape: (1, seq_len, d_model)
x = x + positional_embeddings

• Pros: Extremely simple to implement. The model can potentially learn an optimal positional representation directly from the data.
• Cons: It cannot extrapolate to sequences longer than max_seq_len seen during training, as there is no learned embedding for those positions. The sinusoidal method, being a continuous function, can generalize to any length.

For a clear implementation of this approach, we return to the substack article.

Conclusion

You have now tackled a crucial and non-obvious aspect of the Transformer architecture. Without a way to encode position, the model would be fundamentally handicapped.

Key Takeaways:

• Role: Positional encodings are necessary to overcome the permutation-invariance of the self-attention mechanism, injecting vital information about token order into the model.
• Absolute Encoding: The methods we discussed today (sinusoidal and learned) are absolute, meaning they encode the specific position of a token (e.g., 1st, 2nd, 3rd) in the sequence.
• Sinusoidal Implementation: This classic method uses a fixed set of sine and cosine functions with geometrically increasing wavelengths. Its design allows it to generalize to unseen sequence lengths and makes it easy for the model to learn about relative positions.
• Learned Implementation: A simpler alternative uses a standard nn.Embedding layer to learn a unique vector for each position. It is effective but does not extrapolate beyond the maximum length seen during training.

What's Next?

We have now assembled two of the three key components of a Transformer block: multi-head attention and positional encodings. In the next lesson, we will examine the third component: the Position-wise Feed-Forward Network (FFN). You'll learn why this seemingly simple two-layer network is critical for transforming the representations produced by the attention mechanism and adding non-linear capacity to each block.