RoPE: Relative Attention for Extrapolatable Context

1. The Key Property: From Absolute Rotations to Relative Attention

Let's begin by formalizing the core claim. We need to prove that the dot-product attention score between a query at position $m$ and a key at position $n$ depends only on their relative distance, $m-n$.

Recall from our previous lesson that applying RoPE to a vector $\mathbf{x}$ at position $p$ is equivalent to multiplying it by a 2D rotation matrix, which we can denote as ${\mathbf{R}}_{p\theta }$. For a high-dimensional vector, this operation is applied to pairs of dimensions, resulting in a block-diagonal rotation matrix we'll call ${\mathbf{R}}_p$.

So, the transformed query and key vectors are:

• ${\mathbf{q}}_m^{\prime }={\mathbf{R}}_m{\mathbf{q}}_m$
• ${\mathbf{k}}_n^{\prime }={\mathbf{R}}_n{\mathbf{k}}_n$

The attention score is their dot product:
$\text{Score}(m,n)=⟨{\mathbf{q}}_m^{\prime },{\mathbf{k}}_n^{\prime }⟩=({\mathbf{R}}_m{\mathbf{q}}_m{)}^T({\mathbf{R}}_n{\mathbf{k}}_n)$

Using the matrix transpose rule $(AB{)}^T=B^T{A}^T$, we get:
$\text{Score}(m,n)={\mathbf{q}}_m^T{\mathbf{R}}_m^T{\mathbf{R}}_n{\mathbf{k}}_n$

Now, let's use two fundamental properties of rotation matrices:

1. They are orthogonal, which means their transpose is equal to their inverse: ${\mathbf{R}}_m^T={\mathbf{R}}_m^{-1}$.
2. The inverse of a rotation by angle $m\theta$ is a rotation by angle $-m\theta$. So, ${\mathbf{R}}_m^{-1}={\mathbf{R}}_{-m}$.

Substituting these back into our equation:
${\mathbf{R}}_m^T{\mathbf{R}}_n={\mathbf{R}}_m^{-1}{\mathbf{R}}_n={\mathbf{R}}_{-m}{\mathbf{R}}_n={\mathbf{R}}_{n-m}$
The product of a rotation by $n\theta$ and a rotation by $-m\theta$ is a single rotation by $(n-m)\theta$.

This gives us the final, elegant result for the attention score:
$\text{Score}(m,n)={\mathbf{q}}_m^T{\mathbf{R}}_{n-m}{\mathbf{k}}_n$

As you can see, the score is a function of the original query and key vectors (${\mathbf{q}}_m,{\mathbf{k}}_n$) and a rotation matrix that depends only on the relative position, $n-m$. This completes the proof.

2. The Payoff: Context Length Extrapolation

We've established that RoPE makes attention scores relative. Why is this a game-changer for building large language models? It directly addresses the problem of context length extrapolation.

Models with absolute positional embeddings (APE) learn a unique embedding for each specific position (e.g., position #1, #2, ..., #512). If, during inference, the model encounters a token at position #513, it has no learned embedding for it and its performance degrades catastrophically. The model simply hasn't seen positions that far out.

RoPE sidesteps this entirely. Because the attention score only depends on the relative distance k = n-m, the model learns to interpret the meaning of "5 tokens away" or "100 tokens away" regardless of where in the sequence this occurs.

To generate a token at position 2049 attending to a token at 2040, the model computes an attention score based on the relative distance k = 9. It has seen this relative distance countless times during training (e.g., between positions 9 and 0, 109 and 100, etc.). The model can generalize its understanding of relative distances to absolute positions far beyond its training window.

To solidify this intuition, please read the following brief explanation.

3. Long-Term Decay and Multi-Scale Representation

The final piece of the puzzle is understanding how RoPE effectively models both short- and long-range dependencies. The key lies in using different rotation frequencies (${\theta }_i$) for different dimension pairs, a concept we touched on in the last lesson.

• High-frequency pairs (fast rotation): The angle $m{\theta }_i$ changes rapidly with position $m$. These dimensions are sensitive to small changes in relative distance, making them excellent for capturing relationships between nearby words.
• Low-frequency pairs (slow rotation): The angle $m{\theta }_i$ changes very slowly with $m$. Even for tokens that are far apart, their embeddings in these dimensions will have similar rotations, allowing the model to capture long-range dependencies.

This combination of frequencies creates a natural long-term decay effect. As the relative distance between tokens increases, the high-frequency dimensions become increasingly decorrelated, causing the overall attention score to decrease. However, the low-frequency dimensions maintain some correlation, preserving the model's ability to attend to very distant but relevant tokens.

The following video provides an excellent visual and intuitive explanation of this multi-scale representation.

This GIF provides a concise visualization of the long-term decay property discussed in the video and the RoFormer paper.

Conclusion

In this lesson, we've connected the mechanism of RoPE to its most important properties.

Key Takeaways:

• Because RoPE is implemented as a rotation, the attention score between tokens at positions $m$ and $n$ simplifies to a function of their relative position $n-m$.
• This property is the key to context length extrapolation, as the model learns to understand relative distances that can be applied to absolute positions beyond its training window.
• By using a spectrum of rotation frequencies, RoPE can model both short-range (via fast-rotating dimensions) and long-range (via slow-rotating dimensions) dependencies.
• This multi-scale representation creates a natural long-term decay in attention scores with distance, which is a desirable inductive bias.

Preview of the Next Module:
This concludes our foundational module on the modern Transformer stack. We now have a solid grasp of the key components: RMSNorm, SwiGLU, and RoPE.

In the next module, "Efficient Attention," we will tackle one of the biggest bottlenecks in deploying LLMs: the computational and memory cost of the attention mechanism itself. Our first lesson will quantify the memory usage of the KV Cache and introduce Grouped-Query Attention (GQA), the first step on our path to understanding the highly efficient architectures used in models like DeepSeek-V3.