Transformer Architecture: Encoder-Decoder and Positional Encodings

Introduction

In our previous lesson, we delved into the mechanics of multi-head attention, establishing it as the core computational engine that allows a model to consider different types of relationships within a sequence simultaneously. We saw how this mechanism overcomes the single-perspective bottleneck of single-headed attention. Now that we have this powerful building block, we are ready to construct the full edifice of the Transformer.

This lesson addresses a central aspect of the Transformer's design: its overall architecture. Our objective is to describe the high-level Transformer encoder-decoder architecture, including the role of positional encodings. We will assemble the multi-head attention layers into larger structures—the encoder and the decoder—and see how they collaborate on tasks like machine translation. Crucially, we will also address a fundamental problem: since attention is inherently order-agnostic, how does the model know the sequence of the words? This will lead us to the ingenious concept of positional encodings.

The Problem of Order: Permutation Equivariance in Self-Attention

Before we build the architecture, we must address a foundational issue with the self-attention mechanism. By its mathematical definition—a weighted sum of values where weights are derived from query-key similarities—self-attention is permutation equivariant. This means if you shuffle the input tokens, the output vectors are simply shuffled in the same way. The mechanism has no built-in sense of sequence order.

From the model's perspective, the sentences "The dog bit the man" and "The man bit the dog" are just bags of the same tokens. Without positional information, a pure self-attention network cannot distinguish their meanings, which are in fact opposites. This is a critical limitation, as word order is fundamental to meaning in human language.

To solve this, we must explicitly inject information about each token's position into the model. The following article provides a superb explanation of this problem and why it necessitates a solution.

Injecting Order: Positional Encodings

The solution proposed in the original "Attention Is All You Need" paper is to add a positional encoding vector to each token's input embedding. This ensures that the representation for the same token will be different at different positions in the sequence, making the sequence order visible to all subsequent layers.

While several methods for creating these encodings exist, the original paper introduced a fixed, non-learned method using sine and cosine functions of varying frequencies. For a token at position $pos$ and a $d$-dimensional embedding, the positional encoding vector $PE$ is defined as:

Here, $i$ refers to the dimension index within the embedding vector. The intuition is that each dimension of the positional encoding corresponds to a sinusoid. The wavelengths of these sinusoids form a geometric progression, from $2\pi$ to $10000\cdot 2\pi$.

• Low-indexed dimensions ($i$ is small): Have high frequencies (short wavelengths). Their values change rapidly, providing a fine-grained signal to distinguish between adjacent tokens.
• High-indexed dimensions ($i$ is large): Have low frequencies (long wavelengths). Their values change very slowly, providing a coarse signal about the token's general location within a long sequence.

Think of it like an odometer: the rightmost digit spins quickly, while the leftmost digit barely moves on a short trip. Each digit alone is ambiguous, but together they uniquely identify the total distance. Similarly, the combination of these sine and cosine values across all dimensions provides a unique positional signature for each token.

A key theoretical advantage of this formulation is that it allows the model to easily learn relative positioning. The positional encoding for position $pos+k$ can be expressed as a fixed linear transformation of the encoding for position $pos$. This property is highly desirable, as many linguistic dependencies (like subject-verb agreement) are based on relative, not absolute, positions.

The following resource provides a fantastic deep dive into the mathematics and intuition behind sinusoidal positional encodings.

While sinusoidal encodings are foundational, it's worth noting that later models like BERT and GPT use learned absolute positional embeddings, and modern architectures like LLaMA and Mistral use a more advanced technique called Rotary Positional Encoding (RoPE) that integrates position information directly into the attention calculation.

The Full Encoder-Decoder Architecture

With our attention mechanism defined and the problem of order solved, we can now assemble the complete architecture. The original Transformer is an encoder-decoder model, designed for sequence-to-sequence tasks like machine translation.

Let's break down each part of this diagram.

The Encoder Stack

The encoder's job is to read the entire source sequence and produce a set of rich, contextualized representations. It consists of a stack of $N$ identical encoder layers (in the original paper, $N=6$).

The process begins with the input embeddings:

1. Input: The source sentence is tokenized, and each token is converted to a vector using a learned token embedding layer.
2. Positional Encoding: The positional encoding vector for each token's position is added to its token embedding.

This combined representation is then fed through the stack of $N$ encoder layers. Each encoder layer has two main sub-layers:

1. Multi-Head Self-Attention: This is exactly the mechanism we studied previously. Each token in the sequence attends to every other token in the sequence, allowing information to be exchanged and contextual representations to be formed.
2. Position-wise Feed-Forward Network (FFN): This is a simple two-layer fully connected network that is applied independently to each token's representation. Its purpose is to introduce additional non-linearity and computational capacity. It can be thought of as letting each token "digest" the information it gathered during the attention step.

Crucially, both of these sub-layers are wrapped with a residual connection followed by layer normalization (the "Add & Norm" boxes). We will explore these components in detail in the next lesson, but for now, their purpose is to enable stable training of very deep networks.

After passing through all $N$ layers, the encoder outputs a final set of contextualized vectors, $K_{enc}$ and $V_{enc}$, one for each input token. This output represents the model's "understanding" of the source sentence.

The Decoder Stack

The decoder's job is to generate the target sequence one token at a time, in an autoregressive fashion. It also consists of a stack of $N$ identical decoder layers.

The input to the decoder is the target sequence generated so far. A decoder layer is similar to an encoder layer but has three sub-layers:

1. Masked Multi-Head Self-Attention: The decoder performs self-attention on the target sequence it has generated so far. However, to prevent it from "cheating" by looking ahead at tokens it hasn't predicted yet, a causal mask is applied. This mask ensures that any given token can only attend to previous tokens (and itself), not future ones.
2. Multi-Head Cross-Attention: This is the critical link between the encoder and the decoder. In this sub-layer, the queries ($Q_{dec}$) come from the decoder's previous sub-layer, but the keys ($K_{enc}$) and values ($V_{enc}$) come from the final output of the encoder stack. This allows every token being generated by the decoder to attend to all tokens in the source sentence, enabling it to align source and target content.
3. Position-wise Feed-Forward Network (FFN): This is identical in structure and function to the FFN in the encoder.

Again, each of these three sub-layers has a residual connection and layer normalization. After the final decoder layer, a linear transformation followed by a softmax function produces a probability distribution over the entire vocabulary, from which the next token is sampled.

Autoregressive Inference

It's important to distinguish how the model works during training versus inference. During training, the entire correct target sequence (shifted by one position) is fed into the decoder at once, a technique called "teacher forcing." This allows for massive parallelization.

During inference, however, generation is sequential and autoregressive.

1. The encoder processes the source sentence once to get $K_{enc}$ and $V_{enc}$.
2. The decoder starts with a special [START] token.
3. It generates the first target token.
4. This new token is appended to the decoder's input sequence.
5. The decoder runs again with the updated sequence to generate the next token.
6. This loop continues until a special [END] token is generated.

The following video provides an excellent animated walkthrough of this step-by-step generation process.

Conclusion

In this lesson, we have assembled the complete encoder-decoder Transformer architecture. We moved from the core computational block of multi-head attention to a full sequence-to-sequence model.

Here are the key takeaways:

• Positional Encoding: Self-attention is permutation-equivariant and requires an external mechanism to understand word order. Sinusoidal positional encodings provide a unique positional signature for each token and have the elegant property of representing relative positions via linear transformations.
• Encoder: Its role is to process the entire input sequence bidirectionally using stacked layers of self-attention and feed-forward networks, producing a rich, contextualized representation.
• Decoder: Its role is to generate the output sequence autoregressively. It uses masked self-attention to preserve the autoregressive property and cross-attention to consult the encoder's output at every step.
• Cross-Attention: This is the bridge connecting the encoder's "understanding" of the source to the decoder's "generation" of the target. Queries come from the decoder, while keys and values come from the encoder.

You now have a high-level map of the original Transformer. In our next lesson, we will zoom back in on the building blocks and detail the other components of a Transformer block: the feed-forward networks, residual connections, and layer normalization, to understand their theoretical justifications and practical importance.