Decoder-Only Transformers for Generation

Introduction

In our last lesson, we established a crucial distinction between self-attention and cross-attention, identifying their specific roles within the original encoder-decoder Transformer. You now understand that self-attention builds context within a sequence, while cross-attention creates a bridge between two sequences. This fundamental insight paves the way for understanding the major families of Transformer architectures, which are often defined by which of these mechanisms they employ.

This lesson focuses on the first of these families: the decoder-only architecture. This is the architectural blueprint for the Generative Pre-trained Transformer (GPT) series and most of the large language models you interact with today. Our goal is to dissect this model, understand its components, and grasp why its specific design makes it exceptionally well-suited for generative tasks like writing articles, composing emails, or creating code.

From Two Stacks to One: The Birth of the Decoder-Only Model

The original Transformer was designed for sequence-to-sequence tasks like translation, requiring both an encoder (to understand the source language) and a decoder (to generate the target language). However, many language tasks don't involve a separate source sequence. Tasks like text completion, summarization of a provided text, or question-answering based on a given context all operate on a single, continuous stream of text.

For these tasks, the encoder and the cross-attention mechanism that connects it to the decoder become superfluous. If we strip away the entire encoder stack and the cross-attention sub-layer from each decoder block, we are left with a more streamlined architecture: a stack of modified decoder blocks. This is the essence of a decoder-only model.

This architecture processes a single input sequence and is trained to predict the next token at each position. Because of this inherent "what comes next?" objective, it is also called a causal or autoregressive language model.

The diagram below illustrates the high-level structure. Notice its directness: input tokens are processed through a series of identical blocks, culminating in the prediction of output tokens. There is no separate encoder stack.

A high-level schematic of a decoder-only transformer. Input token vectors, augmented with positional information, are passed through a stack of identical decoder blocks. The output of the final block is used to predict the next token in the sequence.

The Core Component: Causal (Masked) Self-Attention

Without an encoder to provide context via cross-attention, a decoder-only model must rely entirely on self-attention. However, for a generative model that predicts tokens sequentially, we must impose a critical constraint: the prediction for a token at position i can only depend on the known tokens at positions 1 to i. It cannot look ahead to tokens at i+1, i+2, etc., as that would be "cheating" during training and impossible during real-world generation.

This constraint is enforced by causal masking, also known as masked self-attention. The mechanism is identical to the masked self-attention we saw in the decoder of the original Transformer. It prevents each token from attending to any subsequent tokens in the sequence.

A comparison of attention patterns. Standard self-attention (left) is bidirectional, allowing a token to attend to all other tokens in the sequence. Masked self-attention (right) is unidirectional or causal, allowing a token to attend only to itself and preceding tokens.

How is this masking implemented efficiently? A naive approach might be to compute all attention scores and then zero out the ones corresponding to future tokens. However, this has a flaw. The softmax function normalizes scores across the entire row, so the scores of future tokens would have already influenced the denominator (the normalization factor) for the scores of past tokens, causing a "data leakage" from the future.

The correct and standard implementation is more clever. Before the softmax is applied, the attention scores for future positions are set to negative infinity ( $- \infty$ ). When the softmax function exponentiates these scores, $e^{- \infty}$ becomes zero, effectively nullifying their contribution both in the numerator and the denominator.

To see a detailed walkthrough of this crucial concept, let's turn to the video "Understanding causal attention or masked self attention."

Understanding causal attention or masked self attention | Transformers for vision series

This video from Vizuara provides an excellent explanation of why masking is necessary for generative models and how it's implemented correctly to avoid data leakage.

First, watch the segment from the introduction to causal attention to understand the core motivation: a token's context should not include future tokens. The video uses a triangular matrix to visualize which attention scores are allowed. Next, watch the critical explanation from on preventing data leakage. This part details why simply zeroing-out scores after softmax is wrong and explains the elegant solution of setting scores to negative infinity before the softmax operation.

This implementation of causal attention is the absolute heart of a decoder-only model, enabling its powerful generative capabilities.

The Anatomy of a GPT-style Model

With the core mechanism understood, let's assemble the full architecture. The StatQuest video "Decoder-Only Transformers, Clearly Explained!!!" provides a fantastic step-by-step construction of a model like ChatGPT. We will use it as our guide.

1. Input Processing: Token and Position Embeddings

First, the raw input text is converted into a sequence of numerical vectors.

Tokenization & Token Embedding: The text is broken down into tokens, and each token is mapped to a unique dense vector via a trainable embedding matrix E.
Positional Encoding: Since the self-attention mechanism itself has no inherent sense of order, we must explicitly add information about each token's position in the sequence. This is done by adding a positional encoding vector to each token embedding.

Decoder-Only Transformers, ChatGPTs specific Transformer, Clearly Explained!!!

The StatQuest video begins by explaining how words are converted into numerical representations that the model can process.

Watch the section from on embeddings and positional encoding. This will walk you through creating word embeddings and then adding positional information to preserve the sequence order.

2. The Decoder Block

The main body of the model consists of a stack of N identical decoder blocks. The zoomed-in portion of the architecture diagram we saw earlier reveals the components of a single block. It contains two main sub-layers:

Masked Multi-Head Self-Attention: This is the core causal attention mechanism we just discussed, performed in parallel across multiple "heads" to capture different types of relationships.
Position-wise Feed-Forward Network (FFNN): A small, two-layer neural network applied independently to each token's representation. It adds computational depth and allows the model to learn more complex transformations.

Crucially, each of these sub-layers is wrapped with a residual connection and layer normalization. As you know from your deep learning background, residual connections (x + SubLayer(x)) help combat vanishing gradients in deep networks, while layer normalization stabilizes the activations.

The data flow through a single block is as follows:
x = x + MaskedSelfAttention(LayerNorm(x))
x = x + FFNN(LayerNorm(x))

The article "Decoder-Only Transformers: The Workhorse of Generative LLMs" provides a great written summary of these components, including a PyTorch implementation that will resonate with your technical background.

Decoder-Only Transformers: The Workhorse of Generative LLMs

This article from Cameron R. Wolfe's newsletter details the structure of a standard decoder block, explaining layer normalization, the feed-forward network, and residual connections.

Please read the section titled The Decoder-Only Transformer Block. Pay special attention to the PyTorch code snippet for the Block class at the end of the section, as it perfectly encapsulates the data flow we described: x = x + self.attn(self.ln_1(x)) and x = x + self.ffnn(self.ln_2(x)).

3. Generating Output: The Language Modeling Head

After an input sequence passes through the entire stack of L decoder blocks, we are left with a sequence of final output embeddings, $h_{L}$ . To generate the next token, we focus on the final embedding in the sequence, $h_{L, N}$ . This vector is then passed through a final "language modeling head" which consists of:

An Unembedding Layer: A linear layer that projects the d-dimensional output embedding back into the vocabulary space, producing a logit score for every token in the vocabulary. This linear layer's weight matrix is often tied to (i.e., is the transpose of) the input token embedding matrix E.
A Softmax Function: This function converts the logit scores into a probability distribution over the entire vocabulary, representing the model's prediction for the next token.

Let's return to the StatQuest video to see this generation process in action.

Decoder-Only Transformers, ChatGPTs specific Transformer, Clearly Explained!!!

This part of the StatQuest video explains the self-attention calculation and then demonstrates how the model uses the processed input to generate the next word, completing the architectural picture.

First, watch the detailed walkthrough of masked self-attention to solidify your understanding of the Q, K, V calculations in a causal setting. Then, watch the crucial segment from on generating a response. This part explains the feed-forward layer, the final softmax, and the autoregressive loop: the model generates a word, appends it to the sequence, and feeds the new, longer sequence back in to generate the next word. This directly illustrates why the architecture is suited for generative tasks.

Conclusion

In this lesson, we have constructed the decoder-only Transformer from the ground up. By removing the encoder and cross-attention from the original architecture, we are left with a powerful, streamlined model optimized for a single purpose: generation.

Here are the key takeaways:

Architecture: A decoder-only model consists of input embedding layers (for tokens and positions), a stack of decoder blocks, and a final language modeling head.
Core Mechanism: Its operation is defined by causal (masked) self-attention, which ensures that the prediction for each token is only conditioned on itself and preceding tokens. This is implemented by setting attention scores for future tokens to $- \infty$ before the softmax.
Data Flow: Each decoder block contains two sub-layers (masked self-attention and a feed-forward network), each wrapped in a residual connection and layer normalization.
Suitability for Generation: The architecture is inherently autoregressive. It is trained to predict the next token in a sequence, making it a natural text generator. The entire model is optimized to compute the probability of the next word given the sequence so far, $P (w_{i + 1} ∣ w_{1}, ..., w_{i})$ .

You now have a complete picture of the workhorse behind modern generative AI. This unidirectional, causal model stands in contrast to another important architectural family. In our next lesson, we will explore its counterpart: the encoder-only architecture (e.g., BERT), which uses bidirectional self-attention and is optimized not for generation, but for deep contextual understanding.

Recommended resources