Welcome to the first lesson of our new module, "Decoding Strategies and Inference Optimization." In the previous module, we concluded our deep dive into the architecture of modern LLMs, examining the advanced components like RMSNorm and SwiGLU that define the state-of-the-art decoder block. You now have a solid understanding of the static structure of a trained language model.
This lesson marks a crucial shift in focus from architecture and training to inference. We will answer the fundamental question: "Now that we have this highly optimized, trained decoder model, how do we actually use it to generate text?" We will explore the core mechanism that underpins all text generation in models like GPT: autoregressive generation. You will learn how these models produce coherent text one token at a time by repeatedly using their own output as the next input, forming a generative feedback loop.
The Principle of Autoregression
At its heart, a decoder-only language model is a sophisticated probability machine. During its extensive pre-training, the model's singular goal is to learn the conditional probability distribution of a sequence of text. In simpler terms, given a sequence of tokens, it learns to predict the most likely next token.
This process is called autoregressive language modeling. The term "autoregressive" means that the model's prediction for a future value depends on its own past values. For a sequence of tokens , the model learns to factorize the joint probability of the entire sequence into a product of conditional probabilities, using the chain rule of probability:
Here, is the probability of the -th token given all preceding tokens. The model's architecture, particularly the causal mask in the self-attention mechanism, is purpose-built to compute this conditional probability for the very next token.
To solidify this mathematical foundation, let's review a formal definition.
A Tutorial on LLM Reasoning: Relevant Methods behind ChatGPT o1
This paper provides a concise mathematical description of autoregressive language models.
Please read the section titled The Challenges with Autoregressive LLMs. Pay close attention to the equation that factorizes the joint probability. Then, review the appendix section on the pre-training stage to see how the cross-entropy loss directly corresponds to this autoregressive objective.
The Generation Loop: Step by Step
Understanding the probabilistic principle allows us to outline the algorithmic process of text generation. It is an iterative loop that proceeds one token at a time.
- Initialization: The process begins with an input sequence, known as the prompt. For example:
"AI enhances our". - Tokenization: The prompt is converted into a sequence of token IDs using the model's tokenizer. Let's say this results in the sequence
[31, 15334, 524, 2909]. - Forward Pass: This sequence of token IDs is fed into the decoder model. The model performs a full forward pass through its layers (embedding, positional encoding, Transformer blocks).
- Logit Prediction: The model outputs a vector of logits. This vector has the size of the vocabulary (e.g., 50,257 for GPT-2). Crucially, we are only interested in the logits corresponding to the last input token (
2909in our example), as these represent the model's prediction for the token that should follow"AI enhances our". - Probability Distribution: The logits are converted into a probability distribution over the entire vocabulary using a softmax function. Each token in the vocabulary is now assigned a probability of being the next token.
- Token Selection: A single token is chosen from this distribution. (We will cover the strategies for how to choose—such as greedy search or sampling—in the next lesson. For now, just know that one token is selected). Let's say the selected token is
"learning". - Append and Repeat: The token ID for
"learning"is appended to the input sequence. The new input sequence is now[31, 15334, 524, 2909, 1333]. This new, longer sequence is fed back into the model, and the loop repeats from Step 3 to generate the next token. - Termination: The loop continues until a special end-of-sequence
[EOS]token is generated or a predefined maximum length is reached.
The diagram below provides a high-level visualization of a single step in this loop.
A Detailed Walkthrough
Let's trace this process with a more detailed visual guide. The StatQuest video below meticulously explains how the model transitions from encoding the prompt to generating the response, token by token.
Decoder-Only Transformers, ChatGPTs specific Transformer, Clearly Explained!!!
This video, "Decoder-Only Transformers, Clearly Explained!!!", provides an excellent step-by-step visualization of the generation process.
First, watch from generating the next word. This part shows how the model, after processing the entire prompt "what is statquest", uses the final representation to predict the next token, which happens to be the end-of-sequence token in this case. Next, and most importantly, watch from generating the response. This section is the core of autoregressive generation. Observe how the model: Starts the generation process with an initial token (here, EOS). Feeds this token through the entire network, while attending to the original prompt, to generate the first word of the response: "awesome". Takes its own output, "awesome", and uses it as the new input for the next step. Feeds "awesome" through the network to generate the next token, which finally is EOS, terminating the generation.
This cycle of generating a token and then consuming it as input for the next step is the essence of autoregressive generation. The model is literally "talking to itself" to extend the sequence.
From Theory to Code
Your background in computer science and software development makes it valuable to see how this theoretical loop is implemented in practice. Andrej Karpathy's "Let's build GPT" provides a fantastic code-level explanation.
Let's build GPT: from scratch, in code, spelled out.
In this video, Andrej Karpathy builds a GPT model from scratch and explains the generate function, which is the heart of autoregressive inference.
Please watch the segment from implementing the generate function. In this clip, pay close attention to the for loop that iterates for max_new_tokens. Inside the loop, notice these key lines of code: logits, loss = self(idx): The forward pass with the current sequence idx. logits = logits[:, -1, :]: Plucking out the logits for only the last time step. This is the prediction for the next token. idx_next = torch.multinomial(probs, num_samples=1): Sampling one token from the probability distribution. idx = torch.cat((idx, idx_next), dim=1): Concatenating the newly generated token idx_next to the sequence idx to form the input for the next iteration. This code is a direct implementation of the generation loop we've been discussing. To connect this back to the model's architecture, watch the short explanation from why this is a decoder-only model. Karpathy clarifies that the causal (triangular) mask is what makes the model "autoregressive" and enables this sequential generation process by preventing tokens from seeing the future during training.
The process described is further reinforced in many practical guides. For instance, the generate_text function in the article "Building a Decoder-Only Transformer Model" follows the exact same logic of a loop, a forward pass, isolating the last token's logits, sampling, and concatenating the result. This pattern is universal across generative decoder-only models.
Conclusion
In this lesson, we transitioned from the static architecture of a trained LLM to the dynamic process of using it for inference. You've learned how these models produce text through an iterative, token-by-token feedback loop.
Here are the key takeaways:
- Autoregressive Generation: This is the process of generating a sequence one token at a time, where each new token is conditioned on all previously generated tokens (both the original prompt and the model's own prior outputs).
- The Generation Loop: This is an algorithm that involves a repeated cycle: perform a forward pass with the current sequence, select a new token from the resulting probability distribution, append this token to the sequence, and repeat.
- Input becomes Output: The defining characteristic of this process is that the model's output from one step becomes a part of its input for the next, allowing it to build upon its own generation.
- Architectural Enabler: This entire process is made possible by the decoder-only architecture's causal self-attention mechanism, which ensures the model is trained specifically to predict the next token in a sequence.
You now understand the fundamental "how" of text generation. However, a crucial detail we glossed over is the selection step. How does the model choose one token out of a distribution of thousands? This decision has a profound impact on the quality, coherence, and creativity of the output.
In our next lesson, we will dive into this very question, exploring different decoding strategies like greedy decoding, beam search, and various sampling methods.