GPT Text Generation: Iterative Next-Token Sampling

Introduction

In our last lesson, we established the crucial distinction between the three main Transformer architectures. We identified decoder-only models (like GPT, Qwen, and DeepSeek) as the generative workhorses that form the foundation of modern LLMs and the advanced planning systems we'll study.

Today, we will dissect the core mechanism that gives these models their generative power. The learning outcome for this lesson is to describe how decoder-only models like GPT perform text generation via iterative next-token sampling. We will break down the "autoregressive" process into a concrete, step-by-step procedure, exploring how a model moves from predicting probabilities to generating coherent text.

Understanding this iterative loop is fundamental. It's the process that allows a model to write an essay, generate code, or, in the context of our course, "speak" a sequence of driving commands.

1. The Autoregressive Generation Loop

The term autoregressive simply means that the model's prediction at a given step depends on its own previous outputs. In text generation, this translates to a simple, powerful, iterative loop:

1. Start with a prompt: The model receives an initial sequence of tokens (e.g., The cat sat on the).
2. Predict the next token: The model processes this sequence and outputs a probability distribution over its entire vocabulary for the very next token.
3. Sample a token: A "decoding strategy" is used to select one token from this probability distribution (e.g., the model picks mat).
4. Append and repeat: The newly chosen token is appended to the input sequence, creating a new, longer prompt (The cat sat on the mat). This new sequence becomes the input for the next iteration, and the loop repeats to generate the subsequent token.

This process continues until the model generates a special end-of-sequence (<eos>) token or reaches a predefined maximum length.

The following diagram illustrates this core loop perfectly.

To solidify this concept, let's read a clear description of this process.

2. From Logits to Tokens: Decoding Strategies

We've established that at each step, the model produces a probability distribution. But how does it actually choose a token from the tens of thousands of possibilities? Simply always picking the most probable token often leads to dull, repetitive text. The art of generation lies in balancing predictability with creativity, which is handled by decoding strategies.

The video below gives an excellent, intuitive overview of the most common strategies.

Let's summarize the key strategies from the video:

• Greedy Decoding: Always select the token with the highest probability. It's deterministic and safe, but often leads to repetitive and uninteresting output.
• Temperature Sampling: A parameter $T$ is used to scale the raw output scores (logits) before the softmax function.
  • $T<1$ makes the distribution "peakier," increasing the likelihood of high-probability tokens (more like greedy).
  • $T>1$ "flattens" the distribution, increasing randomness and creativity.
• Top-K Sampling: At each step, the model considers only the k most likely tokens and samples from that reduced pool. This prevents highly improbable tokens from being chosen by chance.
• Top-P (Nucleus) Sampling: A more dynamic approach. Instead of a fixed number k, it samples from the smallest possible set of tokens whose cumulative probability is greater than p. This adapts the pool size based on the model's confidence at each step.

In practice, these methods are often combined (e.g., Top-P sampling with a specific temperature) to fine-tune the model's generative behavior.

3. Generation in Practice: A Code-Level View

Now, let's see how this iterative loop is implemented in code. You don't need to write this from scratch, but understanding the logic in a practical example is key. Since you're familiar with Python and PyTorch, this should be quite intuitive.

The following article provides a clear implementation of a GPT-style model and includes a concise function for inference.

Seeing the code makes the process concrete: it's a simple for loop that calls the model, samples a token, and appends it to the input. This exact logic, albeit at a massive scale, is what powers models like ChatGPT.

Conclusion

In this lesson, we've unpacked the core generative mechanism of decoder-only Transformers.

Key Takeaways:

• Autoregressive Generation: Text is generated one token at a time, where each new token is predicted based on all the tokens that came before it (the prompt + previously generated tokens).
• Iterative Loop: The process is a loop of predict -> sample -> append.
• Decoding Strategies: The "sample" step is not random. It's guided by strategies like greedy decoding, temperature scaling, top-k, and top-p (nucleus) sampling to balance coherence and creativity.
• Implementation: This iterative process can be implemented with a straightforward loop that repeatedly calls the model with an ever-expanding input sequence.

Preview of the Next Lesson:
You may have noticed a potential inefficiency in the process we described: at every step, the model re-processes the entire input sequence from the beginning, just to generate one new token. This seems incredibly wasteful. Why re-calculate what we already know?

In our next lesson, we will explore the KV Cache, a crucial optimization that solves this exact problem and makes real-time, efficient text generation possible.