Token-by-Token Text Generation in GPT Models

Recommended resources

The freely available resources this lesson will direct you to.

Hello! Welcome to the next phase of your AI engineering journey.

In our last module, we explored generative models for images, culminating in a comparison between GANs and Diffusion Models. We saw how these architectures create new data from scratch, either through an adversarial game or an iterative denoising process.

Now, we pivot from pixels to words and begin our deep dive into the world of Large Language Models (LLMs). This lesson kicks off our new module, "LLM Applications," by addressing a fundamental question: how do models like GPT actually write?

Our learning outcome for today is to explain how autoregressive language models (GPT-style) generate text token by token. You'll learn that there's no pre-planned script; instead, text is constructed one piece at a time through a fascinating, iterative process of prediction and sampling. This lesson builds directly on your understanding of the Transformer architecture from Module 7.

1. The Autoregressive Principle: One Token at a Time

At its heart, a model like GPT is a next-token predictor. Given a sequence of text, its sole job is to calculate the most probable next piece of text (a "token"). The magic happens when this process is repeated.

The model takes an input (your prompt), predicts the very next token, appends that new token to the sequence, and uses this new, slightly longer sequence as the input for the next step. This process, where a model's own outputs are fed back in as future inputs, is called autoregression. You've encountered a similar principle in Recurrent Neural Networks (RNNs), where the hidden state is passed from one timestep to the next.

This animated graphic provides a perfect high-level illustration of the concept in action.

Autoregressive Text Generation by GPT-3 — This animation from Jay Alammar's blog shows how a prompt is fed into a GPT model, which then generates one token at a time. The newly generated token is added to the input sequence for the next prediction step, demonstrating the core autoregressive loop.

2. The Five Steps of Text Generation

To understand this loop, we can break down a single generation step into a clear sequence. The following video provides an excellent overview of this entire process.

How LLMs Actually Generate Text (Every Dev Should Know This)

This video from LearnThatStack clearly explains the five key stages of how an LLM generates a single token, from input processing to final selection.

Please watch the entire video (about 9 minutes). It walks through the complete lifecycle of text generation. Pay close attention to the five main steps identified: Tokenization: How text becomes pieces. Embeddings: How those pieces get meaning. Transformer: How context is processed. Probabilities: How the model scores every possible next token. Sampling: How one token is chosen.

Now, let's solidify our understanding of each step, drawing on what you've learned about the Transformer architecture.

Step 1 & 2: Tokenization and Embedding

As the video explained, the model first uses a tokenizer to break the input text into manageable pieces (tokens). Each token is then mapped to a high-dimensional vector from an embedding matrix. This vector represents the token's semantic meaning. Finally, a positional encoding is added to give the model information about the token's position in the sequence.

These initial steps transform your plain-text prompt into a sequence of numerical vectors ready for the main event: the Transformer.

LLM Transformer Model Visually Explained

The interactive article "LLM Transformer Model Visually Explained" provides a great visual breakdown of this initial embedding phase.

Read the section titled "Embedding" (from "Let's say you want to generate text..." down to "...their position in the input sequence."). This will reinforce your understanding of how tokenization, token embedding, and positional encoding combine to create the model's input.

Step 3: Processing with Decoder-Only Blocks

The sequence of embeddings is fed into a stack of Transformer decoder blocks. As you'll recall from Module 7, GPT-style models are "decoder-only." This architecture is perfectly suited for autoregressive text generation because its core mechanism, masked self-attention, prevents any token from "seeing" future tokens in the sequence. When predicting the token at position t, the model can only attend to tokens from position 0 to t-1.

Decoder-Only Transformer Architecture for Autoregressive Text Generation — This diagram illustrates the data flow in a decoder-only transformer. Input tokens are embedded and processed through a stack of layers, each containing Masked Self-Attention. The final output is used to predict the next token, which is then fed back into the model in the next step.

Step 4: From Vector to Probabilities

After passing through all the Transformer blocks, the model produces a final output vector for the very last token in the input sequence. This vector is a rich, context-aware representation. But how does this become a word?

Linear Layer: The output vector is multiplied by a final, large weight matrix (often related to the token embedding matrix). This projects the vector into a much larger space, where the dimension is equal to the size of the model's vocabulary (e.g., 50,257 for GPT-2). The resulting raw scores are called logits.
Softmax Function: The softmax function is applied to the logits. This function converts the raw scores into a probability distribution, where each value is between 0 and 1, and all values sum to 1. Now, every token in the vocabulary has an assigned probability of being the next token.

For a deeper intuition behind the softmax function and the concept of "temperature," which we'll discuss next, the following video is invaluable.

Transformers, the tech behind LLMs | Deep Learning Chapter 5

This clip from 3Blue1Brown's excellent series on Transformers explains how the softmax function works and introduces the 'temperature' parameter.

Watch the clip from 22:22 to 25:19. Focus on how softmax turns arbitrary numbers (logits) into a valid probability distribution and how temperature can sharpen or flatten this distribution.

Step 5: Sampling the Next Token

With a probability distribution over the entire vocabulary, the final step is to choose one token. There are several strategies, or sampling methods, for this:

Greedy Decoding: Always pick the token with the highest probability. This is deterministic and safe but often leads to repetitive and boring text.
Temperature Sampling: The temperature parameter adjusts the "peakedness" of the probability distribution.
- A low temperature (e.g., 0.2) makes the distribution sharper, increasing the likelihood of picking high-probability tokens. This leads to more focused, deterministic output.
- A high temperature (e.g., 1.0 or higher) flattens the distribution, giving lower-probability tokens a better chance of being selected. This introduces more randomness and "creativity."
Top-k / Top-p (Nucleus) Sampling: These methods truncate the distribution before sampling. Top-k considers only the k most likely tokens, while top-p considers the smallest set of tokens whose cumulative probability exceeds a threshold p. This prevents the model from picking truly nonsensical tokens while still allowing for diversity.

3. Completing the Loop

Once a single token is sampled, the cycle is complete for one step. The model then appends this new token to the input sequence and runs the entire five-step process again to generate the next token. This continues until a special "end-of-sequence" token is generated or a maximum length is reached.

This article from Jay Alammar provides fantastic illustrations that tie this entire autoregressive process together.

The Illustrated GPT-2 (Visualizing Transformer Language Models)

This classic post, "The Illustrated GPT-2," uses clear visuals to demonstrate the token-by-token generation process.

Please read the following two short sections: "One Difference From BERT": This section introduces the concept of auto-regression with a clear example. "Crash Course in Brain Surgery: Looking Inside GPT-2": Start from this heading and read until the paragraph that begins "In the next step, we add the output from the first step...". This section beautifully visualizes the first two steps of the generation loop.

This step-by-step, iterative generation is why you see text appear word-by-word in interfaces like ChatGPT. The model has no preconceived notion of the full sentence or paragraph it will write; it constructs its response one probabilistic guess at a time.

Conclusion

In this lesson, we deconstructed the process of text generation in autoregressive language models. You learned that it's not a single act of creation but an iterative loop of prediction and sampling.

Key Takeaways:

Autoregressive Generation: LLMs generate text by repeatedly predicting the next token based on all preceding tokens (the prompt + previously generated tokens).
The Generation Loop: A single generation step involves (1) Tokenization, (2) Embedding, (3) Transformer Processing, (4) Calculating Probabilities via logits and softmax, and (5) Sampling a token.
Decoder-Only Architecture: GPT-style models use a decoder-only Transformer with masked self-attention, which is essential for the autoregressive task of predicting the next token without seeing the future.
Controlling the Output: Sampling strategies like temperature, top-k, and top-p are crucial hyperparameters that allow us to control the trade-off between determinism and creativity in the generated text.

Preview of the Next Lesson

Now that you understand the mechanics of how an LLM generates text, our next lesson will focus on how to guide it effectively. We will explore prompt engineering techniques, such as few-shot prompting and chain-of-thought, to steer the model's probabilistic generation toward more accurate, relevant, and complex outputs.