How Next-Token Prediction Generates LLM Responses

Welcome back. In the previous lesson, you saw how system instructions, retrieved context, and a user message are converted into tokens and numeric IDs. Now we focus on what the model does with that token sequence during inference: it generates an answer by repeatedly predicting one next token.

This is the core mechanism beneath an ordinary chat response and, later, a RAG answer. RAG can supply better evidence in the prompt, but the LLM still produces its final wording through the same next-token process.

---

The central idea: complete the sequence

An LLM receives a sequence of input tokens and produces a probability distribution over its vocabulary for the token that could come next.

For a simplified prompt:

User: What is RAG?
Assistant:

the model might judge that tokens beginning an answer such as RAG, Retrieval, or RAG stands are more likely than unrelated tokens such as Volcano or Tuesday.

The model does not retrieve a complete, hidden paragraph and reveal it word by word. Nor does it decide on the fiftieth token before generating the first. At each moment, it has the text so far and makes one new prediction. That selected token becomes part of the text it considers for the following prediction.

Strictly, it predicts tokens rather than words. A token may be a word, a piece of a word, punctuation, or a word with its preceding space.

The useful mathematical statement is:

$$P(t_{n+1}\mid t_1,t_2,\dots ,t_n)$$

This means: given all preceding tokens $t_1$ through $t_n$, estimate the probability of each possible next token $t_{n+1}$.

The entire preceding sequence matters. In a real chat application, that sequence can include:

• system instructions,
• earlier conversation messages,
• retrieved RAG passages,
• the current user question,
• formatting tokens that mark the assistant’s turn,
• tokens already generated in the answer.

---

From a prompt to probabilities

Consider a simplified RAG-style request:

System: Answer using the supplied source material only.

Context:
[Source A] RAG retrieves relevant document passages and places them in the model prompt.

User: What is RAG?

Assistant:

After tokenization, the model processes the complete sequence. A modern transformer uses mechanisms including attention to determine which earlier tokens are useful for predicting the next one. For the first answer token, terms such as “RAG,” “retrieves,” and “document passages” should strongly influence the result. The system instruction also influences which continuation is appropriate.

The model’s final processing layer creates one numerical score for every token in its vocabulary. These raw scores are called logits. A logit is not yet a probability and does not need to fall between zero and one.

A function called softmax transforms all logits into probabilities:

$$P(t_i\mid \text{context})=\frac{\exp (z_i)}{{\sum }_{j\in V}\exp (z_j)}$$

Here:

• $z_i$ is the logit assigned to possible token $t_i$,
• $V$ is the full token vocabulary,
• the resulting probabilities across every token add up to $1$.

A fictional distribution after the prompt might look like this:

Candidate next token	Illustrative probability	Interpretation
RAG	0.42	Strong opening for a direct definition
Retrieval	0.25	Another plausible opening
RAG stands	0.12	Plausible continuation, depending on tokenization
It	0.06	Possible, but less direct
Any unrelated token	Very small	Inconsistent with the prompt’s context

These numbers are only an illustration. A real model scores tens or hundreds of thousands of possible tokens, and the exact tokens depend on its tokenizer.

The diagram’s “next word” label is a useful simplification, but remember that the actual choices are tokens.

---

Selecting one token and repeating

The model has produced a distribution, not an answer. The inference system must select one token from that distribution.

The simplest method is greedy decoding: choose the highest-probability token every time. If RAG has the highest probability, greedy decoding selects it.

Once selected, the token is appended to the running sequence. The model then predicts the next token given the original prompt plus the generated RAG. It may next select a token corresponding to is, followed by tokens completing Retrieval-Augmented Generation.

Here is the logical sequence:

Generation point	Text available to the model	Illustrative selected continuation
Before answer starts	User: What is RAG? Assistant:	RAG
After one output token	... Assistant: RAG	is
After two output tokens	... Assistant: RAG is	a
Later	... Assistant: RAG is a ...	Further answer tokens
End	A completed response	End-of-sequence or end-of-turn token

The displayed pieces are illustrative; a real tokenizer may split them differently. The key fact is unchanged: every new token is conditioned on the prompt and all earlier generated output.

This is called autoregressive generation. “Auto” refers to the model feeding its own selected output back into the sequence it uses for the next prediction.

Why responses can differ

An application does not always choose the single highest-probability token. It may sample from several plausible candidates. Sampling can make outputs less repetitive and can allow multiple reasonable phrasings of the same answer.

Two common controls are:

• Temperature: changes how sharply the model favors high-probability tokens. Lower temperature concentrates choices among the most likely tokens; higher temperature gives lower-probability alternatives more opportunity.
• Top-p sampling: restricts selection to a group of plausible tokens whose combined probability reaches a specified threshold.

For a factual support assistant or a RAG application, lower-variance settings are often useful because they make wording more stable. But a low temperature is not a truth guarantee. If the supplied context is missing, misleading, or ignored, the model can still produce an unsupported claim confidently.

---

Prefill, decode, and stopping

In practice, inference has two broad phases.

1. Prefill: process the supplied prompt

The model first processes all input tokens: instructions, chat history, user input, and any retrieved RAG context. This establishes the model’s internal representation of the current request.

For a long RAG prompt, this stage can be substantial because the model must process the inserted passages as well as the question. This is one reason token budgets matter: adding more context has a cost and can crowd out room for the answer.

2. Decode: generate the response

The system then repeatedly selects one output token. Each new token extends the assistant response and changes the context for the next prediction.

A technical implementation often stores intermediate attention calculations from the prompt and previous output tokens in a key-value cache. This avoids recomputing everything from scratch for each generated token. The logical behavior is still autoregressive: the next token depends on the full sequence available so far.

Generation stops when one of these conditions is met:

• the model emits a special end-of-sequence or end-of-turn token;
• the application reaches its maximum allowed output length;
• the application uses a configured stopping rule.

The special stopping token is normally not shown to the user. The generated token IDs are decoded back into text, often streamed a few tokens at a time so that the answer appears gradually.

---

What next-token prediction does—and does not—mean

Next-token prediction can sound too simple to explain a fluent answer. The complexity lies in the learned model that generates the probability distribution. Its parameters were shaped during training on enormous amounts of text, enabling it to learn patterns of grammar, explanation, style, code, and many factual associations.

But the immediate task during inference remains narrow:

Given the sequence so far, what token is a suitable next continuation?

That has several important consequences:

1. The model responds to the prompt, not directly to reality.
   It produces text that fits its learned patterns and the supplied context. A plausible-sounding sentence can still be false.

2. Prompt wording changes the output distribution.
   Changing a system instruction, a retrieved source, or a question can shift which next tokens become likely.

3. RAG works through the prompt.
   A retrieved passage does not change the model’s parameters during a request. It simply becomes additional tokens that influence subsequent predictions.

4. Generation is sequential.
   A long answer usually takes longer than a short answer because the system must select and process more output tokens.

For example, if a retrieved source says that a company policy allows refunds within 30 days, then tokens describing “30 days” may become much more likely in response to a refund question. If no source answers the question, the best RAG prompt should instruct the model to say that the evidence is insufficient rather than invent a continuation.

---

Key takeaways

An LLM answer is built during inference one token at a time:

• The model processes the prompt, including instructions, user input, prior conversation, and any RAG context.
• It assigns a score, then a probability, to every possible next token.
• A decoding strategy selects one token.
• That token becomes part of the sequence, and the process repeats.
• Generation ends at a special stop token or an application limit.

The model’s probability distribution is influenced by its learned parameters and by the text supplied in the current prompt. This distinction is essential for RAG: retrieved documents guide generation during a request without retraining the model.

Next, you will separate training—how a model’s parameters are learned—from inference—how a trained model uses those fixed parameters to generate an answer.