Decoding Strategies & Temperature

Hello! Welcome to the seventh lesson in our module on LLM and Transformer Fundamentals.

In our last lesson, we explored how a decoder-only transformer is pretrained using an autoregressive, next-token prediction objective. We established that while training is highly parallel, inference is a sequential, one-token-at-a-time process. At each step, the model outputs a probability distribution over its entire vocabulary for the next token.

This brings us to a crucial question: how do we select a token from this vast field of possibilities? The choice we make here profoundly impacts the quality, creativity, and coherence of the generated text.

Today's learning outcome is to compare common decoding strategies (greedy, beam search, top-k/top-p sampling) and the effect of temperature. We will analyze the trade-offs between these methods, moving from simple deterministic approaches to more sophisticated stochastic sampling techniques. This is a critical concept in tuning LLM behavior for production applications, whether for factual accuracy or creative generation.

1. The Starting Point: The Logit Distribution

As a quick recap, after a forward pass, the model produces a vector of raw, unnormalized scores called logits for the next token. A higher logit means the model considers that token more likely. These logits are then converted into a probability distribution using the softmax function:

$$P(toke{n}_i)=\frac{e^{z_i}}{{\sum }_{j=1}^V{e}^{z_j}}$$

where $z_i$ is the logit for token $i$ and $V$ is the vocabulary size. Our task is to choose one token from this distribution.

2. Deterministic Decoding: Predictability and Its Pitfalls

The most straightforward approaches are deterministic: given the same input, they will always produce the same output.

Greedy Search

The simplest strategy of all is greedy search. At each step, you simply select the token with the highest probability (the argmax of the distribution).

• Pros: Fast, simple, and completely deterministic.
• Cons: It's a "myopic" approach. A locally optimal choice at one step can lead to a globally suboptimal sequence. This often results in text that is repetitive, dull, and can get stuck in loops (e.g., "I'm sorry, I'm sorry, I'm sorry...").

The following video provides a great introduction to the need for decoding strategies and explains the limitations of the greedy approach.

Beam Search

To mitigate the shortsightedness of greedy search, we can use beam search. Instead of committing to a single best token at each step, beam search keeps track of the k most probable sequences (called "beams" or "hypotheses") so far.

The process is as follows:

1. At step 1, generate the k most likely next tokens from the input prompt. These are your initial k beams.
2. At step 2, for each of the k beams, predict the next-token distribution. This gives you k * V possibilities (where V is vocabulary size).
3. From all these possibilities, choose the k new sequences with the highest overall probability (usually calculated as the sum of log-probabilities).
4. These k sequences become your new beams. Repeat until an end-of-sequence token is generated or a max length is reached.

Beam search is a heuristic that is more likely to find a higher-probability sequence than greedy search, but it's not guaranteed to find the absolute best one.

• Pros: Produces more coherent and higher-quality outputs for tasks where precision is key, like translation or summarization. It's still deterministic.
• Cons: Computationally more expensive than greedy search (by a factor of k). It still tends to produce safe, high-probability text and can struggle with creativity. It may also favor shorter sequences, which is often addressed by normalizing the sequence score by its length.

Given your background in building high-throughput systems, the implementation of beam search is particularly interesting. It's not k sequential operations; it's a parallelized process. The following video explains how this is implemented efficiently.

3. Stochastic Decoding: Introducing Creativity

For many applications (like chatbots, story writing, or brainstorming), we want the model to be less predictable and more creative. This requires introducing randomness through sampling.

Temperature

The most fundamental parameter for controlling randomness is temperature. Temperature modifies the shape of the probability distribution before sampling. It is applied directly to the logits inside the softmax function:

$$P(toke{n}_i)=\frac{e^{z_i/T}}{{\sum }_{j=1}^V{e}^{z_j/T}}$$

where $T$ is the temperature.

• If $T\to 0$: The distribution becomes extremely "peaky" or sharp. The probability of the highest-logit token approaches 1. This makes the sampling behave like greedy search.
• If $T=1$: The formula is identical to the standard softmax. The model samples from its original probability distribution.
• If $T>1$: The distribution becomes "flatter" or softer. Probabilities are more evenly distributed, increasing the chance of sampling less likely tokens. This leads to more randomness and creativity, but also a higher risk of incoherence and errors.

For a detailed walkthrough of the math and practical implications, the following resource is excellent.

Top-k Sampling

While temperature adjusts the shape of the entire distribution, we often want to prevent the model from considering truly nonsensical tokens from the long tail of the distribution. Top-k sampling does this by truncating the distribution.

The process is simple:

1. Identify the k tokens with the highest probabilities.
2. Discard all other tokens.
3. Renormalize the probabilities of the remaining k tokens so they sum to 1.
4. Sample from this smaller, renormalized distribution.

• Pros: Prevents low-probability, often nonsensical tokens from being chosen. It's a simple way to bound the randomness.
• Cons: The fixed value of k is not adaptive. In a situation where the model is very confident (e.g., one token has 95% probability), a large k might still include bad options. In a situation where the model is uncertain (probabilities are flat), a small k might cut off perfectly valid, creative choices.

Top-p (Nucleus) Sampling

Top-p sampling, also called nucleus sampling, was designed to address the static nature of top-k. Instead of sampling from a fixed number of tokens, it samples from a variable set whose size depends on the model's confidence.

The process is:

1. Sort the tokens by probability in descending order.
2. Sum their probabilities cumulatively.
3. Find the smallest set of tokens whose cumulative probability is greater than or equal to a threshold p. This set is the "nucleus".
4. Discard all other tokens, renormalize the probabilities within the nucleus, and sample from it.

This method is adaptive:

• If the model is confident, the top few tokens will have a high cumulative probability, so the nucleus will be small.
• If the model is uncertain, many tokens will be needed to reach the cumulative probability p, so the nucleus will be larger, allowing for more diversity.

For a deeper dive into why Top-p is generally preferred over Top-k, let's return to the Tom Archer article, which has an excellent section on this exact trade-off.

4. Comparison and Practical Guidelines

Choosing the right decoding strategy is a balancing act. Here is a summary of the methods and their trade-offs:

Strategy	Type	Pros	Cons	Best For
Greedy Search	Deterministic	Fast, simple, predictable	Repetitive, low quality, myopic	Simple testing, debugging
Beam Search	Deterministic	Higher quality than greedy, more coherent	Computationally expensive, not creative, can favor short text	Machine translation, summarization, code generation
Top-k Sampling	Stochastic	More creative than deterministic methods, avoids gibberish	k is not adaptive to the model's confidence	General creative tasks, chatbots
Top-p Sampling	Stochastic	Adapts to model's confidence, balances creativity & coherence	Slightly more complex than top-k	State-of-the-art for most creative generation tasks

Parameters:

• Temperature: Controls the "creativity" vs. "safety" of the output by reshaping the entire probability distribution. Low temperature (~0.2) is for factual tasks; high temperature (~0.8-1.0) is for creative tasks.
• Top-p: A good default is often around 0.9. It's generally recommended to tune either Temperature or Top-p, but not both at once, as their interaction can be complex.

Conclusion

Today, we've explored the critical step that bridges the gap between a model's probability predictions and the final text it generates. We have seen that there is no single "best" decoding strategy; the optimal choice depends entirely on the application's requirements.

Key Takeaways:

• Greedy Search is fast and simple but often produces low-quality, repetitive text.
• Beam Search improves on greedy search by exploring multiple paths, yielding more coherent results for tasks requiring precision, but at a higher computational cost.
• Stochastic Sampling introduces randomness, which is essential for creative and non-repetitive text generation.
• Temperature is a powerful knob to control the level of randomness by sharpening or flattening the probability distribution.
• Top-p (Nucleus) Sampling is an adaptive method that dynamically selects the set of candidate tokens based on the model's confidence, offering a robust balance between coherence and creativity. It is generally superior to the less flexible Top-k Sampling.

Understanding these strategies and their parameters is essential for anyone building applications on top of LLMs, as it allows for fine-grained control over the model's output behavior.

Next Lesson Preview

We have now covered the entire lifecycle of a transformer from its architecture to training and generation. But for these massive models to be practical in production, they must be efficient. In our next lesson, we will explain quantization as a model compression technique and its trade-offs between speed, memory, and quality. This will bring us back to the practical engineering challenges of serving LLMs at scale, a topic highly relevant to your work.