Temperature Scaling for Calibrated Probabilities

Welcome to our third lesson in the "Decoding Strategies and Inference Optimization" module. In our last session, we explored deterministic decoding methods like greedy search and beam search. We concluded that while they are effective at finding high-probability sequences, their deterministic nature often leads to outputs that are repetitive and lack the creative spark of human language. This observation sets the stage perfectly for today's topic.

Today, we will take our first step into the world of stochastic decoding by examining temperature scaling. This simple yet powerful technique allows us to control the level of randomness in a model's output. Our goal is to explain precisely how the temperature parameter modifies the sharpness of the model's output probability distribution, giving us a dial to tune the trade-off between predictability and creativity. This concept is a fundamental building block for many advanced sampling strategies.

From Hard Max to Soft Max

Let's begin by recalling the central issue with greedy search: it always picks the single token with the highest probability (the "hard maximum"). This inflexibility is often undesirable. We'd prefer a mechanism that treats the model's outputs not as a single directive, but as a spectrum of possibilities, allowing even less-likely (but still plausible) tokens a chance to be selected.

The following video provides an excellent intuitive introduction to this idea, drawing an analogy from statistical physics to explain how "temperature" can introduce variation into a system.

Softmax - What is the Temperature of an AI??

This video, "Softmax - What is the Temperature of an AI??," from MarbleScience, introduces the concept of temperature by analogy to physical systems and connects it to the Softmax function in language models.

Please watch the following segments: First, watch the introduction from the beginning to the physical analogy to understand how temperature relates to accessible states and variation. Next, watch the segment from next-word prediction, which frames the problem: simply picking the highest-rated word is boring and predictable. Finally, watch from the Softmax section to see how temperature is introduced to create a "soft" maximum and control the breadth of the output probability distribution.

The video introduces the key idea: we can use a parameter, which we call temperature, to control how "soft" our selection process is. But to understand how this works mathematically, we must first look at the function it modifies: the Softmax function.

The Problem of Overconfidence in Softmax

As you know from your deep learning background, the final linear layer of a Transformer produces raw, unnormalized scores for each word in the vocabulary, known as logits. To convert these logits into a valid probability distribution, we use the Softmax function.

However, the standard Softmax function has a particular characteristic due to its use of the exponential function: it can be overly aggressive in amplifying small differences in logit scores. This can lead the model to be "overconfident," assigning almost all the probability mass to one token, even if others had very similar logit values.

The video below clearly explains this phenomenon and sets the stage for introducing temperature as a remedy.

Softmax (with Temperature) | Essentials of ML

The video "Softmax (with Temperature)" by Kapil Sachdeva provides a clear technical explanation of the standard Softmax function and the motivation for temperature scaling.

Please watch from the Softmax section. This part details how Softmax works and, crucially, explains how the exponential function can lead to overconfident predictions. Then, continue watching from the introduction to temperature to see how a temperature parameter is introduced to counteract this effect.

The Mathematics of Temperature Scaling

As the video explained, temperature scaling works by dividing the logits by a temperature value, $T$ , before they are passed into the Softmax function.

Let $z = (z_{1}, z_{2}, \dots, z_{K})$ be the vector of logits for a vocabulary of size $K$ . The temperature-scaled Softmax function calculates the probability of the $i$ -th token, $π_{i}$ , as:

π_{i} (T) = \frac{exp ( z _{i} / T )}{\sum _{j = 1}^{K} exp ( z _{j} / T )}

Let's analyze the behavior of the temperature parameter $T$ :

When $T = 1$ : This is the standard Softmax function. The output probabilities directly reflect the exponentiated logits.
When $T > 1$ (e.g., $T = 2.0$ ): The logits are scaled down, making them closer to each other. This causes the resulting probability distribution to become flatter or "softer." The model becomes less confident, and the probabilities of less likely tokens are increased. This encourages more diversity and randomness.
When $0 < T < 1$ (e.g., $T = 0.5$ ): The logits are scaled up, exaggerating the differences between them. This makes the probability distribution sharper or "peakier." The model becomes more confident and deterministic, assigning even higher probability to the most likely tokens.
When $T \to 0$ : The differences between logits are magnified toward infinity. The token with the highest logit will have its probability approach 1, while all others approach 0. This effectively turns the Softmax into a hard argmax operation, mirroring greedy search.
When $T \to \infty$ : All logits are scaled toward 0. Since $exp (0) = 1$ , all tokens will have nearly equal probability, resulting in a uniform distribution. This leads to maximum randomness, where the model essentially picks tokens at random from the vocabulary.

The following image provides a clear visual summary of how temperature reshapes the probability distribution.

This series of plots shows the probability distribution over five tokens at different temperature settings. As the temperature increases from 0.1 to 5.0, the distribution flattens, moving from a near-certain choice (Token_0) to a more uniform distribution where all tokens have a similar chance of being selected.

A Formal View: Temperature and Entropy

We can formalize the notion of "randomness" or "uncertainty" using the concept of entropy from information theory. A higher entropy corresponds to a more uniform, uncertain distribution, while a lower entropy corresponds to a sharper, more certain one.

A key theoretical result is that, for a single decoding step, increasing the temperature monotonically increases the entropy of the resulting probability distribution. The research paper "Some Elementary Properties of Temperature Scaling" provides a formal proof.

Some Elementary Properties of Temperature Scaling - arXiv

This paper provides a rigorous mathematical analysis of temperature scaling. We will focus on the formal definition and the relationship between temperature and entropy. The paper uses inverse temperature β, where β = 1/T.

First, read the section "What is temperature scaling?" to see the formal definition. Note the use of β and the connection to the Boltzmann-Gibbs distribution. Next, and most importantly, read the subsection "Entropy is an increasing function of the temperature." Pay close attention to Proposition 2, which gives the derivative of the entropy with respect to the inverse temperature β: \frac{dH(p_\beta(y|\mathbf{x}))}{d\beta} = -\beta\text{Var}_{y\sim p_\beta(y|\mathbf{x})}(z_y) \leq 0 This equation is the mathematical heart of our learning objective.

The equation from Proposition 2 states that the rate of change of entropy $H$ with respect to inverse temperature $β$ is negative and proportional to the variance of the logits. Since temperature $T$ is the reciprocal of $β$ , a decrease in $β$ corresponds to an increase in $T$ . Therefore, the equation proves that as temperature $T$ increases, entropy $H$ increases. This confirms our intuition: heating up the distribution makes it more random.

A Caveat for Autoregressive Models

The monotonic relationship between temperature and entropy holds for a single prediction step. However, it's crucial to understand a subtlety that arises in autoregressive generation. In LLMs, we apply temperature scaling sequentially at each token generation step. This is known as myopic temperature scaling.

Interestingly, in this sequential setting, increasing the temperature at each step does not guarantee that the entropy of the final, complete sequence will also increase monotonically.

The same paper provides an insightful counter-example.

Some Elementary Properties of Temperature Scaling - arXiv

This section challenges the simple intuition about temperature and diversity in the context of full sequence generation.

Please read the subsection "For language models, increasing the temperature can decrease the entropy". The authors present a toy example where making a more random choice early on (due to high temperature) can lead the model down a path where all subsequent choices are highly predictable, resulting in a sequence with lower overall entropy.

This is an important nuance. While temperature is a reliable "knob" for controlling single-step randomness, its effect on the diversity of the final generated text can be more complex and sometimes counter-intuitive.

Conclusion

In this lesson, we have moved beyond deterministic decoding and introduced temperature scaling as our first tool for controlling randomness in text generation.

Here are the essential takeaways:

Temperature scaling modifies the Softmax function by dividing logits by a temperature parameter $T$ , allowing us to control the sharpness of the output probability distribution.
A temperature $T > 1$ "flattens" the distribution, increasing entropy and leading to more random and diverse outputs.
A temperature $0 < T < 1$ "sharpens" the distribution, decreasing entropy and making the output more deterministic and focused on high-probability tokens.
The relationship between temperature and entropy at a single step is monotonic: higher temperature yields higher entropy.
In autoregressive generation, this per-step control does not always translate to a monotonic increase in the entropy of the entire sequence.

You now understand how to adjust the very shape of the probability space from which we sample. However, temperature scaling still considers every single token in the vocabulary, even those with infinitesimal probabilities. This can be inefficient and sometimes lead to nonsensical choices at high temperatures. What if, instead of reshaping the distribution, we could truncate it, focusing only on the most plausible candidates? This question leads us directly to our next lesson on top-k and nucleus (top-p) sampling.