Controlling Output with Generation Parameters

How an LLM Chooses its Words

Before we start turning knobs, it's crucial to understand what's happening under the hood. When an LLM generates a response, it's not writing sentences; it's predicting the next token, one at a time. This process involves a few key steps.

First, for a given sequence of text, the model computes a raw score, called a logit, for every possible next token in its vocabulary. These logits are then passed through a softmax function, which converts the scores into a probability distribution—a list of probabilities for each token, all summing to 1.0.

The simplest way for the model to pick the next token is to always choose the one with the highest probability. This is called greedy decoding. While it's fast and deterministic, it often leads to dull, repetitive, and unnatural-sounding text. To get more interesting and human-like output, we need to introduce a bit of controlled randomness through sampling. This is where generation parameters come in.

Temperature: The Creativity Knob

The most fundamental parameter for controlling randomness is temperature. You can think of it as a dial for creativity.

• Low Temperature (e.g., 0.1 - 0.4): This makes the model more "confident" and deterministic. It sharpens the probability distribution, making the most likely tokens even more probable and suppressing less likely ones. This is ideal for tasks requiring factual accuracy, like code generation or answering questions from a knowledge base.
• High Temperature (e.g., 0.8 - 1.5): This makes the model more "adventurous." It flattens the probability distribution, giving less likely tokens a higher chance of being selected. This is great for creative tasks like writing stories, brainstorming ideas, or generating diverse marketing copy.

The video below offers a superb visual and conceptual explanation of how temperature scaling works by modifying the logits before the softmax function is applied.

As you saw, a temperature of 0 is equivalent to greedy decoding. As you increase the temperature, the model's choices become more varied. However, extremely high values (e.g., > 1.5) can make the output nonsensical as the model starts picking truly random tokens.

Let's see this in a practical Python example. The code below calls the OpenAI API with the same prompt but with different temperature values.

import os
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

try:
    client = OpenAI()

    prompt = "Complete this sentence creatively: The ancient library's secret wasn't in a book, but in"

    print(f"Prompt: '{prompt}'\n")

    for temp in [0.2, 0.8, 1.4]:
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            temperature=temp,
            max_tokens=40, # We'll discuss this next!
        )
        text = response.choices[0].message.content.strip()
        print(f"--- Temperature: {temp} ---")
        print(text)
        print()

except Exception as e:
    print(f"An error occurred: {e}")

Running this code would yield results similar to this:

• Temperature: 0.2: "...the silence between the pages, where forgotten stories whispered their truths." (Focused, poetic)
• Temperature: 0.8: "...the rhythmic hum of the celestial clockwork hidden beneath the floorboards." (Creative, imaginative)
• Temperature: 1.4: "...the shimmering dust motes that danced in sunbeams, each a tiny universe of forgotten spells." (Highly creative, almost surreal)

Controlling Length and Cost: max_tokens

The max_tokens parameter is straightforward but critical. It sets a hard limit on the number of tokens the model will generate in its response.

It's important to understand that this is a cutoff, not a suggestion. If the model reaches the max_tokens limit mid-sentence, it will simply stop. When you make an API call, the response object includes a finish_reason. If the model completes its thought, the reason will be stop. If it hits the token limit, the reason will be length.

This is useful for two main reasons:

1. Controlling Response Length: For tasks that require concise answers, you can set a low max_tokens.
2. Managing Cost: Since most API pricing is based on the total number of tokens (prompt + completion), setting a reasonable max_tokens prevents unexpectedly long and expensive responses.

The following video segment demonstrates exactly how max_tokens works and how it affects the finish_reason in the API response.

A common mistake is to try and use max_tokens to control the "style" of the output (e.g., setting it low for a summary). This is incorrect. Use prompt engineering ("Summarize in one sentence...") to control style and max_tokens to control the absolute maximum length.

Refining the Selection Pool: top_p and top_k

Temperature reshapes the entire probability distribution. Two other parameters, top_k and top_p, work by filtering the number of tokens the model can choose from in the first place.

• Top-K Sampling: This method restricts the model's choices to the top k most probable tokens. If k=10, the model will only consider the 10 most likely tokens and ignore all others. The main drawback is that k is a fixed number, which isn't always ideal. Sometimes the next word is obvious (only 1-2 good choices), and sometimes many words could fit (hundreds of good choices).
• Top-P (Nucleus) Sampling: This is a more dynamic approach. Instead of picking a fixed number of tokens, it selects the smallest set of tokens whose cumulative probability is greater than or equal to p. If p=0.9, the model considers the most probable tokens, adds their probabilities together one by one, and stops once the sum exceeds 90%. This "nucleus" of tokens might include just one token if the model is very certain, or dozens if it's uncertain.

Because of its adaptive nature, Top-P is generally preferred over Top-K and is the recommended approach for most use cases when you want to refine the token pool. OpenAI's API, for instance, supports top_p but not top_k.

A Visual Guide and Practical Cheat Sheet

You now have a set of powerful tools to control LLM generation. The following infographic provides a great one-page summary of the most common parameters.

To put it all together, here is a practical guide for choosing your settings based on the task at hand. Don't memorize it, but use it as a starting point for your own experiments.

The frequency_penalty parameter, as shown in the infographic, is used to discourage the model from repeating the same tokens too often. A positive value will reduce the likelihood of repeating words, which is useful in creative tasks.

Conclusion

In this lesson, you've moved beyond prompt structure to gain fine-grained control over the LLM's generation engine. By manipulating these parameters, you can dial in the exact behavior you need for your application.

Here are the key takeaways:

• LLMs generate text by sampling from a probability distribution of possible next tokens.
• temperature is your main tool for controlling the randomness and "creativity" of the output. Low values produce focused, deterministic text; high values produce diverse, creative text.
• max_tokens is a hard limit on the length of the response, essential for managing output size and API costs.
• top_p (Nucleus Sampling) provides a dynamic way to filter the token pool, ensuring a balance between diversity and coherence. It's generally recommended over top_k.
• The standard practice is to adjust temperature and top_p together to achieve the desired output style for a specific task.

You now know how to prompt the model effectively and how to control the properties of its generated output. In our next lesson, we'll focus on the final piece of the puzzle: parsing and processing the LLM API responses programmatically. You'll learn how to reliably extract the generated text, handle different response formats, and integrate the LLM's output back into your application's logic.