Optimizing Text Generation: Temperature, Top_p, and Max_Tokens

How an LLM "Chooses" its Next Word

Before we can manipulate the model's output, we need to understand its decision-making process. At its core, a Large Language Model is a probabilistic engine. When it generates text, it's not "deciding" in a human sense; it's calculating the probability of every possible next token (a word or part of a word) and then sampling from that distribution.

This process begins with logits, which are raw, un-normalized scores produced by the model for every token in its vocabulary. To turn these scores into usable probabilities, they are passed through a softmax function. Given your background in applied mathematics, the formula will be familiar:

Here, $l_i$ is the logit for a specific token $x_i$, and $V$ is the size of the entire vocabulary. The softmax function exponentiates each logit (making all values positive) and then normalizes them so that the sum of all token probabilities equals 1. A higher logit results in a higher probability.

The parameters we're about to discuss all work by modifying this process, either by altering the probability distribution itself or by changing how we sample from it.

Temperature: The Creativity Dial

The first and most fundamental parameter is temperature. It directly controls the randomness of the output. It works by dividing the logits by a temperature value $T$ before the softmax function is applied.

• Low Temperature (T < 1): Dividing by a value less than 1 increases the magnitude of the logits, making the differences between them more extreme. This "sharpens" the probability distribution, causing the model to become more confident and deterministic, almost always picking the token with the highest initial probability. A temperature of 0 results in greedy decoding, where the model always picks the single most likely token.
• High Temperature (T > 1): Dividing by a value greater than 1 "flattens" the probability distribution, making the probabilities of less likely tokens higher. This increases randomness and creativity, but also the risk of nonsensical or "hallucinated" output.

The following video provides a clear visual intuition for how temperature reshapes the probability distribution.

For your work, you would use:

• Low temperature (e.g., 0.1 - 0.3) for tasks requiring precision, like generating Python code for a backtesting script or summarizing factual data from an earnings report.
• High temperature (e.g., 0.9 - 1.5) for brainstorming tasks, such as generating novel hypotheses for trading strategies or creating varied scenarios for market stress tests.

Top-P (Nucleus Sampling): The Dynamic Candidate Pool

While temperature re-weights the probabilities of all tokens, Top-P sampling (also known as nucleus sampling) truncates the list of candidates. It provides a more sophisticated way to control randomness than just temperature alone.

Top-P works by selecting the smallest set of tokens whose cumulative probability is greater than or equal to the value p. The model then samples only from this dynamically-sized "nucleus" of tokens.

This is its key advantage:

• In a high-confidence situation (e.g., after "The capital of France is..."), the token "Paris" might have a probability of 0.95. With top_p=0.9, the nucleus might contain only that single token.
• In a low-confidence situation (e.g., after "A good trading strategy could be..."), many tokens might have low individual probabilities. It might take hundreds of tokens to reach a cumulative probability of 0.9, and top_p will keep all of them in the candidate pool.

This adaptive nature makes it generally preferable to its predecessor, Top-K, which always samples from a fixed number (K) of the most likely tokens, regardless of their actual probabilities.

Most APIs, including OpenAI's, recommend using either temperature or top_p to control randomness, but not both at the same time, as their interaction can be hard to predict. The common practice is to set top_p to a reasonable default like 0.9 or 1.0 and tune temperature.

Max Tokens: Controlling Output Length and Cost

This parameter is much simpler but equally important. max_tokens sets a hard limit on the total number of tokens (including both the prompt and the generated response) in a single API call.

Its primary uses are:

1. Cost Management: LLM APIs charge based on the number of tokens processed. Setting a max_tokens limit prevents unexpectedly long (and expensive) responses.
2. Latency and Application Control: Ensures responses are concise and fit within the design of your application.
3. Context Window Management: Forcing shorter responses helps manage the total number of tokens you need to track in a conversation, which is critical for maintaining context over multiple turns (the topic of our next lesson).

When a model's output is cut short because it hits this limit, the API response will typically include a finish_reason of length.

Your Turn: Tuning the Quantitative Analyst Agent

Now, let's apply these concepts. We'll modify the Python script from our last lesson to experiment with these new parameters. You will see firsthand how they affect the output of your "Quantitative Analyst" agent.

Instructions:

1. Use the code below. It's an enhanced version of our previous script, structured to make experimentation easy.
2. Run the script multiple times with the different experiment_settings provided.
3. Analyze the output for each scenario. How does the tone, structure, and content of the response change?

import os
from openai import OpenAI




# The OpenAI library automatically looks for the OPENAI_API_KEY environment variable.
client = OpenAI()

def run_analysis_experiment(system_prompt, user_query, temperature, top_p, max_tokens):
    """
    Runs a single experiment with the given parameters and prints the response.
    """
    print(f"--- Running Experiment: T={temperature}, top_p={top_p}, max_tokens={max_tokens} ---")
    try:
        response = client.chat.completions.create(
            model="gpt-4-turbo",
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": user_query}
            ],
            temperature=temperature,
            top_p=top_p,
            max_tokens=max_tokens
        )

        agent_response = response.choices[0].message.content
        finish_reason = response.choices[0].finish_reason
        
        print(agent_response)
        print(f"\n[Finish Reason: {finish_reason}]")
        print("-" * 50 + "\n")

    except Exception as e:
        print(f"An error occurred: {e}")




# --- Agent Persona and Query Definition ---
system_prompt = """
You are a quantitative analyst AI. Your specialty is the statistical analysis of financial time-series data for mid-frequency trading applications.
You must adhere to the following constraints:
1.  Your analysis must be strictly objective and data-driven.
2.  You must explicitly state any assumptions you make.
3.  Under no circumstances will you provide financial advice or market predictions.
4.  Your communication style must be formal and precise.
"""

user_query_factual = "Analyze this sample data: `[101.2, 101.5, 101.3, 102.0, 101.8, 102.5, 100.9, 103.1]`. Provide a brief statistical summary."
user_query_creative = "Given this sample data: `[101.2, 101.5, 101.3, 102.0, 101.8, 102.5, 100.9, 103.1]`, brainstorm three unconventional statistical features one might derive to capture its underlying dynamics."




# --- Experiment Scenarios ---




# Scenario 1: Factual & Deterministic (for precise analysis)
# Expect consistent, dry, and factual output. Run it a few times to see how similar the responses are.
print("SCENARIO 1: Factual & Deterministic Analysis")
run_analysis_experiment(system_prompt, user_query_factual, temperature=0.1, top_p=1.0, max_tokens=500)




# Scenario 2: Creative Brainstorming
# Expect more varied and imaginative ideas. The quality might vary, but the diversity is the goal.
print("SCENARIO 2: Creative Strategy Brainstorming")
run_analysis_experiment(system_prompt, user_query_creative, temperature=1.3, top_p=1.0, max_tokens=500)




# Scenario 3: Controlled Brevity
# This demonstrates the effect of max_tokens. Note the finish_reason.
print("SCENARIO 3: Controlled Brevity")
run_analysis_experiment(system_prompt, user_query_factual, temperature=0.5, top_p=1.0, max_tokens=40)

By running these experiments, you'll develop an intuition for how to configure an agent for different sub-tasks within your trading workflow—from rigorous data analysis to open-ended strategy generation.

Key Takeaways and Next Steps

You now have a foundational understanding of how to control the style and structure of an LLM's output.

• LLMs generate text by sampling from a probability distribution of possible next tokens.
• Temperature adjusts the shape of this distribution, controlling the balance between consistency (low T) and creativity (high T).
• Top-P dynamically truncates the candidate pool to tokens that fall within a cumulative probability threshold, providing adaptive control over randomness.
• Max Tokens is a practical constraint to manage output length, cost, and latency.

In our next lesson, we will address a new challenge: how to build agents that can handle multi-step tasks. We'll explore how to manage conversational history to maintain context across several interactions, which is essential for creating agents that can perform sequential analysis or engage in a dialogue.