Optimizing LLM Parameters for Desired Agent Behaviors

Introduction

In our previous lesson, we established that the probabilistic, autoregressive nature of LLMs is the root cause of their most significant reliability challenges, such as the self-conditioning effect and the "lost-in-the-middle" problem. We concluded by framing agentic systems as a "stochastic convergence spiral," where we use deterministic feedback to guide a probabilistic model toward a correct outcome.

Today, we move from understanding these foundational issues to actively controlling them. This lesson focuses on the three primary levers you can manipulate at the inference stage to steer an agent's behavior: context windows, temperature, and token limits. Mastering these is the first practical step in designing agentic systems that are not just functional, but also reliable and cost-effective for your clients. We will examine how these parameters function, their interdependencies, and how to manipulate them strategically to achieve specific goals.

The Anatomy of an LLM Call: Tokens, Limits, and Windows

Every interaction with an LLM is governed by these three interconnected concepts. An excellent article from dev.to frames them through a lens that is directly relevant to your goal of building production systems: cost, reliability, and scalability.

As the article highlights, tokens are the fundamental currency of LLMs. Understanding their behavior is critical for cost estimation and performance planning. Here are the key system design implications:

• Tokens ≠ Words: Due to subword tokenization (like BPE), morphologically complex words, rare terms, code, and non-English text consume disproportionately more tokens. A system designed for English may see its costs triple when adapted for Japanese.
• Structured Data Overhead: When designing agents that must produce structured outputs like JSON, you must account for the token cost of every bracket, quote, and comma. This overhead is non-trivial and impacts both cost and whether the output fits within the context window.

This brings us to the container for these tokens: the context window.

The Context Window: A Finite Workspace

The context window is the maximum number of tokens an LLM can consider at once, encompassing both the input you provide and the output it generates. It is not an arbitrary limit but a direct consequence of the Transformer architecture.

The core self-attention mechanism, which allows the model to weigh the importance of different tokens, has a computational complexity of $O(n^2)$, where $n$ is the sequence length (number of tokens). Doubling the context length quadruples the computational cost and memory required for the attention matrix. This quadratic scaling is why even a 1M token context window, while impressive, is not a silver bullet for all problems and comes with significant latency and cost trade-offs.

This finite and expensive workspace necessitates a discipline known as Context Engineering.

To get a better sense of the strategies involved, please watch the following video.

From Context Management to Agent Memory

The strategies discussed in the video can be mapped to a powerful architectural mental model: building a memory system for your agent. LLMs are stateless; each API call is independent. Context is what provides the illusion of memory.

A helpful way to structure this is to think of different types of memory that correspond to different context management techniques.

Let's synthesize this into an architectural framework:

1. Working Memory: This is the LLM's active context window—the information it has immediately available. It's managed by simple techniques like keeping a running history of the conversation. The key constraint here is the token limit, which forces you to use strategies like sliding windows (keeping only the last N messages) or summarization (compressing older parts of the conversation), as discussed in the LINK article.

2. Episodic Memory: This corresponds to a history of past tasks or conversations. It's a form of long-term memory. You can't fit all past episodes into the context window. Instead, you store them externally (e.g., in a vector database) and use a selection strategy (like semantic search) to retrieve the most relevant past examples and inject them into the prompt. This is a form of Retrieval-Augmented Generation (RAG).

3. Semantic Memory: This is the agent's knowledge base—a "second brain" of factual information. Like episodic memory, it's stored externally and accessed via RAG. This is the primary way to overcome the LLM's knowledge cutoff and ground it in your client's specific business data, a topic we'll explore in depth in Module 6.

Strategically managing the context window is about designing a system that fluidly moves information between these memory types to provide the agent with precisely the right context at the right time, without exceeding token limits.

Controlling Creativity: Temperature and Sampling

Once you have assembled the context, the next step is to control how the agent uses it. This is done through temperature and other sampling parameters.

Let's dive deeper into the mechanics of temperature.

Now, let's turn to a more detailed guide on how to use these parameters in practice.

A Strategic Framework for Sampling

Based on the resources, here is a strategic summary for manipulating these parameters:

• Temperature: Controls the "peakiness" of the probability distribution.

  • Low (0.0 - 0.3): For tasks requiring high determinism and factuality, like code generation, data extraction, or classification. The agent will stick to the most likely tokens.
  • High (0.7 - 1.0+): For creative and exploratory tasks, like brainstorming, content creation, or conversational agents. This encourages diversity and novelty.

• Top-P (Nucleus Sampling): A more adaptive way to control randomness. It defines a probability mass (e.g., 0.9) and samples from the smallest set of tokens whose cumulative probability exceeds that mass.

  • General advice: It is often better to tune Temperature or Top-P, but not both aggressively at the same time. A common setting for balanced behavior is temperature=0.7, top_p=0.9.
  • For deterministic tasks, you might lower top_p significantly to restrict the model's choices.

• A Critical Nuance: As the LINK article points out, research shows that for large models, changing the temperature has no statistically significant effect on problem-solving accuracy. Its main effect is on the expressiveness and diversity of the output. For reliability in high-stakes applications, architectural choices like implementing RAG or feedback loops (as discussed in the previous lesson) are far more impactful than just tuning temperature.

• Cross-Model Differences: The valid ranges and default behaviors for these parameters differ across providers (e.g., GPT-4's temperature range is 0-2, while Claude's is 0-1). When designing a system, you must consult the specific model's API documentation and never rely on unstated defaults. The decision tree in the Galileo article provides an excellent starting point for troubleshooting.

Conclusion

In this lesson, we have moved from theory to practice, exploring the three fundamental levers you control at inference time.

Key Takeaways:

• Context is a Finite, Expensive Resource: The context window's size is limited by the $O(n^2)$ complexity of self-attention. Effective Context Engineering involves designing systems that manage information flow between a small, active "working memory" (the context window) and larger, external "episodic" and "semantic" memories.
• Tokens are the Unit of Cost and Complexity: The non-linear relationship between characters and tokens, especially for code, structured data, and non-English languages, must be factored into the design and budget of any agentic system.
• Temperature Controls Creativity, Not Accuracy: Temperature and Top-P are your primary tools for tuning the trade-off between deterministic and creative outputs. However, for large models, these parameters have little impact on factual accuracy. Improving reliability requires architectural solutions, not just parameter tuning.

Preview of the Next Lesson:

You now have a foundational toolkit for controlling an agent's basic behavior. However, even with perfectly tuned parameters and well-managed context, LLMs exhibit inherent failure modes that can derail an agentic workflow. In our next lesson, we will identify the key failure modes of LLMs (hallucination, drift, instruction ignoring) and their implications for agent design. Understanding these will prepare you to build the necessary guardrails and validation mechanisms into your systems.