LLM Context Windows: Impact on Agent Performance and Memory

Introduction

In our previous lesson, we established how text embeddings allow an AI agent to convert unstructured text into a numerical format, enabling it to find semantically relevant information through vector search. This is the foundation of an agent's ability to access external knowledge.

Today, we address the next critical question: once information is found, how does the agent use it? This brings us to the concept of the Large Language Model's (LLM) context window. The learning outcome for this lesson is to: Analyze the impact of LLM context windows on agent performance and memory.

The context window is the agent's working memory. It's the finite space where all information—system instructions, user queries, conversation history, and retrieved documents—must reside for the LLM to process it. We will analyze why the intuitive assumption that "a bigger context window is always better" is a dangerous oversimplification. You will learn about performance degradation phenomena like "context rot" and, more importantly, the discipline of context engineering—the set of strategies for managing this crucial, finite resource to build reliable and effective agents.

The Context Window: An Agent's Working Memory

Think of an LLM's context window as the RAM of a computer. It's a temporary workspace that holds all the information the model needs to perform its current task. Once the task is complete (i.e., a response is generated), the contents of that specific window are gone unless explicitly preserved for the next turn.

The contents of this window are a composite of multiple information sources.

As you can see, this space gets crowded quickly. For an agent performing a complex task like underwriting analysis, the context window must accommodate:

• System Prompt: The core instructions defining the agent's persona, goals, and constraints.
• Tool Definitions: The specifications for any APIs or functions the agent can use (e.g., a tool to look up a customer's credit score).
• Conversation History: The record of previous interactions, which provides conversational context.
• Retrieved Data: The relevant document chunks found via semantic search (as discussed in Lesson 1).
• User Prompt: The most recent instruction from the user.

The Costs of Context

Larger context windows come with two significant and non-negotiable costs that a solutions architect must consider:

1. Financial Cost: LLM providers bill based on the number of tokens processed (both input and output). A larger context window directly translates to higher operational costs per agent interaction.
2. Computational Cost: A larger context requires more computational resources, specifically VRAM (video memory) on the GPUs that run the model. This is a critical constraint for performance and a major cost driver, especially for self-hosted solutions.

These costs create a strong incentive to use the context window efficiently. However, there's an even more critical reason to manage it carefully: performance.

The Fallacy of Infinite Context: Context Rot and Attention Budget

While models with million-token context windows are now available, research and practice show that model performance is not uniform across the entire window. As the context gets longer, the model's ability to accurately recall and reason over information begins to degrade. This phenomenon is often called context rot.

To understand this problem in detail, please watch the following video from Chroma, which presents findings from their research on the topic.

The video's findings, which are detailed in their technical report "Context Rot," highlight several critical failure modes:

• The "Lost in the Middle" Problem: Information placed at the beginning or end of a long context is more likely to be recalled correctly than information placed in the middle.
• Increased Sensitivity to Distractors: In a short context, a model can easily distinguish the correct answer from a similar-but-wrong "distractor." In a long context filled with noise, the model is more likely to get confused and select the distractor.
• Inconsistent Reasoning: The model's ability to perform tasks it can handle perfectly well at shorter lengths becomes unreliable as the context grows.

The Architectural Reason: Finite Attention Budget

This degradation isn't a random flaw; it's a consequence of the LLM's underlying transformer architecture. For a more technical explanation, let's turn to a clear article from Anthropic.

As the article explains, every token in the context window must be able to "attend" to every other token. This creates a quadratic increase in relationships ($n^2$) that the model must process. As context length ($n$) grows, the model's fixed capacity to manage these relationships—its "attention budget"—is spread thin, reducing its precision and reasoning ability.

Context Engineering: A Framework for Managing Memory

Given that the context window is a finite and performance-sensitive resource, we cannot simply feed an agent all available information. We must actively manage what goes into it. This discipline is known as context engineering.

A highly effective way to structure your thinking about this is provided in a video from LangChain, which organizes context engineering strategies into four categories.

Let's synthesize these ideas with insights from the Anthropic article to build a robust framework. The goal is always to provide the smallest possible set of high-signal tokens that maximizes the agent's chance of success.

1. Compressing / Compaction

This involves reducing the number of tokens in the context without losing critical information.

• Summarization: For long-running conversations, instead of including the entire chat history, you can periodically use the LLM to summarize the conversation so far. This "compacted" summary replaces the raw history, freeing up thousands of tokens. Anthropic's Claude Code agent, for instance, does this when a session nears the context limit.
• Trimming: A simpler, heuristic-based approach. For example, you might decide to always keep the first message (for initial instructions), the last 5 messages (for immediate context), and discard the rest.

2. Selecting

This strategy involves choosing only the most relevant information to include in the context in the first place.

• Retrieval-Augmented Generation (RAG): This is the most powerful selection technique and the primary application of what we learned in Lesson 1. Instead of giving an agent an entire 500-page policy manual, you use semantic search to retrieve only the 2-3 most relevant paragraphs related to the user's specific query. This is the core topic of our next lesson.
• Selective Tool Availability: Instead of providing the definitions for all 100 tools an agent could use, you can use a preliminary LLM call or an embedding-based search to select only the 5-10 most relevant tools for the current task and include only those in the context.

3. Writing / Structured Note-Taking

This is a form of "agentic memory," where the agent is empowered to write down information to an external store (outside the context window) for later retrieval.

• Scratchpad/Notes: The agent can maintain a NOTES.md or a structured file where it logs key decisions, unresolved questions, or a to-do list. Before a new task, it can read its notes to regain context. This is more reliable and token-efficient than trying to deduce state from a long, messy conversation history.

4. Isolating

This strategy involves breaking down a problem so that different parts can be processed in separate, "clean" context windows.

• Multi-Agent Architectures: Instead of one agent trying to solve a complex problem (e.g., "produce a market research report"), a main "orchestrator" agent can delegate sub-tasks to specialist "sub-agents" (e.g., a "web-search-agent," a "data-analysis-agent"). Each sub-agent operates in its own context window, performing deep work, and returns only a concise summary to the orchestrator. This parallelizes work and dramatically expands the total amount of information the system can process.
• Sandboxing: For agents that write and execute code, the code can be run in a separate, sandboxed environment. This environment can maintain state across turns (e.g., variables, file contents) without having to pass all of that information back into the LLM's context window. Only the essential results (e.g., the output of print(), or an error message) are returned to the agent.

These techniques are not mutually exclusive. A sophisticated agent will use a combination of them.

Conclusion

In this lesson, we have moved from the naive view of the context window as a simple data buffer to a more sophisticated understanding of it as a limited and performance-critical resource.

Key Takeaways:

• The context window is the LLM's finite working memory (analogous to RAM), which has direct financial and computational costs.
• Performance is not uniform across the context window. As it fills, models suffer from "context rot," where their ability to recall information and reason reliably degrades, especially in the presence of ambiguity and distractors.
• This performance degradation is a result of the underlying transformer architecture and its finite "attention budget."
• Context engineering is the critical discipline of managing this resource. It is not about simply making the context bigger, but making it smarter.
• The primary strategies for context engineering are compressing (summarization), selecting (retrieval), writing (external memory), and isolating (multi-agent systems).

Understanding and applying these principles is fundamental to moving from building simple chatbots to architecting robust, autonomous agents that can perform complex, multi-step tasks reliably.

Preview of the Next Lesson:

We identified "Selection" via Retrieval-Augmented Generation as one of the most powerful context engineering strategies. It allows an agent to precisely target the information it needs from vast external knowledge bases. In the next lesson, "Design a Retrieval-Augmented Generation (RAG) pipeline to provide agents with external knowledge," we will combine our knowledge of embeddings and context windows to design a complete, end-to-end RAG system.