Crafting Context: Prioritizing Relevance within Token Limits

Introduction

In our last lesson, we categorized the information an agent consumes into three types: static, dynamic, and conversational. This taxonomy helps us understand the sources of information. Now, we confront a fundamental engineering challenge: what happens when the total information from these sources exceeds the model's token limit?

This lesson addresses that question directly, fulfilling the learning outcome: Design a context assembly strategy that prioritizes the most relevant information within token limits. We will move beyond simply identifying context types to architecting a "compiler" that intelligently selects, compresses, and structures information. Your background in computer science will be valuable here, as we'll draw analogies to memory management, compiler design, and system architecture.

1. The Real Bottleneck: Maximum Effective Context Window

Large Language Model providers often advertise massive context windows—hundreds of thousands or even millions of tokens. However, there's a critical distinction between the architectural limit and the practical limit of a model's attention.

Let's ground this in a formal concept that is gaining traction in the field.

As the paper highlights, simply "stuffing" the context window is not a viable strategy. Beyond the MECW, you face three primary problems:

1. Cost and Latency: Every token has a cost in terms of both money and processing time. Uncurated context leads to slow, expensive, and inefficient systems.
2. Signal Degradation: This is the "lost in the middle" problem. When critical information is buried in a sea of irrelevant data (stale chat history, verbose tool outputs), the model's attention can be diluted, leading to incorrect or unfocused responses.
3. Cascading Failures: In a multi-agent system, if one agent's output is degraded by poor context management, it becomes flawed input for the next agent. This creates a chain reaction of errors, as defined in Appendix A.2 of the paper you just reviewed.

Our goal, therefore, isn't just to stay within the token limit, but to maximize the signal-to-noise ratio of the information we provide to the model.

2. A Systems Approach: Context as a Compiled View

To manage context systematically, we need a robust mental model. Instead of thinking of the prompt as a simple string we append to, we should treat it as a compiled view over a richer, underlying stateful system. This reframes context management from "prompt engineering" to "systems engineering."

This "compiler" thesis gives us a powerful architectural pattern. The Session, Memory, and Artifacts are our source code—the ground truth. The Working Context is the executable—the optimized, just-in-time compilation of that source code, built specifically for one model invocation. This separation allows us to iterate on our context assembly "compiler" without altering the underlying data.

3. Core Assembly Strategies

So, how does this "compiler" work? The process of transforming our source information into the final working context can be broken down into a set of core strategies.

Let's dissect these strategies with a focus on prioritization within token limits.

a) Selection: Choosing What Matters Most

Selection is about pulling the most relevant pieces of information from a large corpus. This is the heart of Retrieval-Augmented Generation (RAG) and tool use.

The most common selection process involves:

1. Retrieval: Using a technique like vector similarity search to find potentially relevant chunks of information from a knowledge base (dynamic context).
2. Reranking: Recognizing that the initial retrieval is often imprecise, a reranking step is applied to re-order the retrieved chunks, pushing the most relevant items to the top. This is a crucial prioritization step.

The diagram below illustrates an advanced RAG pipeline that explicitly includes this reranking stage.

The same selection principle applies to tools. For an agent with dozens of available tools, you can use a RAG-based approach to select only the most relevant tool descriptions to include in the prompt, rather than all of them.

b) Compression: Saying More with Fewer Tokens

Once we've selected relevant information, it might still be too verbose. Compression techniques aim to reduce token count while preserving semantic meaning.

Two primary methods for this are:

• Summarization: For long documents or chat histories, a separate LLM call can be used to generate a concise summary. The Google ADK article refers to this as "context compaction," where older parts of a conversation are summarized and replaced, creating a rolling window of memory.
• Token-level Pruning: More advanced techniques can intelligently remove "low-entropy" words or sentences that don't significantly contribute to the overall meaning.

The LongLLMLingua framework provides a compelling example of an advanced compression strategy.

c) Isolation: Divide and Conquer

Instead of creating one massive context for a single monolithic agent, we can isolate context within specialized agents in a multi-agent system.

• Scoped Handoffs: When a primary agent delegates a task to a sub-agent, it doesn't pass its entire context. Instead, it provides a minimal, "scoped" context containing only the information relevant to that specific sub-task. This prevents the "context explosion" problem and keeps each agent focused.
• Externalizing State ("Artifacts"): Large data objects like files or extensive API responses shouldn't be pasted directly into the context. Instead, they can be stored externally as "Artifacts," as described in the Google ADK article. The agent is given a lightweight reference or summary in its context and can use a tool to load the full artifact only when necessary. This is conceptually similar to passing a pointer or handle instead of the full data structure.

4. A Practical, Tiered Assembly Strategy

Now, let's synthesize these strategies into a practical, repeatable process for assembling the final working_context for each model call. This is a heuristic algorithm that prioritizes information based on its likely signal density.

The Assembly Pipeline:

1. Start with Static Context (Highest Priority):

   • Always include the full system instructions, agent role, boundaries, and output schema. This information is high-signal and usually has a small, fixed token cost.
   • This part of the context is ideal for prefix caching, as it remains stable across multiple turns.

2. Select and Add Dynamic Context:

   • Based on the user's query, retrieve relevant chunks from your knowledge base (e.g., using RAG).
   • Rerank these chunks to place the most relevant ones first.
   • Add the top-k chunks to the context, ensuring you leave room for conversational history.

3. Compress and Add Conversational Context:

   • Retrieve the conversation history.
   • Keep the most recent N turns verbatim, as they are most likely to be relevant.
   • For older turns, apply a compression strategy (e.g., summarization) to create a condensed historical record.

4. Final Assembly and Truncation:

   • Combine these components in a deliberate order, for example: [Static Context] [Retrieved Dynamic Context] [Compressed Chat History] [Current User Query].
   • Calculate the total token count. If it exceeds the MECW, truncate from the lowest-priority part first. In this structure, that would be the oldest, summarized parts of the chat history.

This pipeline ensures that the most critical information—the agent's core instructions and the most relevant retrieved facts—is always preserved, while less critical information is gracefully compressed or truncated.

Conclusion

In this lesson, we established a systematic and architecturally-sound approach to context management. We moved from an ad-hoc process of "stuffing the prompt" to a disciplined "compilation" of a working context.

Key Takeaways:

• Focus on the MECW: The Maximum Effective Context Window, not the advertised limit, is your true design constraint. Exceeding it degrades performance and increases cost.
• Adopt a Systems Mindset: Treating context assembly as a "compiler" pipeline—separating storage (source) from presentation (view)—leads to more robust and maintainable systems.
• Prioritize with a Multi-faceted Strategy: A combination of Selection (retrieval/reranking), Compression (summarization/pruning), and Isolation (scoping/externalization) is required to manage token limits effectively.
• Implement a Tiered Assembly Pipeline: A practical strategy involves prioritizing static instructions, then high-relevance retrieved data, and finally compressed conversational history, truncating the least important information first.

Preview of the Next Lesson

We now have a strategy for assembling the best possible context from the information we have. But what if the information itself is incomplete? Even the most sophisticated assembly strategy cannot invent knowledge that doesn't exist in the agent's information environment.

In the next lesson, we will focus on diagnostics. We will learn how to identify gaps in an agent's information environment that would cause poor or unreliable outputs. This involves analyzing agent failures to determine whether the problem lies in the context assembly strategy or in the underlying source information itself.