Optimizing Agentic System Costs

Introduction

In our last lesson, we built a framework for estimating the costs of an agentic workflow, revealing how "invisible multipliers" like iterative loops and context-passing can cause budgets to spiral. Estimation is the first step, but as an architect of AI services, your ultimate goal is active control. This lesson moves from prediction to optimization.

The learning outcome for this lesson is to apply optimization strategies (model selection, prompt compression, and caching) to reduce the operational costs of an agentic system. These aren't just tricks to save a few dollars; they are fundamental architectural patterns for building efficient, scalable, and profitable agentic solutions. By mastering them, you can deliberately navigate the cost-performance trade-off, delivering maximum value to your clients.

We will explore three pillars of cost optimization:

1. Strategic Model Selection: Using the right model for the right task.
2. Prompt & Context Compression: Reducing the "communication tax" by minimizing token inputs.
3. Caching: Eliminating redundant computations through intelligent reuse of previous results.

1. Strategic Model Selection: The Right Tool for the Job

The most direct way to control cost is by choosing the right LLM. The price difference between a frontier model (like GPT-4o) and a smaller, faster model (like Llama 3 8B or GPT-3.5 Turbo) can be over 100-fold. However, simply defaulting to the cheapest model sacrifices performance. The architectural challenge is to use expensive models only when necessary.

This involves designing systems that can dynamically select models based on the task at hand. The video below introduces several powerful patterns for achieving this.

These patterns—cascading, routing, and specialization—are heuristics for a more fundamental problem: how do you formalize the decision of allocating resources in a multi-agent system to balance cost and performance?

A recent research paper provides a more structured and formal approach to this problem, which will resonate with your background in system design.

The key architectural takeaway is to design your systems with explicit routing logic. This can be as simple as a few rules in your orchestrator or as sophisticated as a machine-learned router like MasRouter. For a client project, a practical starting point is a router agent: a simple agent, powered by a cheap model, whose sole job is to classify incoming tasks and delegate them to specialized agents that may be powered by more capable—and expensive—models.

2. Prompt & Context Compression: Reducing the Communication Tax

In the last lesson, we identified that input tokens—the "communication tax" of providing context to agents—are often the single largest driver of cost. Prompt compression techniques directly attack this problem by reducing the number of tokens sent to the model without losing essential information.

A leading technique in this area is LLMLingua, which uses a smaller, faster LLM to prune a prompt of words that are less critical for the final output.

This next video segment demonstrates the concept of prompt compression and another highly practical technique: summarizing tool outputs.

Applying these patterns in your system design is straightforward:

• For RAG workflows: When retrieving large documents, don't just stuff the raw chunks into the context. First, pass the chunks to a cheap model (e.g., GPT-3.5 Turbo) with a prompt like, "Summarize the following text, focusing on information relevant to the user's query: [user query]". Then, feed this condensed summary into your main agent's context.
• For conversational agents: Instead of using a memory buffer that retains the entire chat history verbatim, use a summary buffer. This type of memory keeps recent messages verbatim but periodically uses a cheap LLM to summarize older parts of the conversation, keeping the context window from growing indefinitely.

3. Caching Strategies: Eliminating Redundant Work

Caching is a classic optimization technique in computer science. In the context of LLMs, it can be applied at two different levels: the application level and the inference level.

3.1. Application-Level Caching: Semantic Caching

Many applications, especially those for customer support or Q&A, receive high volumes of repetitive or semantically identical queries. Calling an LLM for each one is wasteful. Semantic caching solves this by storing the results of previous queries and reusing them.

Unlike a simple key-value cache that requires an exact match of the input string, a semantic cache works by finding queries that are similar in meaning. The architecture for such a system is a perfect fit for your background.

Implementing a semantic cache is a high-leverage optimization for any service that expects query repetition. It directly translates to significant cost savings and latency reduction.

3.2. Inference-Level Caching: The KV Cache

While semantic caching prevents redundant API calls, another form of caching happens during a single API call to make token generation efficient. This is the KV Cache.

To understand it, recall that a transformer's attention mechanism relates every token to every other token that came before it. A naive implementation would require re-computing attention scores over the entire sequence for each new token generated. This would make generation quadratically expensive with sequence length.

The KV Cache optimizes this. For each token in the input, the model computes a Key and a Value vector. These vectors are then stored in GPU memory. When generating the next token, the model only needs to compute the Query vector for the new token and compare it against all the cached Keys and Values.

The following video provides a clear, intuitive explanation of the KV Cache and its critical role in making LLM inference practical.

You don't typically implement the KV cache yourself; it's handled by highly optimized inference engines (like NVIDIA's TensorRT-LLM, vLLM, or Hugging Face's TGI). However, understanding its existence is crucial for an architect. It explains why the cost of processing the initial prompt (prefill) can be high, while subsequent token generation (decode) is faster and more incremental. It also underscores why context length is a major factor in memory usage and cost—every token in the context consumes GPU memory for its K and V vectors.

Conclusion

In this lesson, you have learned to move beyond cost estimation to active cost control. By applying a combination of these three architectural strategies, you can design agentic systems that are not only powerful but also economically viable.

Key Takeaways:

• Optimization is a Design-Time Activity: Cost control is not an afterthought. Decisions about model selection, routing logic, context management, and caching are core architectural choices that define a system's efficiency.
• The Three Pillars of Cost Optimization:
  1. Model Selection: Use routers and cascades to deploy expensive models judiciously.
  2. Prompt Compression: Actively fight the "communication tax" by summarizing tool outputs and using intelligent memory.
  3. Caching: Employ semantic caching to eliminate redundant API calls at the application level, and understand how KV caching makes inference efficient at the hardware level.
• No Silver Bullet: These techniques are most powerful when used in combination. A well-architected system might use a router to select an agent, which then uses a compressed prompt to formulate a query that ultimately hits a semantic cache.

In our next lesson, we will focus on applying version control strategies to system instructions and agent configurations to manage their evolution. As you tweak your prompts, routing logic, and caching thresholds to optimize for cost and performance, you'll need a systematic way to track these changes, measure their impact, and ensure reproducibility—a process that is central to building production-grade systems.