Optimizing Token Costs: Prompt Engineering & Caching

Recommended resources

The freely available resources this lesson will direct you to.

Hello! Welcome to the next lesson in your preparation for a software engineering role at Anthropic.

In our previous lesson, we designed a robust data model and architecture to aggregate and store LLM token usage. This gave us the visibility to understand where the costs are coming from. Today, we shift from measuring to managing. The learning outcome for this lesson is to describe technical strategies for reducing token costs, such as prompt engineering and caching.

This is a core engineering challenge in building applications with LLMs. Just as your work on the Rudi inference scheduler involved optimizing resource utilization for performance, these strategies focus on optimizing the workload itself for cost-efficiency. We'll explore two main pillars: shaping the requests we send (prompt engineering) and avoiding redundant computation (caching).

1. Understanding the Anatomy of LLM Inference Costs

Before we can reduce costs, we must understand their source. The total cost and latency of an LLM call are driven by two distinct phases of computation on the GPU.

Mastering LLM Inference Optimization From Theory to Cost Effective Deployment: Mark Moyou

To get a foundational understanding of the LLM inference process, watch this segment from the talk 'Mastering LLM Inference Optimization' by Mark Moyou. It breaks down what happens on the GPU when a prompt is processed and provides the intuition behind why certain operations are expensive.

Please watch the following segments, paying close attention to the concepts that drive cost: 02:15 - 04:20: Understand the basic flow: the prompt is processed, and tokens are generated one by one. Grasp the concept of the KV cache as the model's 'memory' that is stored on the GPU. 04:20 - 14:46: This is a more technical walkthrough. Focus on the two main stages: prefill (processing the entire input prompt to create the initial KV cache) and decoding (generating output tokens one at a time). Note how the KV cache avoids reprocessing the full prompt for each new token. 15:18 - 19:14: This is the most critical part. Absorb the four main querying patterns (e.g., long input/short output) and how they map to GPU memory usage and performance. This directly explains why different types of tasks have different cost profiles.

As the video explains, the two key phases are:

Prefill (Prompt Processing): The model processes your entire input prompt in one large, parallel computation to create the attention matrices (the KV cache). The cost of this phase is proportional to the number of input tokens. A long prompt means a costly prefill stage.
Decoding (Token Generation): The model generates the response one token at a time in a sequential loop. The cost of this phase is proportional to the number of output tokens. A long response means a costly decoding stage.

This fundamental split—cost from input tokens and cost from output tokens—is the basis for all cost optimization strategies. Our goal is to minimize the tokens in both phases without compromising the quality of the result.

2. Prompt Engineering for Cost Reduction

"Prompt engineering" isn't just about getting better answers; it's also about getting the same quality answer for less money. This involves a set of practical techniques to minimize token count. The following article provides an excellent, code-focused overview of these patterns.

The Frugal Approach to Anthropic Claude API Costs

The article 'The Frugal Approach to Anthropic Claude API Costs' offers a fantastic checklist of efficiency patterns. We'll walk through the most important ones related to prompt and output engineering.

Please read the following sections, which correspond to specific cost-reduction techniques. You can skim the code examples for now, as we'll discuss the concepts. Pattern 1: Route tasks to cheapest capable model Pattern 3: Compress prompts to minimize input tokens Pattern 4: Truncate older conversation messages Pattern 8: Set explicit max_tokens for every request Pattern 9: Use tool use for structured outputs Pattern 11: Test minimum effective example count

Let's synthesize those patterns into a concrete set of strategies.

Strategy 1: Smart Model Selection

As highlighted in Pattern #1 of the article, this is often the single most effective cost-reduction lever. Don't use your most powerful (and expensive) model for every task.

Simple Tasks: For classification, extraction, or simple Q&A, a small, fast model like Claude 3 Haiku is often sufficient and dramatically cheaper.
Complex Tasks: Reserve powerful models like Claude 3/4 Opus for tasks requiring deep reasoning.
Default Choice: A balanced model like Claude 3/4 Sonnet is a good default for a wide range of tasks.

Implementing a "router" that selects the model based on the task type is a common architectural pattern, which we will explore in the next lesson.

Strategy 2: Prompt Compression and Input Filtering

This is about reducing input tokens.

Be Concise: Replace verbose instructions like "Please carefully analyze the following text and provide a classification" with "Classify sentiment:". Every token saved adds up over millions of calls (Pattern #3).
Filter Noise: Before sending unstructured data like a web page to the model, use pre-processing libraries (e.g., BeautifulSoup for HTML) to strip out boilerplate, navigation, ads, and excessive whitespace (Pattern #5). This can reduce token count by over 80%.
Manage Conversation History: In chatbots, don't send the entire conversation history with every turn. Implement a sliding window or summarization strategy to keep only the most recent and relevant context (Pattern #4).

Strategy 3: Output Token Control

Output tokens are typically much more expensive than input tokens. Unconstrained generation is a huge financial risk.

Set max_tokens: Always set an explicit max_tokens limit on your API calls. This acts as a crucial safety rail to prevent unexpectedly long and expensive responses (Pattern #8).
Use Tool Use for Structure: When you need a machine-readable output like JSON, don't ask the model to "write JSON". Instead, use the tool_use feature. This forces the model to generate a compact, structured call to a tool you've defined, which uses far fewer tokens than generating a verbose natural language response with a code block (Pattern #9).

Strategy 4: Optimize Few-Shot Examples

Few-shot examples (providing examples of the task in the prompt) can improve quality, but they also increase input tokens.

Find the Minimum Effective Dose: Systematically test the number of examples needed to achieve your target accuracy. You might find that 3 examples give you 92% accuracy, while 10 examples only increase it to 94% at 3x the input cost (Pattern #11).

3. Caching: Avoiding Redundant Work

Caching is a powerful technique for reducing both cost and latency. If we've already paid to compute something, we shouldn't pay to compute it again. There are two main levels of caching in LLM applications.

Level 1: Prompt Caching (KV Cache Caching)

This is a specific, low-level optimization that avoids the expensive prefill step for prompts that have been seen recently. It doesn't cache the final string output; it caches the model's internal KV cache state.

What is Prompt Caching? Optimize LLM Latency with AI Transformers

First, let's get a clear conceptual understanding of prompt caching. This video from IBM Technology does a great job of explaining what it is and, importantly, what it is not.

Watch the video until 07:52. Focus on these key ideas: The distinction between output caching (caching the final text response) and prompt caching (caching the intermediate KV pairs). The types of content that are ideal for prompt caching: large documents, static system prompts, and few-shot examples. The importance of prompt structure: static, cacheable content must come before dynamic, user-specific content for the cache to work.

Now that you have the concept, let's look at Anthropic's specific implementation, as this is what you'd use in a production system built on Claude.

How Prompt Caching Works in Claude — This flowchart visualizes the logic of prompt caching. An API request's prefix is checked against the cache. A hit results in a 90% cost reduction for that portion, while a miss incurs a write cost (1.25x the base) to populate the cache for future requests.

The official documentation contains the critical details for implementation.

Prompt caching - Claude Docs

The Anthropic 'Prompt caching' documentation is essential reading. It details the API, the pricing trade-offs, and best practices for using this feature effectively.

Please read the following sections from the documentation: Introduction & How it works: Understand the core mechanism and the 5-minute default TTL. Pricing: This is crucial. Notice that a cache write costs more (1.25x) than a standard input, while a cache read is significantly cheaper (0.1x). This is a direct engineering trade-off. Explicit cache breakpoints: Understand how you get fine-grained control by placing cache_control on specific blocks. Best practices for effective caching: This provides a great summary of how to structure your prompts. Optimizing for different use cases: See concrete examples of how you would apply this for chatbots, document Q&A, etc.

Key Trade-offs for Prompt Caching:

Cost: You pay a premium (1.25x) for the initial cache write. The savings only materialize on subsequent cache hits within the TTL (e.g., 5 minutes). This makes it ideal for high-volume applications where the same large prefix is reused frequently.
Latency: It dramatically reduces time-to-first-token for prompts with large prefixes, as the expensive prefill step is skipped.
Flexibility: It requires careful prompt structuring. Any change to the cached prefix, no matter how small, invalidates the cache.

Level 2: Application-Level Caching (Output Caching)

This is a more traditional form of caching where you store the final output of an API call and reuse it for identical or similar future requests.

1. Hash-Based Caching:
The simplest method is to cache the full response using a key generated from the request parameters (model, prompt, etc.). As seen in Pattern #7 of the Frugal article (LINK), this is effective for repeated, identical queries like FAQ answers. You would typically use a store like Redis or Memcached for this.

2. Semantic Caching:
A more advanced technique is to cache based on semantic similarity, not just exact string matches.

Embedding Vector Search Process Flowchart — This flowchart shows the process for semantic caching. A new prompt is converted into a vector embedding. The system searches a vector database for semantically similar prompts that have already been answered. If a close match is found, the cached response is returned, avoiding an LLM call.

This approach uses embedding models to represent the meaning of a prompt. It allows you to serve cached responses for questions that are phrased differently but are substantively the same (e.g., "How much does it cost?" vs. "What is the price?"). This can significantly increase your cache hit rate compared to simple hash-based caching.

Conclusion

Today we explored a suite of powerful engineering techniques to control and reduce LLM API costs. By understanding the fundamental cost drivers of inference, we can apply targeted optimizations that have a real impact on the bottom line.

Key Takeaways:

LLM costs stem from two phases: prefill (input tokens) and decoding (output tokens).
Prompt engineering strategies like model tiering, prompt compression, history management, and output controls can dramatically reduce the number of tokens processed.
Prompt Caching is a powerful, low-level feature that avoids re-computing the expensive prefill step for repeated prompt prefixes, offering significant cost and latency savings at the cost of a small write premium.
Application-Level Caching, from simple hash-based methods to more advanced semantic search, avoids redundant API calls entirely by reusing previous responses.

Next Lesson Preview

We've now identified several cost-saving strategies, with smart model selection being one of the most effective. But how do you implement this in a real system? In our next lesson, we will design a cost-aware request routing system that balances model quality, latency, and token cost, turning the theory of model tiering into a practical architectural design.