Optimizing AI Costs: Caching & Deduplication

1. The Caching Principle and Its Limitation

In traditional software, caching is a standard technique. If a system receives the same request multiple times, it stores the result of the first request and serves it directly for all subsequent identical requests, skipping the expensive computation.

However, with Large Language Models, this traditional, exact-match caching is often ineffective. Why? Because human language is fluid. Consider a user wanting to reset their password. They might ask:

• "How do I reset my password?"
• "I forgot my password"
• "Can't log in, need password help"

To a traditional cache, these are three unique strings, resulting in three separate, expensive calls to the LLM to get what is essentially the same answer. This is where a more intelligent approach is needed.

2. Semantic Caching: Caching by Meaning

The solution is semantic caching: a technique that matches queries based on their meaning (semantics), not their exact wording.

To get a quick overview of how this works, please watch this short video from Redis.

As the video explained, the process relies on converting text into numerical representations called vector embeddings. The system can then mathematically compare these vectors to see how "close" they are in meaning.

This diagram illustrates the architectural flow:

Now, let's dive into the practical impact and architectural considerations of implementing this.

From an architect's perspective, the key takeaways from this reading are:

• Significant Cost & Latency Reduction: Semantic caching can eliminate 40-70% of LLM calls, leading to massive cost savings and drastically lower response times for users—a huge win for UX.
• Tunable Parameters: You don't just "turn on" caching. You must make deliberate design choices.
  • Similarity Threshold: A higher threshold (e.g., 0.95) means higher accuracy but fewer cache hits. A lower threshold (e.g., 0.80) increases hits but risks returning irrelevant answers. Your choice depends on the application's tolerance for error.
  • Time-to-Live (TTL): This determines how long a cached response is valid. For rapidly changing information (stock prices), the TTL might be minutes. For stable content (company policies), it could be hours or days. This ties directly into the data freshness principles we'll discuss later in the course.

3. Request Deduplication: The Underlying Strategy

"Semantic Caching" is a specific implementation of a broader architectural strategy often called Semantic Deduplication. The goal is the same: reduce redundant requests by identifying semantic duplicates.

Let's read a short section from an article that frames this from a systems-scaling perspective.

This article reinforces the concept and highlights a key insight: efficient AI is achieved by reducing unnecessary calls. This principle is a cornerstone of good AI system architecture.

4. An Important Nuance: Output vs. Prompt Caching

So far, we've discussed caching the final output of an LLM call. This is known as output caching. However, there's another powerful technique an architect should be aware of: prompt caching.

Prompt caching doesn't store the final answer. Instead, it stores the LLM's internal, pre-processed understanding of the input prompt. This is extremely useful when a large part of your prompt is static.

Imagine a user uploading a 50-page document and then asking multiple questions about it.

• Query 1: [50-page document] + "Summarize this."
• Query 2: [50-page document] + "What are the key risks mentioned?"

With prompt caching, the expensive process of reading and understanding the 50-page document only happens once. For the second query, the system retrieves the cached understanding of the document and only needs to process the new question, saving significant time and cost.

This video from IBM Technology explains the distinction clearly.

Understanding both output caching (for handling repetitive, semantically similar queries) and prompt caching (for handling repetitive context with varying questions) gives you a more complete toolkit for optimizing AI economic performance.

Conclusion

In this lesson, we've established caching and deduplication as fundamental strategies for controlling the direct operational costs of AI. By intelligently avoiding redundant work, we can build systems that are not only cheaper to run but also faster and more responsive for the user.

Key Takeaways:

• Semantic Caching / Request Deduplication is a critical technique that uses vector embeddings to identify and reuse answers to semantically similar queries, dramatically reducing LLM API costs and latency.
• As an architect, you must configure key parameters like the similarity threshold (balancing accuracy and hit rate) and Time-to-Live (TTL) (ensuring data freshness).
• The choice of caching strategy depends on the use case. Output caching is ideal for high-volume, repetitive questions (like in an FAQ bot). Prompt caching is powerful for scenarios involving a large, static context with multiple follow-up questions.
• The guiding principle is to shift from a "call the LLM for every request" model to one where only truly unique requests incur the full cost of inference.

Next Steps

We've now learned how to avoid making unnecessary calls to the LLM. But what about the calls that are truly unique and necessary? How can we make those individual calls more efficient? In our next lesson, we will explore batch inference, a technique for grouping multiple requests into a single operation to improve throughput and further optimize costs.