Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 44

Quantifying AI Cost Savings Through Caching & Batching

Recommended resources

Introduction

Hello! Welcome back. In our previous lesson, we established the fundamental difference between real-time and batch processing for LLMs. We learned that batching is a powerful strategy for reducing costs on non-urgent tasks by improving hardware efficiency, while caching saves money by avoiding redundant processing of repeated prompts.

Today, we'll make these concepts tangible. This lesson is designed to help you master the learning outcome: Recalculate AI feature costs after applying a caching or batching strategy to quantify savings.

This is a crucial skill for your journey as a systems architect. It's not enough to know that a strategy saves money; you need to be able to prove how much it saves. By the end of this lesson, you will be able to take a baseline feature cost, apply an optimization strategy, and calculate the concrete financial impact. This ability to connect technical decisions to business outcomes is what separates a designer from an architect.

1. The Framework for Calculating Savings

Before we dive into examples, let's establish a simple, four-step framework you can use to calculate the financial impact of any optimization strategy.

  1. Establish the Baseline Cost: Calculate the total cost of the feature without any optimizations. This requires knowing your key metrics:
    • Number of requests (per day or month).
    • Average input and output tokens per request.
    • The pricing of the LLM you are using (e.g., cost per 1 million tokens).
  2. Identify the Optimization Opportunity: Analyze the feature's usage patterns to decide which strategy applies.
    • For Batching: What percentage of requests are not time-sensitive and can be processed asynchronously?
    • For Caching: What parts of your prompts are static and repeated across many requests (e.g., system instructions, few-shot examples)?
  3. Recalculate the Cost: Apply the relevant cost-saving model.
    • Batching: Apply the discount (typically ~50%) to the portion of traffic you identified for batching.
    • Caching: Apply the discount (often ~90%) to the static (cached) tokens in your prompts. Remember to account for any one-time "write" costs if the model has them.
  4. Quantify the Savings: Compare the new cost to the baseline.
    • Absolute Savings: Baseline Cost - New Cost
    • Percentage Savings: (Absolute Savings / Baseline Cost) * 100%

Now, let's apply this framework to both batching and caching.

2. Worked Example: Recalculating Costs with Batching

In our last lesson, we discussed that batching is ideal for non-urgent workloads. Let's see what this means for our budget. We'll use a clear example from an excellent article on LLM cost reduction.

Run-Rate Shock: Quick Math

The article 'Run-Rate Shock' provides clear, worked examples for calculating savings from both batching and caching. We'll focus on the batching example first.

Please read the section 'Quick Math: From Inputs to Monthly Savings' and focus specifically on 'Worked example A (Batch)'. Pay attention to how the baseline cost is modified by the percentage of traffic that can be batched.

Let's break down the calculation from the article step-by-step.

Scenario:

  • Total Monthly Tokens: 30 million (25M input, 5M output)
  • Model Pricing: $0.0025 per 1K input tokens, $0.01 per 1K output tokens.
  • Optimization: We identify that 60% of this workload is from offline analysis and can be moved to a Batch API, which offers a 50% discount.

Step 1: Baseline Cost (100% Real-Time)

  • Input Cost: (25,000,000 tokens / 1,000) * $0.0025 = $62.50
  • Output Cost: (5,000,000 tokens / 1,000) * $0.01 = $50.00
  • Total Monthly Cost (Baseline): The example states a total of ~$112.5K. Let's adjust the token counts to match this total for consistency with the resource. Let's assume the monthly volume is 30 billion tokens, not 30 million, to get to that cost.
    • Input Cost: (25,000 M tokens / 1,000) * $0.0025 = $62,500
    • Output Cost: (5,000 M tokens / 1,000) * $0.01 = $50,000
    • Total Baseline Cost: $112,500

Step 2 & 3: Recalculate with Batching

The traffic is now split into two streams:

  • Real-time Stream (40% of traffic): This portion's cost is 40% of the baseline.

    • $112,500 * 0.40 = $45,000

  • Batch Stream (60% of traffic): This portion's cost is 60% of the baseline, with a 50% discount applied.

    • ($112,500 * 0.60) * 0.50 = $67,500 * 0.50 = $33,750

  • New Total Monthly Cost: $45,000 (Real-time) + $33,750 (Batch) = $78,750

Step 4: Quantify Savings

  • Absolute Savings: $112,500 - $78,750 = $33,750 per month
  • Percentage Savings: ($33,750 / $112,500) * 100% = 30%

By identifying the non-urgent workload and shifting it to a batch API, we've reduced the monthly bill by 30% without affecting the user experience for interactive features.

3. Worked Example: Recalculating Costs with Caching

Caching works differently. Instead of discounting entire requests, it discounts repeated tokens within requests. Let's start with a visual to reinforce how it works.

LLM Query Processing: With and Without Caching
This flowchart shows the architectural difference caching makes. On the left, a query first checks the cache. A 'hit' returns a stored response instantly, avoiding an LLM call. On the right, every query goes directly to the LLM, incurring full cost and latency every time.

Modern LLM APIs (like those from Anthropic and OpenAI) can automatically detect and discount repeated prompt content. Let's see how to calculate this.

How and When to Use Anthropic's Prompt Caching Feature (with code examples)

The video 'How and When to Use Anthropic's Prompt Caching Feature' provides a practical walkthrough. It includes a 'poor man's calculator' to estimate cost savings, which is directly relevant to our goal.

Please watch from 06:49 to 11:28 to understand the pricing mechanics, and then from 18:44 to 21:19 to see how follow-up queries benefit from the cache and how savings are estimated.

As the video explains, the key is that cached tokens are priced much lower (e.g., 90% cheaper) than new tokens. Now, let's use the second worked example from the "Run-Rate Shock" article to put numbers to this.

Run-Rate Shock: Quick Math

Now, let's return to the 'Run-Rate Shock' article and look at the caching example.

Please read 'Worked example B (Caching)' in the same 'Quick Math' section. This time, focus on how the cost of the prompt is broken down into static (cacheable) and dynamic parts.

Let's break down this calculation.

Scenario:

  • Requests per Month: 20,000
  • Prompt Structure: Each prompt has a 1,500-token static header (instructions, examples) and a 1,000-token dynamic portion (user query).
  • Model Pricing: $3 per million input tokens.
  • Optimization: We enable prompt caching, which offers a 90% discount on cached tokens. Let's assume a 100% hit rate for simplicity (the header is always the same).

Step 1: Baseline Cost (No Caching)

  • Total Tokens per Request: 1,500 (static) + 1,000 (dynamic) = 2,500 tokens
  • Total Monthly Input Tokens: 20,000 requests * 2,500 tokens/request = 50,000,000 tokens
  • Total Baseline Cost: (50,000,000 tokens / 1,000,000) * $3 = $150

Step 2 & 3: Recalculate with Caching

We now calculate the cost of the static and dynamic parts separately.

  • Cost of Dynamic Tokens: This part is never cached.

    • Tokens: 20,000 requests * 1,000 dynamic tokens/request = 20,000,000 tokens
    • Cost: (20,000,000 / 1,000,000) * $3 = $60

  • Cost of Static (Cached) Tokens: This part gets the 90% discount.

    • Tokens: 20,000 requests * 1,500 static tokens/request = 30,000,000 tokens
    • Baseline Cost for this part: (30,000,000 / 1,000,000) * $3 = $90
    • Discounted Cost: $90 * (1 - 0.90) = $90 * 0.10 = $9
      (Note: Some APIs have a small one-time "write" cost for the first time content is cached, but for ongoing calculations, the read discount is the main factor.)

  • New Total Monthly Cost: $60 (Dynamic) + $9 (Static) = $69

Step 4: Quantify Savings

  • Absolute Savings: $150 - $69 = $81 per month
  • Percentage Savings: ($81 / $150) * 100% = 54%

By caching the static 1,500-token header, we've cut the feature's monthly cost by more than half.

4. Your Turn: Calculate the Savings

Now it's time to apply what you've learned. Here is a scenario. Take your time to work through the four-step framework.

Scenario: AI-Powered Article Summarizer

An AI feature summarizes news articles for users.

  • Usage: 5,000 users, each making 10 requests per month.
  • Prompt Structure:
    • Static Part: A 2,500-token system prompt containing detailed summarization instructions and examples.
    • Dynamic Part: The article text, averaging 4,000 tokens.
  • Output: The summary averages 400 tokens.
  • Pricing (GPT-4o):
    • Input: $5.00 per million tokens
    • Output: $15.00 per million tokens
  • Optimization Strategy: You will apply prompt caching to the static 2,500-token system prompt, which provides a 90% discount on those input tokens.

Your Task:

  1. Calculate the total monthly baseline cost (without caching).
  2. Calculate the new total monthly cost (with caching).
  3. Calculate the total monthly savings in dollars and as a percentage.

Take a few minutes to perform the calculations. Once you're done, you can check your work against the solution below.

Click to see the solution

Step 1: Baseline Cost Calculation

  • Total Monthly Requests: 5,000 users * 10 requests/user = 50,000 requests
  • Total Input Tokens per Request: 2,500 (static) + 4,000 (dynamic) = 6,500 tokens
  • Total Monthly Input Tokens: 50,000 requests * 6,500 tokens = 325,000,000 tokens
  • Total Monthly Output Tokens: 50,000 requests * 400 tokens = 20,000,000 tokens
  • Baseline Input Cost: (325,000,000 / 1,000,000) * $5.00 = $1,625
  • Baseline Output Cost: (20,000,000 / 1,000,000) * $15.00 = $300
  • Total Monthly Baseline Cost: $1,625 + $300 = $1,925

Step 2 & 3: Recalculating with Caching

The output cost and the cost of dynamic input tokens remain unchanged. We only recalculate the cost of the static input tokens.

  • Cost of Unchanged Parts:

    • Dynamic Input Cost: (50,000 requests * 4,000 tokens/req / 1,000,000) * $5.00 = $1,000
    • Output Cost: $300 (same as above)

  • Cost of Cached Static Input:

    • Baseline Static Cost: (50,000 requests * 2,500 tokens/req / 1,000,000) * $5.00 = $625
    • Discounted Cost: $625 * (1 - 0.90) = $62.50

  • New Total Monthly Cost: $1,000 (Dynamic Input) + $300 (Output) + $62.50 (Cached Input) = $1,362.50

Step 4: Quantifying Savings

  • Absolute Savings: $1,925 - $1,362.50 = $562.50 per month
  • Percentage Savings: ($562.50 / $1,925) * 100% ≈ 29.2%

Conclusion

In this lesson, we moved from theory to practice, focusing on quantifying the economic impact of architectural decisions.

Key Takeaways:

  • Cost calculation is a core architectural skill. It translates technical strategy into measurable business value.
  • Batching savings are calculated by applying a discount to the portion of traffic that can run asynchronously.
  • Caching savings are calculated by breaking prompts into static and dynamic parts and applying a steep discount to the static (cached) tokens.
  • The simple four-step framework (Baseline, Identify, Recalculate, Quantify) is a reliable method for analyzing the financial benefits of these optimizations.

Next Steps

You can now calculate cost savings. The next logical question is: "How does this saving affect the feature's overall profitability?" In our next lesson, we will do just that, addressing the learning outcome: Recalculate the break-even timeline for an AI feature after applying cost optimization strategies. We will connect today's calculations directly to the financial models we built earlier in the course.