Cost Estimation for LLM Request Patterns

Hello! Welcome to the second lesson in our module on LLM Token Economics and Billing Systems.

In our previous lesson, we established the fundamental principle behind LLM API pricing: the cost difference between input and output tokens is a direct reflection of the underlying computational workload. We learned that processing input (the "prefill" stage) is a highly parallel and efficient task, while generating output (the "decode" stage) is a sequential, memory-intensive process that uses expensive GPU resources less efficiently.

Today, we'll put that knowledge into practice. Our goal is to calculate the cost for various request patterns, such as summarization vs. few-shot prompting. By the end of this lesson, you'll be able to take a specific use case, estimate its token footprint, and calculate the associated API cost. This is a crucial skill for any engineer building systems on top of LLMs, as managing cost is as critical as managing latency or throughput.

1. The Core Cost Formula

At its heart, calculating the cost of a single, non-streaming API call is straightforward. It's the sum of the cost of the tokens you send and the cost of the tokens you receive.

The article "Understanding Anthropic Pricing for Claude Models" provides the essential formula.

As the article states, the formula is:

$$\text{Total Cost}=(\text{Input Tokens}\times \text{Input Price per Token})+(\text{Output Tokens}\times \text{Output Price per Token})$$

Typically, prices are given per 1 million tokens (1M). So, a more practical version of the formula is:

$$\text{Total Cost}=\left(\frac{\text{Input Tokens}}{1,000,000}\times \text{Price per 1M Input Tokens}\right)+\left(\frac{\text{Output Tokens}}{1,000,000}\times \text{Price per 1M Output Tokens}\right)$$

Now, let's apply this formula to different request patterns.

2. Pattern 1: Summarization (Large Input, Small Output)

The summarization example from the article you just read is a perfect illustration of a common pattern: you provide a large amount of context and ask for a concise output.

Let's re-examine their calculation for summarizing a 7,500-word report (approx. 10,000 input tokens) into a 750-word summary (approx. 1,000 output tokens) using Claude 3 Sonnet.

• Model: Claude 3 Sonnet
• Input Price: $3.00 per 1M tokens
• Output Price: $15.00 per 1M tokens

Calculation:

• Input Cost: $\frac{10,000}{1,000,000}\times \$3.00=0.01\times \$3.00=\$0.03$
• Output Cost: $\frac{1,000}{1,000,000}\times \$15.00=0.001\times \$15.00=\$0.015$
• Total Cost: $\$0.03+\$0.015=\$0.045$

The key characteristic of this pattern is that the input cost dominates. Even with the 5x price multiplier on output, the sheer volume of input tokens makes it the primary cost driver.

3. Pattern 2: Few-Shot Prompting (Context-Heavy Input)

Few-shot prompting is a powerful technique where you include examples of the desired input-output behavior directly in the prompt. This helps guide the model to produce more accurate or correctly formatted responses. However, these examples can significantly increase your input token count.

Let's imagine a sentiment classification task. We want the model to label a user review as POSITIVE, NEGATIVE, or NEUTRAL.

A. Zero-Shot Prompt: No examples provided.

• Prompt: "Classify the sentiment of the following review as POSITIVE, NEGATIVE, or NEUTRAL. Review: 'The service was okay, but the food was bland.'"
• Input Tokens (estimate): ~30 tokens
• Output Tokens (estimate): ~1 token (NEUTRAL)

B. Few-Shot (3-shot) Prompt: Three examples included.

• Prompt: "Classify the sentiment... \n\nReview: 'I loved it! Best experience ever.'\nSentiment: POSITIVE\n\nReview: 'A complete disaster. I will never return.'\nSentiment: NEGATIVE\n\nReview: 'It met my expectations.'\nSentiment: NEUTRAL\n\nReview: 'The service was okay, but the food was bland.'"
• Input Tokens (estimate): ~100 tokens (30 for instruction/query + 70 for examples)
• Output Tokens (estimate): ~1 token (NEUTRAL)

Let's calculate the cost difference using Claude 3 Sonnet:

• Zero-Shot Cost:

  • Input: $\frac{30}{1,000,000}\times \$3.00=\$0.00009$
  • Output: $\frac{1}{1,000,000}\times \$15.00=\$0.000015$
  • Total: ~$0.0001

• Few-Shot (3-shot) Cost:

  • Input: $\frac{100}{1,000,000}\times \$3.00=\$0.0003$
  • Output: $\frac{1}{1,000,000}\times \$15.00=\$0.000015$
  • Total: ~$0.0003

In this case, adding three examples tripled the cost of the API call. While the absolute cost is tiny for a single call, this multiplier is critical at scale. Your work on Rudi, handling 1000+ TPS, highlights how a seemingly small per-request cost increase can lead to substantial budget overruns across millions of requests.

This creates an important trade-off: Accuracy vs. Cost. The article "The Frugal Approach to Anthropic Claude API Costs" notes that you can often achieve most of the accuracy gains with just a few examples, and adding more yields diminishing returns at a high token cost. The engineering discipline is to find the minimum number of examples needed to meet your performance target.

4. Pattern 3: Conversational Chat (Compounding Input)

Perhaps the most common and financially risky pattern is the conversational chatbot. Because LLM APIs are stateless, you must resend the conversation history with every new user message to maintain context. This leads to a compounding increase in input tokens.

This video gives a crystal-clear demonstration of this effect.

Let's formalize the example from the video:

• User Message 1: "Help me understand this function..." (Prompt 1: 1,000 tokens)
• Assistant Response 1: [Explanation] (Output 1: 2,000 tokens)

Cost of Turn 1 (using Claude 3 Sonnet prices):

• Input Cost: $\frac{1000}{1M}\times \$3=\$0.003$
• Output Cost: $\frac{2000}{1M}\times \$15=\$0.03$
• Total for Turn 1: $0.033

Now, for the second turn:

• User Message 2: "Can you refactor it?" (Prompt 2: 1,000 tokens)

To process this, the API call must include the history.

• New Input Payload: Prompt 1 + Output 1 + Prompt 2
• Total Input Tokens: 1,000 + 2,000 + 1,000 = 4,000 tokens
• Assistant Response 2: [Refactored Code] (Output 2: 2,000 tokens)

Cost of Turn 2:

• Input Cost: $\frac{4000}{1M}\times \$3=\$0.012$
• Output Cost: $\frac{2000}{1M}\times \$15=\$0.03$
• Total for Turn 2: $0.042

Notice that the second turn, which involved a similar amount of new text and generation, was significantly more expensive than the first because the input payload grew to include the entire history. This cost grows quadratically with the number of turns, quickly becoming unsustainable. This is why strategies like conversation summarization or windowing are critical for production chatbots.

Conclusion

In this lesson, we moved from the theory of token pricing to the practice of cost calculation. By analyzing different request patterns, we've seen how the structure of a prompt can have a dramatic impact on its final cost.

Key Takeaways:

• The total cost of an LLM API call is a sum of its input and output token costs, calculated using the formula: $\text{Total Cost}=(\frac{\text{Input Tokens}}{1M}\times \text{Input Price})+(\frac{\text{Output Tokens}}{1M}\times \text{Output Price})$.
• Summarization patterns are typically dominated by input costs due to the large volume of context provided.
• Few-shot prompting increases input token counts with each example, creating a direct trade-off between prompt-tuning for accuracy and managing costs.
• Conversational chat is a major cost driver due to the stateless nature of LLMs, which requires resending the growing conversation history and leads to compounding input token costs.
• Choosing the right model for the task (e.g., Haiku for simple tasks, Opus for complex reasoning) is one of the most significant levers for cost control.

Next Lesson Preview

Now that you can calculate the cost of a given API call, the next logical step in building a production system is to track and report this usage for billing and analysis. In our next lesson, we will design the data model and API for reporting token usage from a single, non-streaming LLM response. This will move us further into the practical engineering of building robust, cost-aware services around LLMs.