Idempotent Usage Reporting API for Billing

Hello! Welcome back to our course module on LLM Token Economics and Billing Systems.

In our last lesson, we designed the foundational API and data model for reporting token usage. We created a POST /v1/usage-events endpoint and a schema to capture essential details like user_id, model_id, and token counts. However, we ended on a critical point: in a distributed system, network failures are inevitable. If a client retries a request to our reporting API, how do we prevent the same usage from being recorded—and billed—twice?

This lesson directly addresses that challenge. Our learning outcome is to extend the usage reporting API with an idempotency mechanism to prevent duplicate billing.

Given your experience building high-throughput distributed systems at AWS, the concepts of "exactly-once" semantics and designing for failure will be very familiar. Today, we'll apply these principles to our billing system, transforming it from a simple data collection endpoint into a robust, fault-tolerant service.

1. The Problem: The Unreliability of Networks

In any distributed system, when a client makes a request to a server, there isn't just one failure mode. The request might never reach the server, the server might fail mid-process, or the server might complete the work but fail to send the response back to the client.

From the client's perspective, any timeout or network error leaves the state of the operation ambiguous. Was the usage event recorded or not? The simplest recovery strategy is to retry. But a blind retry is dangerous for a billing system, as it could lead to duplicate charges.

This is where idempotency becomes essential. An operation is idempotent if performing it multiple times has the same effect as performing it once. Our goal is to make our usage reporting API idempotent.

To get a clear picture of why this is so critical, please watch the first part of this video. It uses the familiar example of a payment service to illustrate the risks of non-idempotent APIs.

2. The Solution: The Idempotency-Key Pattern

The standard solution for making non-idempotent operations (like creating a new resource via POST) safe to retry is the idempotency key pattern.

Here's the contract:

1. Client-Side: Before making the first request for a specific operation, the client generates a unique identifier, the Idempotency-Key. This is typically a UUID.
2. Transmission: The client sends this key in an HTTP header (e.g., Idempotency-Key: <your-unique-key>) with the initial request and every subsequent retry of that same request.
3. Server-Side: The server is responsible for tracking these idempotency keys.
   • When a request arrives, the server checks if it has seen the Idempotency-Key before.
   • If the key is new, the server processes the request and stores the result before sending the response.
   • If the key has been seen before, the server does not re-process the request. Instead, it sends back the same response it generated for the original request.

This pattern shifts the responsibility of managing retry state from the client to the server, which is the only party with definitive knowledge of whether the operation was completed.

Stripe's API is a classic example of this pattern in action, and their documentation provides a great summary.

3. Server-Side Design and Implementation

Now, let's design the server-side components needed to support this pattern for our usage reporting API. The core of this is a persistent store to track the state of each idempotent request.

3.1. Architectural Choices

There are several ways to implement the persistence layer. The image below from ByteByteGo illustrates the most common patterns.

For a critical system like billing, durability is non-negotiable. In-memory tracking is not durable, and while a distributed cache like Redis can be used as a fast-path optimization, the ultimate source of truth must be a database. We will focus on a database-backed approach.

3.2. The Idempotency Data Model

We need a table or collection to store information about each idempotent request. Given your background with AWS, you could implement this with a DynamoDB table. Here's a schema that would work well:

Attribute Name	Type	Description
idempotency_key	String (Partition Key)	The unique key provided by the client (e.g., a UUID).
owner_id	String	A unique identifier for the calling entity (e.g., tenant_id or user_id) to scope the key. Prevents one user's key from conflicting with another's.
request_hash	String	A hash (e.g., SHA-256) of the request body. This prevents a client from reusing a key with a different payload.
status	String	The current state of the request (e.g., IN_PROGRESS, COMPLETED).
response_code	Number	The HTTP status code of the saved response (e.g., 202).
response_body	String	The serialized JSON body of the saved response.
ttl	Number	A Unix timestamp indicating when this record can be safely deleted.

The owner_id is a crucial security detail. An idempotency key should only be considered a duplicate if it's from the same authenticated entity.

The article "Idempotency Strategies for Modern Payment Systems" provides an excellent reference for a SQL-based version of this table and the server-side logic, which we will now adapt.

3.3. The Server Logic Flow

The following diagram illustrates the two main paths: handling a new request and handling a retried request.

Let's break down the logic that a middleware or decorator in your API service would execute:

1. Extract Key and Hash Payload:

   • Get the Idempotency-Key from the header and the owner_id from the authentication context.
   • Calculate the SHA-256 hash of the canonicalized request body.

2. Check for Existing Key:

   • Query your idempotency store for a record with the matching idempotency_key and owner_id.

3. Handle the Result:

   • Case A: Record Found (Retry)

     • If record.request_hash does not match the current request's hash, return a 409 Conflict. This is a misuse of the key.
     • If record.status is COMPLETED, return the stored response_code and response_body immediately. The work is already done.
     • If record.status is IN_PROGRESS, it means a concurrent request is being processed. Return 409 Conflict to indicate the client should wait and try again shortly.

   • Case B: No Record Found (New Request)

     • This is the most critical step. You must atomically create a new record and begin processing. We'll detail how to do this safely in the next section.

4. Concurrency, Atomicity, and Race Conditions

What happens if two identical requests arrive at the exact same time? If both check the database, find no key, and then both proceed, you've defeated the purpose of the idempotency check. This is a classic "check-then-act" race condition.

The solution is to use an atomic "insert-if-not-exists" operation provided by your database.

• In a SQL Database: You would rely on a UNIQUE constraint on the (idempotency_key, owner_id) columns. You attempt an INSERT, and if it fails due to a uniqueness violation, you know another request got there first. You then SELECT the existing record and treat it as a retry.

• In DynamoDB: You use a Condition Expression. Your PutItem call would include ConditionExpression: "attribute_not_exists(idempotency_key)". If the call succeeds, your process has acquired the "lock" and can proceed. If it fails with a ConditionalCheckFailedException, you know another process beat you to it.

This "insert-then-catch-duplicate" pattern is the most robust way to handle concurrency.

Furthermore, the business logic (writing the usage event) and updating the idempotency record must be atomic.

In an AWS environment, you would achieve this atomicity using DynamoDB Transactions. The TransactWriteItems operation allows you to group multiple Put, Update, or Delete actions into a single all-or-nothing operation. Your transaction would consist of two items:

1. PutItem for the new usage event in your main table.
2. UpdateItem to change the status of the record in your idempotency table from IN_PROGRESS to COMPLETED and store the response.

5. Implementation Sketch in Python

Let's modify the client code from our previous lesson to support idempotency.

# --- Client-Side Modification ---
import requests
import uuid




# This function would be part of your application's API client
def report_token_usage_with_retry(
    user_id: str,
    tenant_id: str,
    model_id: str,
    usage_data: dict
):
    """Constructs and POSTs a usage event with an idempotency key."""
    
    event_payload = {



        # event_id is now handled by the server or a stable hash
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "user_id": user_id,
        "tenant_id": tenant_id,
        "model_id": model_id,
        "input_tokens": usage_data.get("input_tokens", 0),
        "output_tokens": usage_data.get("output_tokens", 0),
    }




    # Generate the key ONCE for the logical operation
    idempotency_key = str(uuid.uuid4())
    headers = {
        "Authorization": "Bearer YOUR_INTERNAL_SERVICE_TOKEN",
        "Idempotency-Key": idempotency_key
    }




    # Simplified retry loop (in production, use exponential backoff and jitter)
    for attempt in range(3):
        try:
            response = requests.post(
                "https://internal-api.my-company.com/v1/usage-events",
                json=event_payload,
                headers=headers,
                timeout=5
            )
            



            # 2xx is success, 409 might mean in-progress, so don't retry
            if response.status_code < 500 and response.status_code != 409:
                print(f"Request successful with status {response.status_code} on attempt {attempt+1}")
                response.raise_for_status()
                return
        
        except requests.exceptions.RequestException as e:
            print(f"Attempt {attempt+1} failed: {e}. Retrying...")



            # time.sleep(...) # Add backoff here
    
    print("Failed to report usage after multiple retries.")

On the server side, your API framework (e.g., FastAPI) would have a dependency or middleware to encapsulate the logic discussed.

Conclusion

Today, we've elevated our simple usage reporting API into a resilient, production-grade service capable of safely handling network failures and retries. By implementing the idempotency-key pattern, we provide a crucial guarantee that prevents duplicate billing.

Key Takeaways:

• Idempotency is a Contract: It's a deliberate architectural choice where the server takes on the state management of retries, identified by a client-provided Idempotency-Key.
• Durability is Essential: For financial operations, the state of idempotent requests must be stored in a durable database (like PostgreSQL or DynamoDB), not just a cache.
• Concurrency is the Hardest Part: The "check-then-act" race condition is real. You must use atomic database operations (UNIQUE constraints or ConditionExpressions) to ensure only one of multiple concurrent requests gets processed.
• Atomicity Guarantees Consistency: Your business logic (writing the usage event) and the final update to the idempotency record must happen within a single transaction to prevent inconsistent states.

Next Lesson Preview

Our design now robustly handles usage reporting for single, non-streaming API calls. However, many modern LLM applications use streaming to improve perceived performance. In a streaming context, you don't get a single usage object at the end; tokens arrive one by one. How do you adapt our billing system for this? In the next lesson, we will adapt the API design to support incremental usage reporting for streaming responses.