Robust Error Handling for LLM APIs

Welcome to your fourth lesson. In our previous sessions, we've focused on shaping the interaction with LLMs, first by guiding their reasoning and then by enforcing structured, machine-readable outputs. You now have the tools to control the what and the how of an LLM's response. Today, we address a fundamental reality of building any production system: things will go wrong.

LLM APIs are network services, subject to the same unreliability as any other remote dependency. They experience rate limits, temporary outages, performance degradation, and network hiccups. A robust AI application cannot simply hope for the best; it must be engineered for resilience. This lesson is about building that resilience. We will cover the essential error handling strategies you need to implement, focusing on handling transient issues with exponential backoff and preparing for major provider problems with fallback mechanisms.

This is a core software engineering discipline applied to the AI domain, moving your application from a prototype to a production-ready service.

The Unreliable World of API Calls

When your application makes a call to an LLM provider, it's crossing a network boundary to a complex, multi-tenant system. This introduces several potential points of failure:

• Rate Limiting: To ensure fair usage and prevent abuse, all providers enforce rate limits. If you send too many requests in a short period, the API will respond with a 429 Too Many Requests error.
• Transient Server Errors: The provider's servers might be temporarily overloaded or experiencing an internal issue, resulting in 5xx errors (e.g., 500 Internal Server Error, 503 Service Unavailable). These are often short-lived.
• Timeouts and Network Issues: The request or response might get lost or delayed somewhere on the internet, leading to timeouts.
• Provider-wide Outages: Occasionally, a provider may suffer a major outage, making their services completely unavailable for a period.

A naive implementation that fails on the first error will create a brittle and frustrating user experience. Our goal is to handle these common issues gracefully and automatically.

Strategy 1: Retries with Exponential Backoff

For transient errors like rate limits and temporary server issues, the best strategy is often to simply wait a moment and try again. However, retrying immediately and repeatedly is counterproductive. If you're hitting a rate limit, hammering the server will only make the problem worse for you and everyone else—a phenomenon known as a "thundering herd."

The correct approach is to retry with exponential backoff. This means you increase the waiting time between each successive retry attempt exponentially.

The excellent "Retry Pattern" video from the ArjanCodes channel provides a clear, step-by-step guide to implementing this logic in Python.

As the video concludes, you rarely need to build this logic from scratch in a production environment. You should use a well-tested library. For Python, the tenacity library is the de-facto standard. In fact, modern LLM client libraries like openai-python (v1.0 and later) have tenacity built in and are configured to automatically retry on transient errors like 429, 500, and 503 by default. Your main job is to be aware that this is happening and know how to configure it if the defaults aren't suitable.

Strategy 2: Fallbacks and Circuit Breakers

What happens when retrying isn't enough? A provider might be down for an extended period, or a specific model might be consistently failing. In these cases, repeatedly retrying the same failed request is pointless. You need a fallback strategy.

A provider fallback means that if the primary choice fails, the system automatically routes the request to a secondary, tertiary, or even quaternary provider. For example, if a request to OpenAI's gpt-4o fails permanently, your system could automatically try again with Anthropic's claude-3-opus, and if that fails, Google's gemini-1.5-pro.

This requires designing your application to be provider-agnostic. Given your background in Go, you can appreciate the power of defining your own interfaces.

A more advanced and robust pattern for managing fallbacks is the Circuit Breaker. This pattern, common in microservices architecture, prevents an application from repeatedly trying to execute an operation that is likely to fail.

A circuit breaker wraps a protected function call in an object that monitors for failures.

• Closed State: Initially, the circuit is "closed," and requests pass through to the provider.
• Open State: If the number of failures exceeds a threshold, the breaker "opens." For a set period, all subsequent calls fail immediately without even attempting to contact the provider.
• Half-Open State: After the timeout, the breaker moves to a "half-open" state, allowing a single test request through. If it succeeds, the breaker resets to "closed." If it fails, the breaker returns to the "open" state, starting a new timeout.

This prevents a failing service from causing cascading failures throughout your system.

A Layered Approach to Resilience

Error handling isn't a single technique; it's a layered strategy. You start with the simplest, highest-impact layer and add complexity as needed.

As you can see, implementing retries with exponential backoff is the foundational first step, estimated to catch nearly half of all failures in a typical application. Provider fallbacks, often managed by a circuit breaker, add another critical layer of defense.

Hands-on Exercise: Building a Resilient Bug Report Parser

Let's apply these concepts by making the bug report parser from our previous lesson more resilient. You will take the existing Python script and add both retry and fallback logic.

Your Task:
Modify the extract_bug_report function to handle API errors gracefully.

1. Install tenacity: If it's not already in your environment, install the tenacity library: pip install tenacity.
2. Add Retries: Use the @tenacity.retry decorator on your function to handle transient errors.
   • Configure it to retry on openai.RateLimitError and openai.APIError.
   • Use an exponential backoff strategy (e.g., wait=tenacity.wait_exponential(multiplier=1, min=2, max=60)).
   • Set a stop condition (e.g., stop=tenacity.stop_after_attempt(3)).
   • Add a before_sleep function to log the retry attempt, so you can see it in action.
3. Implement Fallback: If the primary model (gpt-4o) fails after all retries, the function should fall back to a secondary, more reliable (and cheaper) model like gpt-3.5-turbo. You can achieve this by wrapping your decorated function call in a try...except block. If tenacity.RetryError is raised, it means all retries have failed, and you can then make a second call with the fallback model.

Starter Code:

import os
import logging
from openai import OpenAI, RateLimitError, APIError
from pydantic import BaseModel, Field
from typing import Literal, List
import tenacity




# --- Basic Setup ---
# It's good practice to use logging instead of print for this
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')




# Ensure you have your OPENAI_API_KEY set as an environment variable
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))




# --- Pydantic Schema (from previous lesson) ---
class BugReport(BaseModel):
    reasoning: str = Field(description="First, think step-by-step about the content of the bug report to understand the core issue, its severity, and its context.")
    title: str = Field(description="A concise, one-line title for the bug ticket.")
    priority: Literal["Low", "Medium", "High"] = Field(description="The estimated priority of the bug.")
    is_ui_bug: bool = Field(description="True if the bug is related to the user interface.")
    reproduction_steps: List[str] = Field(description="A list of clear, numbered steps to reproduce the bug.")




# --- Helper function for logging retries ---
def log_retry_attempt(retry_state):
    """Logs the details of a retry attempt."""
    logging.warning(f"Retrying API call... Attempt: {retry_state.attempt_number}, Wait: {retry_state.next_action.sleep}s")




# --- 2. Implement the extraction function with error handling ---




# Use the tenacity decorator to add retry logic
@tenacity.retry(
    wait=tenacity.wait_exponential(multiplier=1, min=2, max=30),
    stop=tenacity.stop_after_attempt(3),
    retry=tenacity.retry_if_exception_type((RateLimitError, APIError)),
    before_sleep=log_retry_attempt
)
def _call_openai_with_retries(model_name: str, messages: list) -> BugReport:
    """A private function that makes the actual API call, decorated for retries."""
    logging.info(f"Attempting API call with model: {model_name}")
    response = client.beta.chat.completions.parse(
        model=model_name,
        messages=messages,
        response_format=BugReport,
    )
    return response.choices[0].message.parsed


def extract_bug_report_resilient(text: str) -> BugReport:
    """
    Uses OpenAI's structured output to parse a bug report with retry and fallback logic.
    """
    messages = [
        {"role": "system", "content": "You are an expert bug report analyst. Extract structured info."},
        {"role": "user", "content": text},
    ]
    
    primary_model = "gpt-4o"
    fallback_model = "gpt-3.5-turbo"

    try:
        logging.info(f"--- Attempting with primary model: {primary_model} ---")
        return _call_openai_with_retries(model_name=primary_model, messages=messages)
    except tenacity.RetryError as e:
        logging.error(f"Primary model '{primary_model}' failed after all retries. Error: {e}")
        logging.info(f"--- Falling back to secondary model: {fallback_model} ---")



        # In a real scenario, the fallback might not need retries, or have its own retry policy.
        # For simplicity, we call it directly here.
        try:
             response = client.beta.chat.completions.parse(
                model=fallback_model,
                messages=messages,
                response_format=BugReport,
            )
             return response.choices[0].message.parsed
        except Exception as fallback_e:
             logging.critical(f"Fallback model '{fallback_model}' also failed. Error: {fallback_e}")



             # Re-raise the exception or return a default error object
             raise




# --- 3. Test your function ---
raw_bug_report = """
The site is broken!! I'm trying to update my profile on the mobile app and the save button is greyed out. 
This is super urgent, I can't do anything. I tried restarting the app and my phone. Nothing works.
It's on the profile settings screen.
"""




# To test this, you'd ideally mock the OpenAI API to raise errors.
# Since that's complex, you can review the logic and imagine the flow.
# For a simple test, just run it and see the successful path.
structured_report = extract_bug_report_resilient(raw_bug_report)

if structured_report:
    print("\n--- Structured Bug Report (Success) ---")
    print(structured_report.model_dump_json(indent=2))

This solution separates the retry-able call into its own decorated function, which is a clean pattern. The main function then orchestrates the primary-fallback logic.

Conclusion

You have now added a critical layer of production readiness to your AI engineering toolkit. Building with LLM APIs means building distributed systems, and assuming failure is the first step toward robustness.

Key takeaways from this lesson:

• Plan for Failure: Networked services like LLM APIs will fail. It's not a matter of if, but when.
• Retry with Backoff: For transient errors (rate limits, temporary server issues), implement automatic retries with exponential backoff to give the service time to recover. Use standard libraries like tenacity.
• Implement Fallbacks: For catastrophic or persistent failures, retries are not enough. A fallback mechanism that switches to a secondary provider or model is essential for high availability.
• Use Circuit Breakers: To prevent a failing service from overwhelming your own application, use the circuit breaker pattern to "fail fast" and manage fallbacks intelligently.

In our next lesson, we will explore a topic that is fundamental to implementing an effective fallback strategy: how to compare LLM inference providers based on cost, latency, and capability trade-offs for production use cases. After all, you can't choose a good backup if you don't know what makes it good.