API Error Handling with Retries and Rate Limits

Not All Errors Are Created Equal

When an API call fails, the first step is to understand why it failed. HTTP status codes are our primary tool for this diagnosis. A 404 Not Found error, for example, means the resource you asked for doesn't exist. Retrying this request is pointless; the resource won't magically appear. This is a permanent error that requires a code or configuration change to fix.

On the other hand, a 503 Service Unavailable error indicates a temporary problem on the server's end. The service might be restarting or overloaded. This is a transient error, and retrying the request after a short delay is a sensible strategy.

The video "How to Work with API's as a Data Engineer!" provides a good overview of the common status code ranges.

The key takeaway is that we should retry on transient issues: server-side problems (5xx status codes), specific client-side codes that ask us to wait (429 Too Many Requests), and network-level problems (like ConnectionError or Timeout).

Implementing Automatic Retries with Exponential Backoff

Now that we know what to retry, let's discuss how. A naive approach would be to immediately retry a failed request in a loop. This is often a bad idea. If a service is overloaded, hammering it with immediate retries will only make the problem worse for everyone.

A much better strategy is exponential backoff. This means we increase the waiting period between each successive retry. For example, wait 1 second after the first failure, 2 seconds after the second, 4 after the third, and so on. This gives the struggling service time to recover.

To prevent a "thundering herd" problem—where many clients all retry at the exact same moments after an outage—we can add jitter. Jitter introduces a small, random amount of time to each backoff delay, spreading out the retry attempts.

The video "Retry Pattern" by ArjanCodes provides an excellent from-the-ground-up explanation of building a retry mechanism, including exponential backoff.

Implementing Retries in Python

While it's useful to understand how to build a retry function manually, for our project, we will use a robust, battle-tested library. In Python, there are two excellent ways to approach this with the requests library.

1. The Built-in HTTPAdapter Approach

The requests library can be configured to handle retries automatically at the session level. This is done by creating a Retry object (from the underlying urllib3 library), configuring its behavior, and "mounting" it to a requests.Session via an HTTPAdapter.

This approach is great for applying a consistent, simple retry policy to all requests made with a session.

2. The tenacity Library Approach

For more complex or granular control, the tenacity library is the gold standard in the Python ecosystem. It uses decorators to wrap functions with retry logic, offering immense flexibility. You can specify complex wait strategies (like exponential backoff with jitter), define retry conditions based on exceptions or return values, and add logging hooks to observe retry behavior.

Given its flexibility and explicit nature, we will use tenacity to make our API client more robust.

Handling API Rate Limits

A common reason for an API to temporarily reject a request is rate limiting. When you make too many requests in a given time period, a server will often respond with a 429 Too Many Requests status code.

A well-behaved API will also include a Retry-After header in the response, which tells you exactly how many seconds (or until what specific date-time) you should wait before trying again. Respecting this header is crucial; ignoring it can lead to your IP address being blocked.

The urllib3.util.Retry object we saw earlier has a respect_retry_after_header=True parameter that handles this automatically. When using tenacity, we would need to inspect the response and handle it more explicitly, though this gives us more control.

A Critical Consideration: Idempotency

Before we wrap up, we must address a crucial concept: idempotency. An operation is idempotent if making it multiple times has the same effect as making it once.

• GET, HEAD, PUT, and DELETE requests are generally idempotent. Requesting GET /users/123 ten times is the same as requesting it once. Deleting a user twice has the same result as deleting them once (they're gone).
• POST requests are generally not idempotent. If you POST to /users to create a new user, retrying that request after a timeout could result in creating two identical users.

This has major implications for retries. It is only safe to automatically retry idempotent requests. For non-idempotent requests like POST, you should only retry on connection errors where you are certain the request never reached the server. If you get a 5xx error, the server may have processed the request before crashing, so a blind retry is unsafe.

Modern APIs solve this problem using an Idempotency-Key header. You generate a unique ID for each logical operation and send it with your POST request. If the server sees the same key again (e.g., on a retry), it knows not to re-process the request and can simply return the original result.

Upgrading Our API Client

Let's refactor the pagination function from our previous lesson to incorporate what we've learned. We'll use tenacity to add robust retries to each individual API call within our pagination loop.

import os
import requests
import logging
from tenacity import retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception




# Configure basic logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

API_TOKEN = os.getenv("MY_API_TOKEN", "your-secret-token")
API_BASE_URL = "https://api.example.com/v1"

def is_retryable_error(exception: BaseException) -> bool:
    """Return True if the exception is a retryable network error or a 5xx/429 status code."""
    if isinstance(exception, (requests.exceptions.ConnectionError, requests.exceptions.Timeout)):
        logging.warning(f"Retryable network error occurred: {exception}")
        return True
    if isinstance(exception, requests.exceptions.HTTPError):
        is_server_error = 500 <= exception.response.status_code < 600
        is_rate_limit = exception.response.status_code == 429
        if is_server_error or is_rate_limit:
            logging.warning(f"Retryable HTTP error occurred: {exception.response.status_code}")
            return True
    return False




# The @retry decorator wraps our function with retry logic.
@retry(
    wait=wait_exponential_jitter(initial=2, max=60, jitter=1), # wait 2s, then 4s, etc. up to 60s, with jitter
    stop=stop_after_attempt(5), # Give up after 5 attempts
    retry=retry_if_exception(is_retryable_error), # Use our custom function to decide if we should retry
    reraise=True # If all retries fail, re-raise the last exception
)
def fetch_page(session: requests.Session, url: str) -> dict:
    """
    Fetches a single page of data with automatic retries for transient errors.
    """
    logging.info(f"Fetching data from {url}")
    response = session.get(url, timeout=15) # A reasonable timeout
    response.raise_for_status() # Raises HTTPError for 4xx/5xx responses
    return response.json()

def fetch_all_data(endpoint: str) -> list[dict]:
    """
    Fetches all pages of data from a paginated API endpoint,
    with robust error handling for each page request.

    Args:
        endpoint: The API endpoint path (e.g., '/conversations').

    Returns:
        A list containing all records from all pages.
    """
    all_records = []
    
    with requests.Session() as session:
        session.headers.update({"Authorization": f"Bearer {API_TOKEN}"})
        
        next_page_url = f"{API_BASE_URL}{endpoint}"
        
        page_num = 1
        while next_page_url:
            try:



                # The retry logic is applied here by the decorator on fetch_page
                logging.info(f"--- Attempting to fetch page {page_num} ---")
                data = fetch_page(session, next_page_url)
                
                records = data.get("results", [])
                all_records.extend(records)
                
                next_page_url = data.get("next") 
                
                logging.info(f"Successfully fetched page {page_num} with {len(records)} records. Total: {len(all_records)}.")
                page_num += 1

            except requests.RequestException as e:



                # This block catches non-retryable errors (like 404 Not Found)
                # or the final exception after all retries have failed.
                logging.error(f"Failed to fetch data for page {page_num} at {e.request.url}. Error: {e}")



                # Depending on requirements, you might want to stop entirely or just skip the failing page.
                # For this pipeline, we'll stop to ensure data completeness.
                break
                
    return all_records




# Example usage:
# conversations = fetch_all_data("/conversations?status=active")
# print(f"Successfully fetched a total of {len(conversations)} conversations.")

Notice how we've separated the logic. The fetch_page function is responsible for a single atomic request and is decorated with our retry logic. The fetch_all_data function orchestrates the pagination, calling our new, robust function for each page. This is a clean and powerful pattern.

Conclusion

In this lesson, we've significantly hardened our API client. You learned how to differentiate between transient and permanent errors and to build a resilient data ingestion function that can automatically recover from temporary network or server issues.

Here are the key takeaways:

• Retry on transient failures: Only retry on server errors (5xx), specific client errors (429), and network connection issues.
• Use exponential backoff with jitter: This strategy prevents overwhelming a struggling service and avoids synchronized retry storms.
• Leverage libraries: Use requests.HTTPAdapter for simple, session-wide policies, and tenacity for fine-grained control with decorators.
• Respect Retry-After: When an API tells you how long to wait, listen to it.
• Understand idempotency: Be very careful when retrying non-idempotent operations like POST to avoid creating duplicate data. Use an Idempotency-Key where possible.

Now that we can reliably fetch raw data from our sources, our next challenge is to ensure its quality before loading it. In the next lesson, we will learn how to validate this incoming JSON data against a predefined schema using Pydantic, preventing malformed data from corrupting our pipeline.