Implementing Circuit Breakers for Resilient Microservices

Hello! In our last lesson, we focused on ensuring the reliability of individual LLM pods using a robust three-probe strategy. This is crucial for single-component resilience. Today, we'll broaden our scope to system-wide reliability, addressing how services interact with each other, especially when things go wrong.

This lesson is about implementing the circuit breaker pattern to handle failures in downstream model services. In modern AI applications, like the RAG systems you've worked with, a single user request can trigger a chain of calls to multiple backend services: a vector database, a search API, a content filter, and one or more LLMs. A failure or slowdown in any one of these can jeopardize the entire system. Your background in SRE and distributed systems makes you well-acquainted with the dangers of cascading failures, and the circuit breaker is a primary defense against them.

1. The Problem: Cascading Failures

When a downstream service becomes slow or unresponsive, an upstream service that depends on it can quickly run into trouble. If the upstream service keeps retrying or waiting for responses, its own resources—like connection pools or worker threads—get exhausted. This makes the upstream service itself unresponsive, causing a "cascading failure" that spreads through your infrastructure.

Let's start by watching a short video that clearly illustrates this problem and introduces the fundamental concept of the circuit breaker.

The key takeaway is that blindly continuing to call a failing service is not only futile but actively harmful. The circuit breaker pattern provides an intelligent, automated way to stop doing this.

2. The Three States of a Circuit Breaker

A circuit breaker acts as a stateful proxy that wraps calls to a remote service. It monitors for failures and uses a state machine to decide whether to allow calls to proceed. The pattern is named after the electrical circuit breakers in your home, which trip to prevent damage during a power surge.

The three states are:

1. CLOSED: This is the normal operating state. Requests are passed through to the downstream service, and the circuit breaker monitors the outcomes. If the number of failures within a given window exceeds a configured threshold, the breaker "trips" and transitions to the OPEN state.

2. OPEN: In this state, the circuit is "open," and electricity (requests) cannot flow. All calls to the downstream service are immediately rejected with an error without any attempt to contact the service. This is called failing fast. This protects the calling service from resource exhaustion and gives the failing downstream service time to recover. After a configured timeout period, the breaker transitions to HALF-OPEN.

3. HALF-OPEN: The breaker allows a limited number of "probe" requests to pass through to the downstream service. This is a test to see if the service has recovered.

   • If these probe requests succeed, the breaker concludes the service is healthy and transitions back to CLOSED.
   • If any probe request fails, the breaker trips again and returns to the OPEN state, restarting the recovery timer.

The following video segment provides a great walkthrough of this state machine logic.

It's also important to distinguish the circuit breaker from the retry pattern, which you've likely implemented many times.

• Retry Pattern: Best for transient faults (e.g., temporary network glitches). It assumes the fault is short-lived and will likely be gone on the next attempt.
• Circuit Breaker Pattern: Best for systemic faults (e.g., service is down, overloaded, or has a bug). Retrying here would worsen the problem by "dogpiling" on the failing service. The circuit breaker prevents this.

3. Implementing a Circuit Breaker in Python

Given your background in Python and preference for practical application, we will implement a circuit breaker within a FastAPI application. This scenario is highly relevant for an LLM Ops role, where you might build API gateways or aggregator services. We will use the pybreaker library, a popular and straightforward choice.

The example below simulates a RAG endpoint that depends on two external services: an LLM and a web search API. Each of these dependencies will be protected by its own circuit breaker.

Let's break down the key parts of the implementation from the article:

1. Initialize the Breaker:
You create a CircuitBreaker instance for each external dependency you want to protect.

import pybreaker




# Fail after 2 consecutive errors, then open the circuit for 10 seconds.
circuit_breaker_llm = pybreaker.CircuitBreaker(fail_max=2, reset_timeout=10)
circuit_breaker_web = pybreaker.CircuitBreaker(fail_max=2, reset_timeout=10)

2. Protect the External Call:
You use the breaker instance as a decorator around the function that makes the external call.

@circuit_breaker_llm
def fake_llm_call():
    """Simulates a call to an LLM API service."""
    return simulate_external_service("LLM")

3. Handle the Open-Circuit Error:
In your main application logic, you wrap the call in a try...except block to gracefully handle the case where the circuit is open.

try:
    llm_response = fake_llm_call()
    responses["llm"] = llm_response
except pybreaker.CircuitBreakerError:



    # The circuit is open, so fail fast.
    msg = "LLM API temporarily unavailable due to previous errors (circuit open)"
    raise HTTPException(status_code=503, detail=msg)

This pattern effectively isolates failures from the downstream LLM service, preventing them from taking down your main RAG endpoint.

4. Circuit Breakers in a Broader SRE Context

While the application-level implementation with pybreaker is powerful, in a large-scale system this logic might also live in the infrastructure layer, such as an API Gateway (e.g., Kong, Apigee) or a service mesh (e.g., Istio, Linkerd).

The code you just reviewed is what an application developer might write. As an LLM Ops Engineer, you would also be concerned with how this pattern is integrated into the platform itself. The following resource shows a more advanced Python implementation within a custom LLM load balancer, which aligns closely with your platform engineering background. You don't need to memorize the code, but appreciate how the pattern is structured.

This second example demonstrates how circuit breaker logic is a fundamental building block of any resilient routing or load-balancing layer in a distributed system.

5. Observability for Circuit Breakers

A silent circuit breaker is not very useful. To operate a reliable system, you need to know when your circuit breakers are tripping. This is a critical observability signal. The state of each circuit breaker should be exposed as a metric that can be scraped by a monitoring system like Prometheus.

The LLMLoadBalancer article shows an excellent example of this.

By exposing this metric, you can:

• Build Dashboards: Create a panel in Grafana that shows the state of all your circuit breakers over time. A "tripped" circuit is an immediate, clear indicator of a downstream dependency issue.
• Configure Alerts: Set up alerts in Alertmanager to fire whenever a circuit breaker's state transitions to 1 (Open). This allows your on-call team to immediately begin investigating the failing dependency.

Conclusion

Today, we've taken a significant step from single-component health to system-wide resilience. By implementing the circuit breaker pattern, you can protect your services from cascading failures caused by unreliable downstream dependencies—a common and critical challenge in building complex AI systems.

Key Takeaways:

• Cascading failures occur when a slow or failing service exhausts the resources of its callers, causing them to fail as well.
• The Circuit Breaker pattern prevents this by acting as a stateful proxy that trips when a failure threshold is met, "failing fast" instead of repeatedly calling the broken service.
• The three states are CLOSED (normal), OPEN (failing fast), and HALF-OPEN (probing for recovery).
• In Python, libraries like pybreaker make it easy to implement this pattern using decorators and exception handling.
• For an LLM Ops platform, it's crucial to expose circuit breaker states as metrics for monitoring, dashboarding, and alerting.

In our next lesson, we will answer the question: "What should our application do when the circuit is open?" This leads us directly to our next reliability pattern: configuring fallback strategies, such as routing to a smaller, more reliable model or returning a cached response when primary inference fails.