Handling Inference Failures with Fallback Strategies

Recommended resources

The freely available resources this lesson will direct you to.

Welcome back! In our last lesson, we established a critical defense mechanism, the circuit breaker, to protect our system from cascading failures when a downstream model service becomes unresponsive. This raises an immediate and practical question: what should our application do when the circuit is open? Just returning an error is safe, but often we can do better to maintain a good user experience.

This lesson directly answers that question by focusing on how to configure fallback strategies, such as routing to a smaller model or returning a cached response, when primary inference fails. As an SRE, you know that designing for failure is paramount. These fallback patterns are the next logical step in building a truly resilient LLM service, moving from simply failing safely to degrading gracefully.

We will explore two primary fallback strategies:

Model Fallback: Automatically routing a failed request to an alternative model.
Cache Fallback: Serving a previously stored response for a similar query.

1. Fallback Strategy 1: Routing to an Alternate Model

The most direct fallback strategy is to have a secondary model ready to take over when the primary one fails. For example, if your application relies on a large, powerful model like GPT-4, but it becomes unavailable or starts returning errors, you can automatically reroute the request to a smaller, faster, and often more reliable model like gpt-3.5-turbo or a self-hosted open-source model.

This ensures your application remains functional, albeit potentially with a slight degradation in response quality, which is usually far preferable to a complete outage. The concept is cleanly illustrated by an AI gateway architecture.

This diagram contrasts an architecture without fallbacks, where an application fails if the AI provider is down, with an architecture using an AI Gateway. The gateway intelligently routes requests to a fallback provider if the primary target fails, ensuring service continuity.

Your experience with load balancers and service routing in distributed systems provides the perfect context for this pattern. Here, the "routing" is not just for load but for reliability. Let's look at a practical implementation using LiteLLM, a tool designed to be a universal interface for over 100 LLM APIs.

Fallbacks - LiteLLM

The LiteLLM documentation provides a clear, code-centric guide to configuring fallbacks. It acts as an intelligent proxy, making this pattern easy to implement.

Please read through the "Fallbacks" documentation, paying attention to the following: Basic Configuration: Review the initial examples showing how to define a fallbacks list in the configuration. Notice the simple mapping from a primary model to a list of potential fallback models. Testing Fallbacks: Look at the section "Test Fallbacks". The mock_testing_fallbacks=true parameter is a useful feature for verifying your configuration without waiting for a real outage. Types of Fallbacks: Read the "Explanation" section. Note that LiteLLM supports different fallbacks for specific errors, such as content_policy_fallbacks and context_window_fallbacks. This allows for more granular control, which is a key consideration for a production platform. Client-Side Configuration: Skim the "Client Side Fallbacks" section. This shows how fallbacks can be specified per-request, offering dynamic control from the client application (e.g., via cURL or the OpenAI Python client).

The core idea is simple but powerful: define an ordered list of models to try if the initial one fails. LiteLLM handles the complexity of switching providers and retrying the request.

The same pattern can be found in other frameworks like LangChain. This short video demonstrates the concept using LangChain's middleware.

Model Fallback Middleware (Python)

This video, "Model Fallback Middleware (Python)" from the LangChain channel, shows a live demonstration of the model fallback pattern.

Watch the video to see the pattern in action. Problem Statement: The introduction explains the motivation for fallbacks. Configuration: See how a primary model and a list of fallbacks are specified. Live Demo: The trace view clearly shows the system trying a failing model, then another, before succeeding with the final model in the chain.

2. Fallback Strategy 2: Returning a Cached Response

What if the user is asking a very common question? Sending it to an LLM every time is expensive and introduces latency. If the primary LLM fails, we might not even need a fallback model. Instead, we can serve a stored answer from a cache.

This is where semantic caching becomes a powerful fallback mechanism. Unlike traditional caching which requires an exact string match, semantic caching uses vector embeddings to find and return answers to semantically similar queries.

For example, these queries would all be cache misses in a traditional cache, but could be a cache hit in a semantic cache:

"How do I reset my password?"
"I forgot my password, what do I do?"
"Need help with password change"

The following video from Redis provides an excellent walkthrough of implementing both standard and semantic caching using LangChain and Redis.

A Semantic Cache using LangChain

The video "A Semantic Cache using LangChain" demonstrates how to build and use caching layers for LLM applications.

Focus on these two key demonstrations: Standard Cache: First, watch the segment on standard caching. Observe the dramatic reduction in latency for an identical, repeated query. The key benefit mentioned is avoiding the cost and latency of a redundant LLM call. Semantic Cache: Next, watch the more advanced segment on semantic caching. Pay close attention to how it handles queries that are worded differently but have the same meaning. The code shows the use of an embedding model and a distance_threshold to control the "fuzziness" of the match.

The video demonstrates the "how" from a developer's perspective. For your LLM Ops role, understanding the configuration levers and production considerations is equally important.

Reduce Costs by 40-80% and Speed up by 250x Semantic Caching ...

This blog post from Percona, "Semantic Caching for LLM Apps," delves into the practicalities of building and deploying a semantic cache, offering an operational perspective.

Read through the article, focusing on these aspects relevant to your SRE background: How It Works: The initial section provides a concise explanation of the process: convert query to vector, search for similar vectors, and return the cached response. Implementation: Review the Python code in the "Building a Semantic Cache" section. You don't need to memorize it, but recognize the core components: a vector database (Redis/Valkey), an embedding model, and the logic to search the cache before calling the LLM. Configuration Levers: The "Configuration: important levers" section is crucial. Pay close attention to the Similarity Threshold and Time-to-Live (TTL). These are the main knobs you would turn to tune the cache's performance and behavior. Production Checklist: The final checklist includes items like monitoring hit rates, error handling (i.e., fallback to the LLM if the cache fails), and cache warming. This aligns perfectly with a production-readiness mindset.

3. Combining Strategies with Semantic Routing

These two fallback strategies are not mutually exclusive. A sophisticated system can combine them into a resilient, multi-stage process orchestrated by a semantic router.

This workflow illustrates how a semantic router works. A user query is embedded and compared against predefined routes in a vector index to determine the most appropriate action, which could be directing to a specific model or a cache.

A typical flow for a highly resilient service might look like this:

Request Ingested: A user submits a prompt.
Semantic Cache Check: The router first checks a semantic cache. If a sufficiently similar query with a valid (non-expired) response is found, it's returned immediately. (Cache Fallback)
Primary Model Call: On a cache miss, the router forwards the request to the primary, high-performance model (e.g., GPT-4o). A circuit breaker wraps this call.
Secondary Model Call: If the primary model call fails and the circuit breaker trips, the router automatically retries the request with a secondary, more reliable model. (Model Fallback)
Final Failure: If all else fails, the system returns a definitive error message to the user.

This layered approach, which you'll recognize from designing multi-region failover strategies, maximizes availability and manages costs effectively.

Conclusion

Today we've moved beyond simply detecting failures to actively mitigating them with intelligent fallback strategies. By planning for model outages and leveraging caching, you can significantly improve the reliability and perceived performance of your LLM services.

Key Takeaways:

Model Fallback involves routing a request to a secondary model (often smaller or cheaper) when the primary one fails. Tools like LiteLLM can manage this routing automatically.
Cache Fallback, particularly Semantic Caching, provides an effective way to handle failures for common or semantically similar queries by returning a stored response, drastically reducing latency and cost.
The effectiveness of a semantic cache is controlled by key operational parameters like the similarity threshold and TTL.
These strategies can be combined into a multi-layered resiliency approach, often orchestrated by a semantic router.

In our next lesson, we will explore a more proactive form of this pattern: graceful degradation. We'll see how to route traffic to a smaller model not just on outright failure, but when system performance metrics like P99 latency start to exceed their SLOs, allowing the system to shed load before it fails completely.