Designing Resilient Systems: Handling Unavailable Downstream Services

1. What is Graceful Degradation?

In a distributed system, it's not a question of if a service will fail, but when. Graceful degradation is the practice of designing a system to handle these failures in a way that minimizes the impact on the end-user. Instead of showing a total failure (like a "500 Internal Server Error" page), the application continues to operate in a limited but still useful capacity.

Think of it like a car's "limp mode": if the engine detects a serious problem, it reduces power but still allows you to drive slowly to a mechanic. The core function (driving) is maintained, but non-essential performance is sacrificed.

To explore this concept with some real-world examples, watch the following segment from a talk on resilience patterns.

Graceful degradation is a key aspect of fault tolerance, which is about designing a system to handle internal failures. This is distinct from, but complementary to, high availability, which typically involves redundancy to ensure the system is always accessible.

2. When to Degrade vs. When to Fail Fast

Not all services are created equal. The first step in designing for graceful degradation is to distinguish between critical and non-critical dependencies.

• Critical Dependencies: These are services essential for a core business transaction. If they fail, the entire operation must be stopped to prevent data corruption or an invalid state. For these, the correct strategy is to fail fast.
• Non-Critical Dependencies: These services enhance the user experience but are not essential for the core transaction. If they fail, the system can continue operating, albeit with reduced functionality. This is where graceful degradation applies.

How do you decide? A key factor is data integrity.

Let's consider an e-commerce application:

• Placing an order: This involves the Order Service, Payment Service, and Inventory Service. A failure in any of these is critical. You cannot confirm an order if you can't process the payment or verify stock. Here, you must fail fast and inform the user that the order could not be placed.
• Viewing the user dashboard: This might involve a User Service (to get profile info), a Recommendation Service, and a Notification Service. The user profile is critical, but recommendations and notifications are not. If the Recommendation Service is down, you should gracefully degrade by showing the rest of the dashboard without the recommendations.

3. Core Strategies for Graceful Degradation

Once you've identified a non-critical service, you can implement one of several degradation strategies when it fails. These strategies can be triggered by a timeout, a retry mechanism giving up, or a circuit breaker opening.

Here are the most common strategies:

1. Return a Fallback/Default Value: This is the simplest strategy. If a service that provides personalized content is down, return a generic, non-personalized default. For example, show a default user avatar instead of the user's custom one, or return an empty list [] for a list of recommended products.

2. Return Stale Data from a Cache: If you're using a cache like Redis, you can serve the last known good data when the primary data source is unavailable. For many use cases, slightly stale data is far more useful to the user than an error message.

3. Return a Partial Response: Construct a response object using data from the services that are still healthy, while providing null or default values for the data from the failed service. The frontend can then render the page with the available components.

4. Reduce Functionality: Provide a simpler, less-featured version of the function. For instance, if a sophisticated search service (e.g., Elasticsearch) is down, you could fall back to a basic LIKE query against the primary database. The user can still search, but without advanced features like faceting or typo correction.

The following resource provides excellent code examples (in TypeScript, but the patterns are universal) for these strategies.

4. Implementing Graceful Degradation in FastAPI

Now, let's translate these concepts into practical Python code within our FastAPI and async context.

Approach 1: Simple try...except Block

The most straightforward way to implement graceful degradation is with a try...except block around the call to a non-critical service.

Let's imagine our API Gateway needs to fetch a user's basic profile and their premium content. The basic profile is critical, but the premium content is not.

# A simplified example within a FastAPI endpoint

from fastapi import FastAPI, HTTPException




# These would be client calls to other gRPC services
async def get_basic_data(user_id: int):



    # In a real app, this makes a gRPC call to the User service
    return {"user_id": user_id, "name": "Alex"}

class ServiceDownError(Exception):
    """Custom exception for when a downstream service is unavailable."""
    pass

async def get_premium_content(user_id: int):



    # In a real app, this makes a gRPC call to the Premium Content service
    # We simulate a failure here
    raise ServiceDownError("Premium content service is down")

app = FastAPI()

@app.get("/user/{user_id}/profile")
async def user_profile(user_id: int):



    # The basic data is critical. If this fails, the whole request should fail.
    basic_data = await get_basic_data(user_id)
    
    premium_content = None # Default value
    try:



        # The premium content is non-critical.
        premium_content = await get_premium_content(user_id)
    except ServiceDownError:



        # The service is down. We've already set premium_content to None.
        # We can log this event for monitoring purposes.
        print("Graceful degradation: Premium content service is down. Returning partial response.")
    
    return {
        "basic": basic_data,
        "premium": premium_content
    }

In this pattern, if get_premium_content() fails, the except block catches the error, and the endpoint returns a partial response with the premium field set to None. The user still gets their essential data.

Approach 2: A Reusable with_fallback Helper

While try...except works, it can become repetitive. We can create a more robust and reusable pattern by encapsulating this logic in an asynchronous helper function.

The "Graceful Degradation" guide we looked at earlier provides a Python implementation of a with_fallback function. It's a powerful pattern that also incorporates a timeout.

Here is that function, which we can add to a utility file in our project:

# in a file like 'utils/resilience.py'
from typing import TypeVar, Callable, Awaitable, Optional
import asyncio
import logging

logger = logging.getLogger(__name__)

T = TypeVar('T')

async def with_fallback(
    primary: Callable[[], Awaitable[T]],
    fallback: Callable[[], Awaitable[T] | T],
    timeout: float = 2.0,
    on_fallback: Optional[Callable[[Exception], None]] = None,
) -> T:
    """
    Executes a primary async function with a timeout. If it fails or times out,
    executes a fallback function and returns its result.
    """
    try:



        # asyncio.wait_for will raise asyncio.TimeoutError if it takes too long
        result = await asyncio.wait_for(primary(), timeout=timeout)
        return result
    except Exception as e:
        logger.warning(f"Primary function failed or timed out: {e}. Executing fallback.")
        if on_fallback:
            on_fallback(e)
        
        fallback_result = fallback()
        if asyncio.iscoroutine(fallback_result):
            return await fallback_result
        return fallback_result

Hands-On Exercise

Now, let's use this helper function. Your task is to implement the user dashboard scenario we discussed earlier.

Scenario: You are building a FastAPI endpoint at /dashboard/{user_id}. This endpoint needs to get data from two downstream services:

1. user_service.get_profile(user_id): Critical. Must succeed.
2. recommendation_service.get_recs(user_id): Non-critical. Can fail.

Instructions:

1. Set up a new Python file for this exercise.
2. Copy the with_fallback helper function into your file.
3. Create mock async functions for get_profile and get_recs. Make get_recs simulate a failure (e.g., by raising an exception or taking too long).
4. Create a fallback function, get_popular_products, that returns a generic list of products.
5. Implement the /dashboard/{user_id} FastAPI endpoint.
   • Call get_profile directly.
   • Use with_fallback to call get_recs, providing get_popular_products as the fallback.
6. Verify that when get_recs fails, the endpoint still returns a 200 OK response with the profile data and the generic popular products.

import asyncio
import logging
from typing import TypeVar, Callable, Awaitable, Optional
from fastapi import FastAPI




# --- Assume this is in utils/resilience.py ---
logger = logging.getLogger(__name__)
T = TypeVar('T')
async def with_fallback(...) -> T:



    # (Paste the full with_fallback function here)
    try:
        result = await asyncio.wait_for(primary(), timeout=timeout)
        return result
    except Exception as e:
        logger.warning(f"Primary function failed or timed out: {e}. Executing fallback.")
        if on_fallback:
            on_fallback(e)
        fallback_result = fallback()
        if asyncio.iscoroutine(fallback_result):
            return await fallback_result
        return fallback_result



# --- End of utility code ---




# --- Mock Service Calls ---
async def get_profile(user_id: int) -> dict:
    """Mock for a critical service call."""
    await asyncio.sleep(0.1) # Simulate network latency
    return {"user_id": user_id, "name": "Bishal", "email": "bishal@example.com"}

async def get_recommendations(user_id: int) -> list:
    """Mock for a non-critical service that fails."""
    await asyncio.sleep(3) # This will time out (timeout is 2.0s by default)
    return [{"id": 101, "name": "Personalized Item A"}]

async def get_popular_products() -> list:
    """Fallback function."""
    return [
        {"id": 999, "name": "Generic Bestseller 1"},
        {"id": 998, "name": "Generic Bestseller 2"},
    ]




# --- FastAPI Application ---
app = FastAPI()

@app.get("/dashboard/{user_id}")
async def get_dashboard(user_id: int):



    # Critical data is fetched directly. If this fails, the request fails.
    profile_data = await get_profile(user_id)
    



    # Non-critical data is fetched with a fallback.
    recommendations_data = await with_fallback(
        primary=lambda: get_recommendations(user_id),
        fallback=get_popular_products
    )
    
    return {
        "profile": profile_data,
        "recommendations": recommendations_data,
    }




# To run this:
# 1. Save as, for example, `main.py`.
# 2. Run with `uvicorn main:app --reload`.
# 3. Access http://127.0.0.1:8000/dashboard/123 in your browser.
# 4. You will see a warning in your console about the primary function failing,
#    and the browser will show the profile data with the generic bestsellers.

Conclusion

You've now learned how to design and implement one of the most important resilience patterns in microservice architecture. By proactively planning for failure, you can build systems that are more robust and provide a significantly better user experience.

Key Takeaways:

• Graceful degradation is about maintaining partial system functionality during failures of non-critical components.
• The first design step is to distinguish between critical services (where you should fail fast) and non-critical ones (where you can degrade). Data integrity is a key factor in this decision.
• Common degradation strategies include returning fallback values, cached data, or partial responses.
• In Python, this pattern can be implemented with a simple try...except block or encapsulated in a reusable asynchronous helper function like with_fallback for cleaner, more maintainable code.

Preview of the Next Lesson

We have now covered the major resilience and security patterns that are crucial for designing robust services. With this solid foundation, we are ready to move from design and implementation to deployment.

The next module marks a significant step in our course: Deployment and Observability. We will begin by learning the core concepts of Kubernetes, the industry-standard container orchestrator. You will understand what Pods, Services, and Deployments are, and how they enable us to run, scale, and manage our microservices in a production environment.