Graceful Handling of Timeouts and Provider Failures

Good to see you again. Last lesson, your OfferInsight reached the dashboard only after it passed schema and business-data validation. There is another path to handle before that validation can run: Ollama might take too long to respond, briefly lose its connection, or return a service error.

Today you’ll make that path predictable. You’ll set a timeout, retry only selected temporary failures, cap the attempts, and return a clear unavailable state when recovery fails. The result is a small resilience layer around the existing model call, not a replacement for your validation boundary. Allow about 40 minutes.

---

Decide what deserves another attempt

Imagine an analyst requesting an explanation of February’s repeat-purchase decline. If the model server briefly disconnects, a second attempt might succeed. If the request names a nonexistent model, repeating it unchanged will not help. And if Ollama returns JSON claiming the metric went up, that is an invalid response—not a connection problem for this layer to retry.

The retry flowchart captures the central decision: after failure, ask whether the error is retryable and whether attempts remain before waiting and trying again.

Watch the selected parts of ArjanCodes’ Retry Pattern: The Secret to Resilient Python Code for the reason behind retries, increasing waits, and caution about repeating operations.

For this application, use the following policy:

• Retry: a timeout, a network interruption, or a temporary server error such as HTTP 503.
• Do not retry unchanged: a malformed request, invalid credentials, or a missing model. Also do not retry InvalidOfferInsight here; last lesson’s acceptance boundary should withhold it.
• Treat rate limiting separately: HTTP 429 may be temporary, but a short automatic retry can ignore the provider’s requested cooldown. This first version reports it as unavailable rather than immediately trying again.

A timeout needs one more caveat: the provider may have received and processed a request even though your client stopped waiting. Retrying an insight generation can produce another response—and, with a paid provider, another charge. Keep dashboard writes and other consequential actions outside the retried operation.

---

Add a bounded retry policy

Your model_client.py already constructs an Ollama Client with timeout=settings.model_timeout_seconds. Keep that setting. It bounds how long the client waits on its HTTP operations; it is not a guaranteed deadline for the entire multi-attempt user request. We will also bound the number of attempts and the waits between them.

Add the dependencies explicitly to your existing uv project:

uv add tenacity httpx

httpx is listed explicitly because the new code imports its exception classes, even if it was previously installed as another package’s dependency. uv add updates the project and lockfile; subsequent runs can continue using uv run --locked.

Before coding, read the indicated portions of the official Tenacity documentation. One important default to notice: an unconfigured retry can continue indefinitely. We will specify both an exception filter and a stop condition.

Create provider_reliability.py:

from collections.abc import Callable
from typing import TypeVar

import httpx
from ollama import ResponseError
from tenacity import (
    Retrying,
    retry_if_exception,
    stop_after_attempt,
    wait_random_exponential,
)


T = TypeVar("T")

# HTTP 408 is a request timeout; these 5xx codes indicate server trouble.
RETRYABLE_STATUSES = {408, 500, 502, 503, 504}


class ModelUnavailable(RuntimeError):
    """No model result is available to the caller right now."""


class ModelRequestFailed(RuntimeError):
    """The request or backend configuration needs attention."""


def is_transient(exc: BaseException) -> bool:
    if isinstance(
        exc,
        (httpx.TimeoutException, httpx.NetworkError, httpx.RemoteProtocolError),
    ):
        return True

    return (
        isinstance(exc, ResponseError)
        and exc.status_code in RETRYABLE_STATUSES
    )


def call_with_retries(call: Callable[[], T]) -> T:
    """Retry one provider operation, never its downstream consumers."""
    try:
        for attempt in Retrying(
            retry=retry_if_exception(is_transient),
            stop=stop_after_attempt(3),
            wait=wait_random_exponential(multiplier=0.5, max=2),
            reraise=True,
        ):
            with attempt:
                return call()
    except (
        httpx.TimeoutException,
        httpx.NetworkError,
        httpx.RemoteProtocolError,
    ) as exc:
        raise ModelUnavailable(
            "Model service did not respond after limited attempts"
        ) from exc
    except ResponseError as exc:
        if exc.status_code == 429:
            raise ModelUnavailable(
                "Model service is rate-limited; try again later"
            ) from exc
        if is_transient(exc):
            raise ModelUnavailable(
                "Model service is temporarily unavailable"
            ) from exc
        raise ModelRequestFailed(
            "Model request failed; check its configuration and inputs"
        ) from exc

    raise AssertionError("Retry loop ended without a result or exception")

There are at most three calls total, not three retries after the first call. wait_random_exponential adds a capped, increasing random wait between eligible failures; it does not delay the first attempt. With reraise=True, the final underlying failure remains available to this wrapper, which translates it into an application-level error.

The except blocks deliberately contain fixed messages. They do not send the provider’s raw error text to a dashboard user; the original exception remains chained for controlled diagnosis. A non-provider programming error is not caught and disguised as a temporary outage.

---

Apply retries only to the provider call

In last lesson’s generate_offer_insight() in model_client.py, add:

from provider_reliability import call_with_retries

Keep the existing prompt, Client(...), and validation code. Replace just the client.chat(...) assignment with:

response = call_with_retries(
    lambda: client.chat(
        model=model_name,
        messages=[{"role": "user", "content": prompt}],
        format=OfferInsight.model_json_schema(),
    )
)

Leave this part after the retried call, unchanged:

if response.message.content is None:
    raise InvalidOfferInsight("Model returned no JSON content")

return validate_offer_insight(
    response.message.content,
    expected_metric=expected_metric,
    previous_value=previous_value,
    current_value=current_value,
)

That placement is the design choice. A temporary transport failure can trigger another client.chat() attempt. A response that fails validate_offer_insight() cannot: it is withheld by the acceptance boundary you already built. The retry wrapper also does not repeat a dashboard write.

Now add the two provider-failure cases to the existing try block in app.py:

from provider_reliability import ModelRequestFailed, ModelUnavailable

# Keep the existing generate_offer_insight(...) call and its arguments.
try:
    insight = generate_offer_insight(
        note,
        Settings(),
        expected_metric="repeat_purchase_rate",
        previous_value=previous_rate,
        current_value=current_rate,
    )
except InvalidOfferInsight:
    print("Insight withheld: model output failed validation.")
    return
except ModelUnavailable:
    print("Insight unavailable right now; please try again later.")
    return
except ModelRequestFailed:
    print("Insight unavailable: model request needs attention.")
    return

print(insight.model_dump_json(indent=2))

In a FastAPI or dashboard version, these branches would become appropriate unavailable/error states rather than printed lines. Do not show raw model text as a fallback: a failed request has no accepted insight to display.

This implementation targets your current local Ollama structured-output path. If generate_text() uses a different provider client, reuse the policy, but map that client’s timeout and service-error classes at its own adapter boundary; do not assume it raises Ollama or HTTPX exceptions.

---

Check the behavior without waiting for a real outage

A real server is a poor way to reproduce “timeout, then recovery.” Create retry_demo.py with a controlled call:

import httpx
from ollama import ResponseError

from provider_reliability import (
    ModelRequestFailed,
    ModelUnavailable,
    call_with_retries,
)


events = iter(
    [
        httpx.ReadTimeout("simulated slow response"),
        ResponseError("simulated service outage", status_code=503),
        "model response",
    ]
)
attempts = 0


def recovers() -> str:
    global attempts
    attempts += 1
    event = next(events)
    if isinstance(event, Exception):
        raise event
    return event


print(call_with_retries(recovers), f"after {attempts} attempts")


def bad_request() -> str:
    raise ResponseError("simulated bad request", status_code=400)


try:
    call_with_retries(bad_request)
except ModelRequestFailed:
    print("Bad request: stopped without retrying")


try:
    call_with_retries(
        lambda: (_ for _ in ()).throw(
            httpx.ReadTimeout("simulated persistent timeout")
        )
    )
except ModelUnavailable:
    print("Persistent timeout: stopped after the attempt limit")

Run:

uv run --locked python retry_demo.py

You should see recovery after three attempts, immediate failure for the bad request, and an unavailable result for persistent timeouts. This is a quick behavior check, not yet an integration test of generate_offer_insight(); the next lesson will test that full path with a controlled model response.

If your application later uses a hosted provider, revisit the limits using its latency, cost, and rate-limit guidance. In particular, honor any Retry-After instruction before automatically retrying HTTP 429. Also remember that three per-operation timeouts plus waits may be too slow for an interactive request even though each individual setting is bounded.

---

Takeaway

A graceful model failure is not necessarily a successful retry. It means the application makes a limited, selective recovery attempt and then returns a clear unavailable state without exposing unvalidated content. Your timeout limits waiting on the provider, Tenacity limits repeated calls, and last lesson’s validation boundary still decides whether a returned insight may be used.

Next, you’ll verify the combined call-and-validation path with an integration test that supplies a controlled model response.