Integration Testing with Controlled Model Responses

Good to see you again. Your application now has two important boundaries: the retry layer handles selected provider failures, and the validation layer withholds unacceptable model output. Last lesson’s simulated outage checked the retry function, but it did not prove that a model response can travel through the full generate_offer_insight() path and emerge as an accepted OfferInsight.

Today you’ll write that test. You’ll replace only the Ollama client during the test, supply a response you control, and leave the prompt construction, retry wrapper, and validation code running normally. This should take about 40 minutes, including the short resources and implementation.

Choose the boundary you want to test

A live Ollama call would make this test slow and dependent on which model is installed—and the model might phrase its answer differently on the next run. Instead, keep your application code real up to the provider boundary and substitute a predictable client there.

That makes this an integration test of application components, not a live-provider test. It checks that generate_offer_insight() sends a structured-output request, receives response content, and passes it through the real acceptance boundary. It cannot establish that Ollama is currently reachable; that would require a separate live check.

Watch the opening and API-mocking example in NeuralNine’s Professional Python Testing with Mocks. The example uses unittest.mock.patch rather than the pytest monkeypatch fixture you’ll use below, but the boundary decision is the same: replace the external call, not the processing you intend to verify.

Pytest’s official How to monkeypatch/mock modules and environments shows the fixture we’ll use. Its API example returns an object with just the behavior the application needs—much as our fake will return an object with message.content.

The last point is easy to miss. If model_client.py says from ollama import Client, then generate_offer_insight() looks up model_client.Client. Patch that name, not an unrelated reference to ollama.Client.

Freeze one accepted business response

Create tests/fixtures/valid_offer_insight.json. Put in it the exact JSON content of one response that your existing validate_offer_insight() accepted for a repeat-purchase-rate decline. Use a sanitized example, not customer information or secrets. The file is controlled test data: pytest will replay it, never ask a model to regenerate it.

The code below uses these reference inputs:

METRIC = "repeat_purchase_rate"
PREVIOUS_RATE = 0.32
CURRENT_RATE = 0.28
NOTE = "February repeat purchase rate fell from 0.32 to 0.28."

Make the fixture agree with those inputs and with your actual OfferInsight schema. In particular, its metric and any stated numerical change must be consistent with the reference values. If your previously accepted example uses different values, change the constants instead of inventing fields for the JSON. If you have not saved an accepted response, obtain and validate one using your existing local-model path once, then freeze its content in the fixture. Test runs themselves should need no model server.

This small fixture is not yet a representative evaluation dataset. It is one known response for checking the application wiring.

Write the integration test

Add pytest to your existing uv project, then create tests/test_model_client_integration.py:

uv add --dev pytest

The example below assumes Settings, OfferInsight, and InvalidOfferInsight are accessible as names in model_client.py, and that it imports Ollama’s constructor as Client. If your earlier files put those names elsewhere, import them from their actual modules; patch the constructor name that generate_offer_insight() looks up.

from pathlib import Path
from types import SimpleNamespace

import pytest

import model_client


FIXTURE = Path(__file__).parent / "fixtures" / "valid_offer_insight.json"
METRIC = "repeat_purchase_rate"
PREVIOUS_RATE = 0.32
CURRENT_RATE = 0.28
NOTE = "February repeat purchase rate fell from 0.32 to 0.28."


@pytest.fixture
def controlled_client(monkeypatch):
    state = SimpleNamespace(content=None, calls=[], constructors=[])

    class FakeClient:
        def __init__(self, *args, **kwargs):
            state.constructors.append((args, kwargs))

        def chat(self, **kwargs):
            state.calls.append(kwargs)
            return SimpleNamespace(
                message=SimpleNamespace(content=state.content)
            )

    monkeypatch.setattr(model_client, "Client", FakeClient)
    return state


def test_generate_offer_insight_accepts_controlled_response(
    controlled_client,
):
    payload = FIXTURE.read_text(encoding="utf-8")
    controlled_client.content = payload
    settings = model_client.Settings()

    insight = model_client.generate_offer_insight(
        NOTE,
        settings,
        expected_metric=METRIC,
        previous_value=PREVIOUS_RATE,
        current_value=CURRENT_RATE,
    )

    assert isinstance(insight, model_client.OfferInsight)
    assert insight == model_client.OfferInsight.model_validate_json(payload)

    assert len(controlled_client.calls) == 1
    request = controlled_client.calls[0]
    assert request["format"] == model_client.OfferInsight.model_json_schema()
    assert any(
        NOTE in message["content"] for message in request["messages"]
    )
    assert controlled_client.constructors[0][1]["timeout"] == (
        settings.model_timeout_seconds
    )


def test_generate_offer_insight_withholds_wrong_metric(
    controlled_client,
):
    controlled_client.content = FIXTURE.read_text(encoding="utf-8")

    with pytest.raises(model_client.InvalidOfferInsight):
        model_client.generate_offer_insight(
            NOTE,
            model_client.Settings(),
            expected_metric="gross_margin",
            previous_value=PREVIOUS_RATE,
            current_value=CURRENT_RATE,
        )

    assert len(controlled_client.calls) == 1

The first test checks more than “the JSON parses.” It checks the returned typed object, the schema sent to the client, inclusion of the business note in the request, the configured timeout, and the fact that one successful provider operation took one call. Do not replace validate_offer_insight() in this test: its real behavior is part of what you are verifying.

The second test replays the same response but asks the application to accept it as gross_margin. A schema-valid response about repeat purchases must not silently become an insight about a different metric. The single-call assertion also checks the boundary from last lesson: validation rejection happens after the provider call, outside the retry wrapper, so it does not request another generation.

monkeypatch restores model_client.Client after each test. The fixture also gives each test fresh call records, so results do not depend on execution order.

Run it and read failures as evidence

From the project root, run:

uv run --locked pytest -q tests/test_model_client_integration.py

If the first test fails, use the assertion to narrow the cause:

• Fixture rejected: check that its fields match your real schema and that its metric, values, and claimed direction match the reference inputs. Do not weaken validation merely to make the test green.
• No fake client calls recorded: check how model_client.py imports and looks up Client. You may have patched a name the function never uses.
• Request assertion fails: inspect the request your code actually builds. For example, if it legitimately reformats NOTE rather than including it verbatim, assert the meaningful business information that must appear instead of insisting on exact text.
• Settings construction fails: supply the required non-secret test configuration before constructing Settings; the test should not need a real provider credential.

The assertions concerning format and timeout reflect the structured-output and client configuration from your preceding lessons. If your implementation expresses either through a different argument, adjust the assertion to inspect that actual contract—not to stop checking it.

A passing result means that, for this controlled response, the call-and-validation path behaves as specified without network access. It does not mean every plausible request or response has been covered.

Takeaway

You now have a repeatable integration check at the right boundary: the provider response is controlled, while your request construction, retry placement, typed parsing, and business validation remain real. One accepted response reaches the caller; the same response cannot be relabeled as another metric.

Next, you’ll expand from one carefully chosen fixture to a representative golden dataset of user requests, so evaluation reflects the range of questions the application will actually face.