Concurrent HTTP Requests with Async/Await and Timeouts

Hello! Last lesson, you used pytest to check how a model-call function behaves when its HTTP dependency succeeds, fails, or times out. This lesson changes how requests are scheduled: you’ll use async and await to let several HTTP requests make progress during the same period of network waiting, then put limits on how long they may take.

Allow about 40 minutes to watch the selected clips, read the short documentation passages, and run the example. You won’t need a running model server; the example uses HTTPX’s controlled transport, as your earlier tests did.

An async function does not create concurrency by itself

Calling a function defined with async def produces a coroutine object. Its body runs when the coroutine is awaited or scheduled on an event loop. From a normal Python script, asyncio.run(main()) starts that loop and runs the top-level coroutine.

Watch Patrick Loeber’s HTTPX Tutorial - A next-generation HTTP client for Python. The first clip shows the request syntax; the later clip contrasts merely using async syntax with actually scheduling multiple requests together.

The Python documentation’s examples make the scheduling rule precise:

Consider two independent model calls. This code waits for the first response before starting the second:

first = await fetch_reply(client, "leave")
second = await fetch_reply(client, "travel")

Both calls use async syntax, but their waits are sequential. For independent requests, asyncio.gather() schedules the coroutines together and waits for their results:

first, second = await asyncio.gather(
    fetch_reply(client, "leave"),
    fetch_reply(client, "travel"),
)

While one request waits for I/O, the event loop can run the other. This is concurrency, not a promise that Python will perform CPU-heavy work in parallel or that every server will respond faster under load.

In the documentation’s “Running tasks concurrently” subsection, check two details that will matter in the example: successful gather() results retain input order, and an unexpected exception has different consequences from an individual request returning an error value.

Use one async HTTP client for the batch

HTTPX supplies httpx.AsyncClient for async requests. Its post() method must be awaited, and async with closes the client when the batch finishes—even if the batch raises an exception. Create the client outside the request loop so the requests can share its connection pool.

The distinction from your earlier synchronous call_model is small but essential: use an AsyncClient and write response = await client.post(...). Calling a synchronous HTTP client from inside async def would still block the event loop during the request.

Put a limit on both requests and the batch

There are two useful, different time limits:

• An HTTPX timeout limits waits in HTTP operations, such as connecting or waiting for response data. It is not necessarily a maximum wall-clock duration for an entire batch.
• An asyncio.timeout() deadline limits the time spent inside its block. Here it protects the whole batch if the work does not finish promptly. It requires Python 3.11 or newer.

Read the first asyncio.timeout() entry in the documentation’s “Timeouts” subsection. Notice where the TimeoutError must be caught.

Here is a runnable batch using the /generate route from the previous lessons. MockTransport handles the HTTP requests locally, so this demonstrates scheduling without depending on a model server. Save it as async_batch.py in your uv project:

import asyncio
import json
import time

import httpx


async def fake_model(request: httpx.Request) -> httpx.Response:
    prompt = json.loads(request.content)["prompt"]
    await asyncio.sleep(0.25)  # Simulated I/O wait; does not block the loop
    return httpx.Response(
        200,
        json={"response": f"Reply to {prompt}"},
    )


async def fetch_reply(
    client: httpx.AsyncClient, prompt: str
) -> tuple[str, str]:
    try:
        response = await client.post(
            "/generate",
            json={"prompt": prompt},
        )
        response.raise_for_status()
        return prompt, response.json()["response"]
    except httpx.TimeoutException:
        return prompt, "request timed out"
    except httpx.HTTPStatusError as exc:
        return prompt, f"HTTP {exc.response.status_code}"


async def main() -> None:
    prompts = ["leave", "travel", "security"]

    async with httpx.AsyncClient(
        base_url="http://localhost:11434",
        transport=httpx.MockTransport(fake_model),
        timeout=httpx.Timeout(1.0),
    ) as client:
        start = time.perf_counter()

        try:
            async with asyncio.timeout(2.0):
                replies = await asyncio.gather(
                    *(fetch_reply(client, prompt) for prompt in prompts)
                )
        except TimeoutError:
            print("The batch exceeded its 2-second deadline.")
            return

        for prompt, reply in replies:
            print(f"{prompt}: {reply}")
        print(f"Elapsed: {time.perf_counter() - start:.2f}s")


if __name__ == "__main__":
    asyncio.run(main())

Run it with uv run python async_batch.py. You should see the three prompts in their original order and an elapsed time nearer 0.25 seconds than 0.75 seconds, though exact timing varies. Each coroutine waits for 0.25 seconds, but those waits overlap. The expression passed to gather() supplies one coroutine per prompt; gather() schedules them, so separate create_task() calls are unnecessary here.

The two timeout handlers intentionally mean different things. httpx.TimeoutException concerns an individual HTTP request, so fetch_reply() can return a result for that prompt while preserving other results. The outer TimeoutError means the batch deadline expired; it is caught after the asyncio.timeout() block exits. Cancelling the batch’s gather() also cancels its unfinished submitted requests. Other failures, such as malformed JSON, are not silently turned into timeout results.

One testing caveat: the simulated asyncio.sleep() demonstrates overlapping waits and can demonstrate the batch deadline, but MockTransport is not a real socket. Do not treat this run as proof that HTTPX’s network-phase timeout fires for a slow model server. The /generate handler is likewise a local stand-in, not a verified live model API.

Takeaway

async def makes a coroutine possible; await runs and waits for async work; and asyncio.gather() lets independent HTTP requests progress concurrently while returning results in input order. Use one scoped AsyncClient, set HTTP timeouts for individual requests, and use an overall deadline when the entire batch must finish within a limit.

Next, you’ll replace the stand-in handler with a real Ollama HTTP call and deal with its actual success and failure responses.