Running Ollama Locally with Python Completions

Hello! You’ve sent the same fictional document question to Groq and Mistral. Now you’ll run a model on your own computer and send it that question from Python. This gives you a third provider to compare before you build the common interface—but no document retrieval is involved yet.

Plan on about 40 minutes of active work, plus any time needed to download a model. You’ll install and run Ollama, make one non-streamed chat request through its Python library, and inspect what comes back.

Get a model running locally

Ollama has two parts worth distinguishing: a local service that runs models and a command-line client you can use to talk to that service. Your Python program will contact the same service, normally at http://127.0.0.1:11434. Unlike the hosted calls, a request to a model running there needs no Groq or Mistral API key.

For a quick view of the setup, watch the selected portions of Learn Ollama in 15 Minutes – Run LLM Models Locally for FREE by Tech With Tim. The video uses different model names from this lesson; concentrate on the installation, model-download, and local-service workflow rather than copying its model choice.

Install Ollama for your operating system and start its application or service. On Linux, you may need to run ollama serve in a separate terminal; if Ollama is already serving requests, do not start a second copy. Then, in a terminal, download a relatively small model:

ollama pull phi3
ollama list

ollama list should show phi3. The download needs internet access and disk space; generation will also need enough available memory. If this model does not fit your machine, choose a smaller Ollama model you can run and use its exact listed name throughout the rest of the lesson.

Check that it answers interactively:

ollama run phi3

Ask it when the fictional sample library opens on weekdays, using the test note from the previous lessons: “The sample library opens at 9 a.m. on weekdays.” Type /bye to leave the chat. This terminal test separates a model or service problem from a Python-code problem.

Call the model from your project

Return to your existing doc-qa project. Ollama’s Python library makes a direct provider call—there is no LangChain layer here. In the Ollama Python Library README, read how its prerequisites, installation, and Usage example fit together.

Install the library into the same reproducible project as your other smoke tests:

uv add ollama

Extend your existing config.py so that ollama is an accepted APP_PROVIDER. In Ollama mode, it should not require either hosted-provider key. Give your settings ollama_model and ollama_host fields, loaded from OLLAMA_MODEL and OLLAMA_HOST respectively; you can use phi3 and http://127.0.0.1:11434 as defaults. Add these entries to your ignored local .env:

APP_PROVIDER=ollama
OLLAMA_MODEL=phi3
OLLAMA_HOST=http://127.0.0.1:11434

If you maintain .env.example, document those three values there too, without adding a key for Ollama. If you selected a different installed model, change OLLAMA_MODEL to match. The script below uses the settings-field names just described; adapt them if your config.py uses different names.

Create ollama_smoke.py:

# ollama_smoke.py
from time import perf_counter

from ollama import Client

from config import load_settings


def main() -> None:
    settings = load_settings()
    if settings.provider != "ollama":
        raise ValueError("Set APP_PROVIDER=ollama to run this Ollama test")

    messages = [
        {
            "role": "system",
            "content": "You are a concise assistant. Answer in one sentence.",
        },
        {
            "role": "user",
            "content": (
                "Test note: The sample library opens at 9 a.m. on weekdays.\n"
                "Question: When does it open on weekdays?"
            ),
        },
    ]

    client = Client(host=settings.ollama_host, timeout=120.0)

    started = perf_counter()
    response = client.chat(
        model=settings.ollama_model,
        messages=messages,
        options={"num_predict": 96},
        stream=False,
    )
    elapsed = perf_counter() - started

    text = response.message.content
    if not isinstance(text, str) or not text.strip():
        raise RuntimeError("Ollama did not return a nonempty text answer")

    print(f"Returned model: {response.model}")
    print(f"Returned role: {response.message.role}")
    print(f"Generated text: {text}")
    print(f"Prompt tokens: {response.prompt_eval_count}")
    print(f"Generated tokens: {response.eval_count}")
    print(f"Measured SDK call time (s): {elapsed:.3f}")

    if response.total_duration is not None:
        print(
            "Ollama-reported total duration (s): "
            f"{response.total_duration / 1_000_000_000:.3f}"
        )


if __name__ == "__main__":
    main()

Run it from the project root:

uv run --locked python ollama_smoke.py

The messages are identical to those in your Groq and Mistral tests. num_predict is Ollama’s output-token limit setting for this call; the hosted SDKs used differently named parameters. We have not fixed every sampling setting, so differences in wording are not a controlled quality comparison. A successful run should return an assistant message that says the library opens at 9 a.m. on weekdays, though its exact phrasing may vary.

Read the result—and diagnose failures at the right layer

Your Python code sends messages to a local service and waits for a completed response. That makes the workflow familiar, but the response shape differs: Ollama puts the answer at response.message.content, rather than in a first choices entry. Its prompt_eval_count and eval_count are token counts to inspect, not fields you should assume every provider supplies under the same names. The script also measures elapsed time around the call; Ollama’s reported duration is a separate, provider-specific measurement in nanoseconds. A first call may be slower because the model must load into memory.

If the run fails, use what you already tested to narrow it down:

• Connection refused: Check that the Ollama application or service is running and that OLLAMA_HOST points to it. Try the terminal model test again.
• Model not found: Compare OLLAMA_MODEL with ollama list. A model name used by a hosted API is not automatically an Ollama model name.
• Very slow response or memory failure: The local machine, not a hosted provider, is doing inference. Try a smaller installed model and allow extra time for its first load.
• Hosted-key error before the call: Your config.py still requires Groq or Mistral credentials in Ollama mode. Make key validation conditional on the selected provider.

The fixed test note is safe for a smoke test. Once your application accepts real documents and questions, avoid printing their contents in diagnostics. Also, “local” describes this setup because you selected a locally running model at a loopback address; it is not a blanket privacy guarantee for every Ollama configuration.

What you have established

You can now download and run a local model, reach it from Python, and inspect its answer, token counts, and timing without a hosted API key. Keep this smoke script beside the Groq and Mistral versions: each independently verifies one provider.

Next, you’ll turn those three provider-specific calls into one configurable model interface with a consistent response shape.