Invoking Language Models with a Provider-Neutral Python Interface

Hi! Last lesson, you made the application environment reproducible and moved model settings out of source code. The project still cannot invoke a model: local-placeholder is deliberately not a real model name, and the configuration smoke check makes no inference request.

In this lesson, you’ll replace that smoke check with a working ecommerce analysis request. The deliverable is a generate_text() function that the rest of your application can call without importing an Ollama or hosted-provider SDK. We’ll use a small local model first; a paid hosted API is optional. Allow about 40–45 minutes, including setup and the first run.

---

See the interface you’re building toward

Suppose an offer-analysis feature needs one sentence of model-generated text. Its business code should be able to call generate_text(prompt, settings) without knowing whether the request goes to Ollama or a hosted service. That separation is useful even when you expect to keep one provider: model routing and credentials have a clear place to live, rather than spreading through the feature.

Watch the SDK portion of LiteLLM Crash Course by Data Science Basics. It shows what stays the same—and what must change—when a Python call moves between hosted and local models. We are using the Python SDK directly, not setting up the LiteLLM proxy shown later in the video.

The LiteLLM documentation is the reference for the two pieces your code will consume: completion(model=..., messages=...) and the returned response.choices[0].message.content. Its page shows several providers; you only need the Ollama and OpenAI examples today.

LiteLLM provides a common call and response interface; it does not make providers identical. A model still needs a valid route, a reachable endpoint, and whatever authentication that provider requires.

---

Get one local model ready

Continue in the uv project from last lesson. Add LiteLLM as a declared dependency so it enters pyproject.toml and uv.lock:

uv add litellm
uv sync --locked

For the free/local path, use Ollama with the small qwen2.5:1.5b model. If Ollama is already running on your machine, leave it running; otherwise start its installed application or run ollama serve in a separate terminal. Then download and confirm the model:

ollama pull qwen2.5:1.5b
ollama list

The pull needs internet access and disk space; inference runs through your local Ollama server. If you have a different suitable Ollama chat model already installed, you can use its exact name instead. If you are running Python in a hosted notebook, localhost refers to the notebook’s machine, not your laptop—run Ollama where the Python process runs, or use the optional hosted route below.

We’ll use LiteLLM’s ollama_chat/ prefix to send chat messages to Ollama’s chat endpoint. Its Ollama documentation shows that prefix alongside the request format.

Update the existing fields in config.py and add model_api_base. Keep the other settings and validation from last lesson:

class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
    )

    model_backend: Literal["local", "hosted"] = "local"
    model_name: str = "ollama_chat/qwen2.5:1.5b"
    model_api_base: str = "http://localhost:11434"
    model_timeout_seconds: float = Field(default=20.0, gt=0, le=120)
    model_api_key: SecretStr | None = None

Set the same local values in your ignored .env. Update the committable .env.example too, but do not put a real key in either example or Git:

MODEL_BACKEND=local
MODEL_NAME=ollama_chat/qwen2.5:1.5b
MODEL_API_BASE=http://localhost:11434
MODEL_TIMEOUT_SECONDS=20
# MODEL_API_KEY=

The model name has two parts here: ollama_chat/ selects LiteLLM’s route, and qwen2.5:1.5b names the model you pulled. Changing only one of those to an unrelated value will not create a working model.

---

Put provider details behind one Python function

Create model_client.py:

from litellm import completion

from config import Settings


def generate_text(prompt: str, settings: Settings) -> str:
    """Return text from the configured model."""
    if not prompt.strip():
        raise ValueError("Prompt must not be blank")

    provider_options: dict[str, object] = {}

    if settings.model_backend == "local":
        if not settings.model_name.startswith("ollama_chat/"):
            raise ValueError("Local MODEL_NAME must begin with ollama_chat/")
        provider_options["api_base"] = settings.model_api_base
    else:
        if (
            settings.model_api_key is None
            or not settings.model_api_key.get_secret_value().strip()
        ):
            raise ValueError(
                "MODEL_API_KEY is required when MODEL_BACKEND=hosted"
            )
        provider_options["api_key"] = (
            settings.model_api_key.get_secret_value()
        )

    response = completion(
        model=settings.model_name,
        messages=[{"role": "user", "content": prompt}],
        timeout=settings.model_timeout_seconds,
        **provider_options,
    )

    text = response.choices[0].message.content
    if text is None or not text.strip():
        raise RuntimeError("The model returned no text")
    return text.strip()

There are two interfaces at work. LiteLLM’s completion() gives this adapter a common way to submit messages and read a response. Your application’s generate_text() gives business code an even smaller contract: supply text and settings; receive text. The application need not handle LiteLLM response objects or know about Ollama’s endpoint.

The local and hosted branches contain genuine differences: Ollama needs an api_base, while the optional hosted route needs a key. The timeout setting is passed into the request rather than left unused. This is a first invocation boundary, though—not yet a complete policy for retries, failure recovery, or validating business-facing output. Those come in later lessons.

Replace the previous app.py smoke check with a call that represents your analytics work:

from config import Settings
from model_client import generate_text


def main() -> None:
    settings = Settings()
    prompt = (
        "A demo ecommerce store has a declining repeat-purchase rate. "
        "Suggest one question an analyst should investigate next. "
        "Answer in one sentence."
    )
    print(generate_text(prompt, settings))


if __name__ == "__main__":
    main()

No customer data or confidential sales figures are needed for this connectivity check. The answer’s wording may vary; success means the command reaches the configured model and prints a nonempty response.

---

Run it, then locate the boundary

With Ollama running, execute:

uv run --locked python app.py

If the call fails, check the boundary in order: does ollama list show the exact model you named; is the Ollama server running; does .env contain ollama_chat/ followed by that model name; and does MODEL_API_BASE point to the server from this Python process? If your machine is slow on the first call, you can raise MODEL_TIMEOUT_SECONDS within the validated range and rerun. Do not put a key in the local configuration to solve a local connectivity problem.

You can now see the architectural payoff in concrete files. app.py expresses the business request, config.py holds deployment choices, and model_client.py owns model invocation. In the vibe-coded feature you audited, the next refactor would replace its provider-specific call with generate_text() while leaving its surrounding business logic intact.

If you already have a paid OpenAI API key and want to verify the alternate route, you may change only the ignored .env to MODEL_BACKEND=hosted, MODEL_NAME=openai/gpt-4o-mini, and MODEL_API_KEY= followed by your key, then run the same command. This is optional and may incur charges. The code does not send the local MODEL_API_BASE to that hosted route. Other hosted providers can require different credentials or settings; add those within the adapter rather than assuming that every vendor is an .env-only swap.

---

Takeaway

You’ve made a real model call through a provider-neutral application interface, using LiteLLM for a consistent request and response shape. Routing, endpoint, credentials, and timeout stay outside the ecommerce feature, while the local path works without a paid API.

Keep the changed pyproject.toml, uv.lock, config.py, model_client.py, app.py, and safe .env.example; keep .env ignored. Next, you’ll move beyond receiving arbitrary text and ask a model for output shaped to a typed business data model.