Evaluating Quantized Language Models with Ollama on Apple Silicon

Welcome back. Last lesson placed a local language model behind the Python API in your document-assistant architecture. Today you’ll run that component on your Apple Silicon Mac and collect evidence about whether it is practical for the application: memory use, latency, and output quality. Allow about 40 minutes of hands-on work and study, plus any time needed for the initial model download.

You will use invented receipt data, not personal files. A successful result is not merely “the model answered”: it is a recorded model and quantization level, repeatable timing measurements, an observed memory change, and a judgment about what the answers got right or wrong.

Why start with a quantized model?

A language model’s weights occupy memory while it runs. Quantization stores those weights at reduced numerical precision, making a model more feasible on a machine with limited memory. It is a trade-off, not a free compression setting: a smaller representation may affect answer quality. The model’s weights are also not its only memory expense; the context being processed, runtime buffers, and other Mac applications need room.

Watch Matt Williams’s Optimize Your AI - Quantization Explained for the basic idea and his recommendation to start with a Q4 variant. Treat that recommendation as a starting point to test, not a guarantee about your workload.

We’ll try qwen2.5:7b, provided that the downloaded model reports a quantized format. On a Mac with 16 GB of unified memory, a roughly 7-billion-parameter model is an experiment, not a promise that memory pressure will remain comfortable. The model download’s disk size is not a measurement of its total memory use during inference. If it causes sustained memory pressure or substantial swapping, use qwen2.5:3b instead and record that change. In either case, verify the quantization shown for the model you actually ran.

Install, run, and identify the model

Ollama supplies both a command-line interface and a local API. The quickstart shows the macOS installation route, a local model download, and an API request. Its example uses a different model from ours, so follow its method rather than copying its model name into this experiment.

If the Mac application is not installed, install and open Ollama using the Quickstart’s macOS instructions. Alex Ziskind’s FREE Local LLMs on Apple Silicon | FAST! also shows what the running app and Activity Monitor look like. His demonstration uses a 64 GB Mac: use it to locate the controls, not to predict results on your machine.

With Ollama running, use Terminal:

ollama --version
ollama pull qwen2.5:7b
ollama show qwen2.5:7b
ollama run qwen2.5:7b

The pull needs internet access to fetch model files; subsequent requests in this lesson go to the local server. In ollama show, find the quantization field and write down its value—for example, a Q4-family value if that is what your installed tag reports. Tags and defaults can change; do not label a model “Q4” solely because an example says it is. If the field does not show a quantized format, choose a quantized model variant before making that claim.

At the interactive prompt, send a harmless sentence such as “Reply with one short greeting.” Type /bye to leave. If you switch to the smaller model, pull and inspect qwen2.5:3b instead, and substitute that name in the remaining commands and script. Close memory-heavy apps if needed, but record what you changed so the observation has context.

Measure a cold request and a warm request

An interactive chat establishes that the model works, but it does not give you an easily comparable record. Ollama’s API returns timing and token counts. Read its short metric definitions before using them.

We will submit the same evidence question twice. First unload the model so the first request has an opportunity to measure loading; the next request should normally find it resident. A third request tests behavior when the supplied evidence lacks the requested fact.

Create measure_ollama.py with the following code. It uses Python’s standard library and sends requests only to the loopback address. If python3 is not available, you will need a local Python installation for this script and for the FastAPI work in the next lesson.

import json
import time
import urllib.request

MODEL = "qwen2.5:7b"  # Change to "qwen2.5:3b" if that is your chosen model.
URL = "http://127.0.0.1:11434/api/generate"

evidence = (
    "Evidence (invented for this test):\n"
    "File: sample-receipt.pdf; page: 1\n"
    "Item: prescription glasses\n"
    "Purchase date: 2024-03-14\n"
    "Total: $129.00\n"
)
instruction = (
    "Answer using only the supplied evidence. Cite the file and page "
    "for facts you report. If a requested fact is absent, reply "
    "'Insufficient evidence.' Do not guess.\n\n"
)
grounded = instruction + evidence + "\nQuestion: What were the purchase date and total?"
missing = instruction + evidence + "\nQuestion: What were the payment card's last four digits?"

cases = [
    ("first grounded request", grounded),
    ("repeat grounded request", grounded),
    ("missing-fact request", missing),
]

for label, prompt in cases:
    payload = {
        "model": MODEL,
        "prompt": prompt,
        "stream": False,
        "keep_alive": "10m",
        "options": {"temperature": 0, "num_ctx": 2048, "num_predict": 120},
    }
    request = urllib.request.Request(
        URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
    )
    start = time.perf_counter()
    with urllib.request.urlopen(request, timeout=180) as response:
        result = json.load(response)
    wall_seconds = time.perf_counter() - start

    seconds = lambda field: result.get(field, 0) / 1_000_000_000
    output_seconds = seconds("eval_duration")
    output_tokens = result.get("eval_count", 0)
    rate = output_tokens / output_seconds if output_seconds else 0

    print(f"\n{label} | model: {MODEL}")
    print(
        f"wall: {wall_seconds:.2f}s | API total: "
        f"{seconds('total_duration'):.2f}s | load: "
        f"{seconds('load_duration'):.2f}s"
    )
    print(
        f"prompt: {result.get('prompt_eval_count', 0)} tokens / "
        f"{seconds('prompt_eval_duration'):.2f}s | output: "
        f"{output_tokens} tokens / {output_seconds:.2f}s | "
        f"output rate: {rate:.1f} tokens/s"
    )
    print("Answer:", result.get("response", ""))

The fixed context limit and output limit help make these runs comparable. temperature: 0 reduces sampling variation, but does not establish that every run will be identical. keep_alive gives you time to inspect memory while the model remains loaded.

Before running the script, open Activity Monitor → Memory. Note the Mac’s Memory Used, Swap Used, and memory-pressure graph while idle. Then unload the chosen model and check whether it is still listed:

ollama stop qwen2.5:7b
ollama ps
python3 measure_ollama.py
ollama ps

Substitute qwen2.5:3b in the stop command if appropriate. After ollama stop, wait briefly for memory to settle before recording the baseline. Watch Activity Monitor during generation if you can; immediately after the script, record Memory Used, Swap Used, and memory pressure while ollama ps still shows the model loaded. ollama ps can also help you check whether the model is running on GPU, CPU, or a mixture on your setup.

Subtracting the idle Memory Used reading from the loaded reading gives a rough whole-system change, not a precise allocation for the model. Other processes, caching, and unified memory affect it. Record a peak only if you actually observed one; otherwise call your second reading “loaded after requests.” If memory pressure stays high, swapping grows, or the system becomes unresponsive, stop the model and try the smaller choice rather than forcing the 7B run.

If the API request fails, first check that Ollama is running and that ollama ps responds. If a request times out, note the failure rather than treating its partial run as a successful latency measurement.

Decide what the numbers and answers mean

Make a compact record for the chosen model: its name and reported quantization, Ollama version, the two Activity Monitor readings, and the script’s three sets of timings and answers. The first request’s load duration helps explain why it may be slower than the repeat. API total duration and your measured wall time have different boundaries, so they need not match exactly. For output speed, the script divides output-token count by output-generation time; it does not divide by the full request time.

Now inspect the text rather than letting speed stand in for quality:

• For each grounded answer, are 2024-03-14 and $129.00 correct, and are both attributed to sample-receipt.pdf, page 1? Note any changed fact or invented citation.
• For the missing-fact answer, does it decline to supply card digits that were never in the evidence? Note a guessed number as a failure even if the prose sounds confident.
• Did the repeated grounded request change meaningfully? Record the difference if it did.

This is a small diagnostic, not a benchmark proving that the model can safely answer questions about your documents. The evidence was supplied directly in the prompt; retrieval, citation verification, and protection against hostile document text have not been implemented yet. Conversely, an incorrect answer here is useful evidence when selecting a model or designing checks—it is not something the React client should quietly present as a verified fact.

What you have established

You have turned the architecture’s “local model” box into a running, quantization-verified Ollama model and tested three practical constraints: whether it fits comfortably in unified memory, how long cold and warm requests take, and whether it handles a supplied fact and a missing fact responsibly. Keep the model tag, settings, observations, and failures together; they will give you a baseline when the assistant begins using real retrieval.

Next lesson, you’ll put a minimal FastAPI endpoint between the React client and Ollama and consume a streamed response. The model will remain local, but the API will become the controlled boundary through which the application requests its output.