Inspecting Groq Chat Completions: Messages, Text, Tokens, and Latency

Hello! Your project already has a reproducible Python environment and a configuration boundary that selects a provider without exposing its key. In the document question-answering workflow, generation is still a placeholder. Today you’ll replace that placeholder in a small standalone test with one real Groq request, then inspect exactly what you sent, what came back, how many tokens were used, and how long the call took.

Allow about 40 minutes. You’ll use a short fictional note rather than a real document: retrieval and evidence handling come later, and there is no reason to send private document text to a hosted API for this first test.

See the request shape before writing code

A chat completion takes a model ID and an ordered list of messages. Each message has a role and content. For this test, a system message sets a general instruction and a user message supplies the note and question. The response contains an assistant message; it does not simply return your input messages with an answer appended.

For a quick visual demonstration, watch these portions of “Getting Started with Groq API | Making Near Real Time Chatting with LLMs Possible” from Prompt Engineering. The recording uses an older model and a notebook workflow; keep the project setup and model ID shown below instead.

Next, read Groq’s “Groq Client Libraries” documentation. Its Python example is the basis for our SDK call; the sample response is a map of fields to inspect, not a prediction of the text or timings you will receive.

Send one non-streamed completion

From the doc-qa project created previously, add the SDK:

uv add groq

This updates both the declared dependency and uv.lock. In your local, ignored .env, select Groq and provide your own key:

APP_PROVIDER=groq
GROQ_API_KEY=your_actual_key_here

Leave .env.example free of real credentials. Because the previous lesson used load_dotenv(..., override=False), a GROQ_API_KEY or APP_PROVIDER already set in your shell takes precedence over .env.

Create groq_smoke.py in the project root. Keep main.py as the configuration-only check from the previous lesson.

# groq_smoke.py
from time import perf_counter

from groq import Groq

from config import load_settings


def main() -> None:
    settings = load_settings()
    if settings.provider != "groq":
        raise ValueError("Set APP_PROVIDER=groq to run this Groq test")

    model = "llama-3.3-70b-versatile"
    messages = [
        {
            "role": "system",
            "content": "You are a concise assistant. Answer in one sentence.",
        },
        {
            "role": "user",
            "content": (
                "Test note: The sample library opens at 9 a.m. on weekdays.\n"
                "Question: When does it open on weekdays?"
            ),
        },
    ]

    # Safe here because these are fixed, fictional test messages.
    print("Messages sent:")
    for message in messages:
        print(f"  {message['role']}: {message['content']}")

    client = Groq(api_key=settings.groq_api_key)
    started = perf_counter()
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        max_completion_tokens=96,
    )
    elapsed = perf_counter() - started

    if not response.choices:
        raise RuntimeError("Groq returned no completion choices")

    choice = response.choices[0]
    print(f"\nReturned model: {response.model}")
    print(f"Returned role: {choice.message.role}")
    print(f"Generated text: {choice.message.content}")
    print(f"Finish reason: {choice.finish_reason}")

    usage = response.usage
    if usage is None:
        print("Token usage and server timing were not returned.")
    else:
        print(f"Prompt tokens: {usage.prompt_tokens}")
        print(f"Completion tokens: {usage.completion_tokens}")
        print(f"Total tokens: {usage.total_tokens}")
        print(f"Server prompt time (s): {getattr(usage, 'prompt_time', None)}")
        print(
            "Server completion time (s): "
            f"{getattr(usage, 'completion_time', None)}"
        )
        print(f"Server total time (s): {getattr(usage, 'total_time', None)}")

    print(f"Measured SDK call time (s): {elapsed:.3f}")


if __name__ == "__main__":
    main()

Run it from the project root:

uv run --locked python groq_smoke.py

Groq’s documentation uses pip install groq; uv add groq serves the same dependency-installation purpose while also updating this project’s lockfile. The model ID above comes from the documented example. If the API reports that it is unavailable to your account, replace only the model ID with one currently available to you; do not change message structure to troubleshoot model access.

This is deliberately a non-streamed call: create(...) returns a completed response before the program prints its answer. That makes the first request and its usage straightforward to inspect. Streaming is a later lesson.

Read the request and response like an application developer

Start with the printed messages sent. They are the input you constructed locally, in order. The fictional note sits inside the user message; no PDF was ingested and no passage was retrieved. Although the answer should be something like “It opens at 9 a.m. on weekdays,” this test establishes API connectivity and response inspection, not that the application can yet find or verify evidence in documents.

Now compare those messages with the returned fields:

• response.choices[0].message.role should identify the generated message as assistant; .content holds its text. The first choice is appropriate here because this request asks for one completion.
• response.model tells you which model the API says handled the request. finish_reason tells you why generation ended; a length-related result would be a reason to inspect the completion limit before treating an answer as complete.
• usage.prompt_tokens counts input tokens and usage.completion_tokens counts generated tokens. usage.total_tokens reports overall usage. Tokens are not words: your system instruction, user note, question, and message formatting all contribute to input usage. These counts matter later when document chunks make prompts longer.

There are also two different timing perspectives in the printout. perf_counter() measures elapsed time around the SDK call as observed by your Python process. Groq’s sample response also includes prompt_time, completion_time, and total_time under usage; when present, these are provider-reported server timings in seconds. They need not match your measured call time: your measurement includes client-side and network waiting that the server’s generation timings do not necessarily include. Neither measurement in this non-streamed test is “time to first visible token.”

Do not infer a performance guarantee from one run. Response length, network conditions, and service load can change the numbers. For now, the useful result is that you can identify where each measurement came from.

Finally, notice what the script does not print: the API key. Printing the two fixed messages helps you debug this smoke test, but remove that prompt logging before accepting real user questions or document text.

If the request does not succeed

A missing-key error from load_settings() means the selected provider has no effective GROQ_API_KEY; check the local .env and whether your shell environment is overriding it. An authentication error after the request usually points to the credential itself. A model-not-found error calls for checking the model ID available to your account. A rate-limit error means the service declined the request for now; repeatedly rerunning immediately is not a useful diagnostic.

Keep these failures distinct from a successful request with an imperfect answer. The former are transport or configuration problems; the latter is a generation-quality problem. You will address both systematically as the application grows.

What you now have

You’ve made a real Groq chat-completion call through the Python SDK, inspected the exact messages you supplied, extracted the assistant’s text, and separated token usage and provider timing from Python-observed latency. The test remains intentionally small, so each part of the exchange is visible.

Next, you’ll send an equivalent request through Mistral and compare response fields. That comparison will reveal what the eventual common provider interface must normalize.