Tracing a Document Q&A Request End to End

Hello! We’ll build a document question-answering application in stages: first the direct Python workflow, then the real model providers and retrieval components, and eventually a web interface and a LangChain version. Today’s goal is to see the whole request path before implementing its individual parts.

We’ll start with a small runnable trace. It deliberately uses a simple text matcher and a placeholder instead of an embedding index and an LLM. That lets you inspect what each stage receives and produces without mistaking a framework call for the entire application.

Run the request path once

Save this as trace_rag.py and run it with python trace_rag.py. It needs only the Python standard library.

import re

DOCUMENT = {
    "document_id": "demo-policy",
    "text": (
        "Returns:\n"
        "Returns are accepted within 30 days of purchase. Keep the receipt.\n\n"
        "Shipping:\n"
        "Standard delivery takes 3 to 5 business days."
    ),
}


def ingest(document):
    chunks = []
    for number, paragraph in enumerate(document["text"].split("\n\n"), start=1):
        section, text = paragraph.split("\n", maxsplit=1)
        chunks.append({
            "source_id": f"{document['document_id']}#{number}",
            "section": section.rstrip(":"),
            "text": text,
        })
    return chunks


def words(text):
    return set(re.findall(r"[a-z0-9]+", text.lower()))


def retrieve(question, chunks):
    query_words = words(question) - {"what", "is", "the", "a", "for"}
    scored = [
        (len(query_words & words(chunk["text"])), chunk)
        for chunk in chunks
    ]
    ranked = sorted(scored, key=lambda item: -item[0])
    return [
        {"score": score, **chunk}
        for score, chunk in ranked[:1]
        if score > 0
    ]


def build_messages(question, hits):
    excerpts = "\n\n".join(
        f"[{hit['source_id']}] {hit['text']}" for hit in hits
    ) or "(no excerpts retrieved)"

    return [
        {
            "role": "system",
            "content": (
                "Answer using only the supplied excerpts. Cite the source ID. "
                "If the excerpts do not support an answer, say there is "
                "insufficient evidence."
            ),
        },
        {
            "role": "user",
            "content": f"Excerpts:\n{excerpts}\n\nQuestion: {question}",
        },
    ]


def generate_stub(_messages, hits):
    if not hits:
        return "Insufficient evidence in the document."
    hit = hits[0]
    return (
        "Demo excerpt, not a model-generated answer: "
        f"{hit['text']} [{hit['source_id']}]"
    )


if __name__ == "__main__":
    chunks = ingest(DOCUMENT)
    print("INGESTED CHUNKS:", chunks)

    question = input("Ask about the document: ").strip()
    hits = retrieve(question, chunks)
    print("RETRIEVAL RESULT:", hits)

    messages = build_messages(question, hits)
    print("MODEL MESSAGES:", messages)

    answer = generate_stub(messages, hits)
    print("USER INTERFACE:", answer)

Enter “What is the returns window?” The retrieval result should contain the returns excerpt and its demo-policy#1 source ID. Then try “What is the warranty period?” With no matching excerpt, the placeholder reports insufficient evidence.

The printed stage labels are developer instrumentation; a finished interface would normally show the question, answer, and sources—not the entire model prompt. The script is also intentionally incomplete: word overlap is not semantic search, its chunks live only in memory, and generate_stub merely displays an excerpt. Those are precisely the components we’ll replace as we build the project.

What happened before—and after—the question?

A document question-answering system has two different timescales. Ingestion prepares a document when it is supplied or changed. Question answering uses that preparation each time a user asks something. Our script rebuilds its tiny in-memory collection whenever it starts; a persistent application would store the prepared data for later requests.

Watch this short demonstration with those two timescales in mind:

The script gives you a concrete counterpart to each part of the demonstration:

1. Ingestion: ingest splits the supplied text into two chunks. Each chunk keeps its text, section, and a source ID. With PDFs, extracting readable text and page metadata would come first. Later, we’ll embed chunks and store them in a persistent index.
2. Retrieval: retrieve scores the chunks against the question and selects the best match. In the full application, the question and stored chunks will be represented with a compatible embedding model so retrieval can find related meaning even when the words differ. The retrieved text, not just its score, is what the next stage needs.
3. Prompt construction: build_messages places instructions in a system message and the selected excerpts plus question in a user message. The source ID travels with the excerpt so an answer can refer back to it.
4. Generation: generate_stub marks the boundary where a Groq, Mistral, or Ollama call will go. A real generating model receives the messages and returns answer text; it does not automatically search your documents.
5. User interface: the final print stands in for the answer display. Later, a web interface will accept a document and question, call the backend, and display the answer with its source.

For a broader architecture view, read the short AWS guide:

“Retrieval-augmented generation,” or RAG, names the combination: retrieve relevant external material, add it to the model’s input, and generate an answer. It is not a procedure for retraining the generating model on every uploaded file.

Trace the boundaries, not just the functions

The most useful way to debug this application is to ask what crosses each boundary. After ingestion, a chunk needs its text and source identity as well as anything used for searching it. If you keep only an embedding, you may locate a relevant vector but lack the excerpt needed to construct the prompt. If you keep text but lose its source ID, the interface cannot reliably point back to the document.

Retrieval then returns candidates, not proven answers. Our demo’s score means “number of overlapping words”; a future vector similarity score means something different, but neither guarantees that a passage answers the question. For example, a passage mentioning “returns” could discuss an exception rather than the requested time limit. The prompt must make the evidence visible to the model, and the application must eventually check whether the answer and its citations are supported. Those checks come later in the course.

The model boundary is equally important. The messages list is a request payload: an instruction about how to answer, followed by the question and selected evidence. Changing the generating provider should not require re-ingesting the document or changing the user interface. Conversely, switching a generation provider does not by itself change which chunks were retrieved. The embedding model used for search and the LLM used for writing the answer have different jobs.

Finally, trace the answer back outward. A useful interface should distinguish answer text from source references and from errors or insufficient-evidence responses. Printing demo-policy#1 beside our placeholder shows where a citation might appear, but it is not citation validation: the stub simply echoes the retrieved ID. In the finished application, the backend will check source references before the interface displays them.

Where this leaves the project

You have now followed one question through document preparation, evidence selection, message construction, a generation boundary, and an interface boundary. You have also seen the central design constraint: preserve the evidence and its identity throughout the request, rather than treating the LLM as if it knows or searches the uploaded document on its own.

Next, we’ll set up the reproducible Python project and environment-based configuration. That foundation will let us replace the generation placeholder with real providers without putting API credentials in the codebase.