Grounding LLM Responses with Retrieval-Augmented Generation

Hello! We’ll start with the idea that makes the rest of your RAG project intelligible: an LLM can answer using relevant material supplied from outside its training data. In your case, that material will come from Markdown and Python files. This first module builds the end-to-end mental model; later lessons will examine each stage and its Python implementation.

By the end of this lesson, you should be able to explain what it means for an answer to be grounded, follow a question from source material to answer, and recognize why retrieval helps without guaranteeing correctness. No APIs or advanced Python are needed yet.

---

Why give an LLM source material?

Imagine asking an assistant, “What does our project’s refresh_index function do?” A standalone LLM may know common meanings of “refresh” and “index,” but it has not necessarily seen your refresh_index function. It could produce a plausible description that does not match your code.

Retrieval-augmented generation, or RAG, changes what is available when the answer is written. The application finds relevant material in an external collection—such as your project’s Markdown documentation and Python source—and supplies selected excerpts alongside the question. The LLM then writes a response using that material.

Watch this short introduction before we trace a project-specific example.

The video describes the idea as if the LLM “reaches out” to a repository. For your project, a more precise mental model is that the surrounding application performs retrieval and passes the results to the LLM. The model does not automatically browse your files or know everything in the repository.

Oracle’s introduction places the same idea in an enterprise setting:

A useful distinction from the reading is where the information lives. The model’s training has already happened; your project files exist outside it. RAG makes selected file content available for this question. Adding or editing a file does not, by itself, update the model. Nor does it update a searchable RAG collection unless the application has processed that change—a workflow we will examine later.

---

What “grounded” actually means

An answer is grounded in source material when its factual claims can be supported by the relevant source passages supplied for that answer. That is stronger than merely saying, “We searched the files first.” Search might find an irrelevant passage, and a model might still add unsupported details.

For one question, think of the work in three parts:

1. Retrieve: Find excerpts likely to contain the answer.
2. Augment: Present those excerpts to the LLM together with the user’s question and instructions about how to use them.
3. Generate: Write an answer whose claims stay within what the excerpts support.

This is why RAG has that name. The retrieved material augments the input to generation; it does not become permanent knowledge inside the model.

In practice, the retrieved material might be two short excerpts rather than two entire files. The LLM needs enough context to answer, but passing every Markdown and Python file on every question would often be impractical and would bury useful evidence among irrelevant text. For now, treat “find the right excerpts” as a distinct job. Later lessons cover how files become searchable chunks and how those chunks are selected.

Citations make grounding inspectable. An answer might identify a documentation path and line range for a claim, allowing someone to check it. But a citation is useful only if the cited passage actually supports the nearby claim. A source label attached to an invented statement does not make the statement true.

---

A question about your files, traced carefully

Consider a fictional project with these two source excerpts. The paths and line numbers are illustrative, not taken from a real repository.

In docs/index.md, lines 8–9:

8  ## Updating the index
9  After editing a Markdown or Python file, run refresh_index
   with its path to update the searchable index.

In src/refresh.py, lines 20–24:

20  def refresh_index(paths):
21      selected = [path for path in paths if needs_update(path)]
22      for path in selected:
23          index_file(path)
24      return len(selected)

Suppose the user asks: “What should I do after editing a file, and what does refresh_index do?”

The application searches its prepared source collection and selects these excerpts. It then gives the LLM the question, the excerpts, and instructions. Conceptually, the input could look like this:

Instruction:
Answer using the supplied source excerpts. Cite the excerpt supporting
each factual claim. If they do not establish an answer, say so.

Question:
What should I do after editing a file, and what does refresh_index do?

[1] docs/index.md, lines 8–9:
After editing a Markdown or Python file, run refresh_index with its path
to update the searchable index.

[2] src/refresh.py, lines 20–24:
def refresh_index(paths):
    selected = [path for path in paths if needs_update(path)]
    for path in selected:
        index_file(path)
    return len(selected)

A grounded answer would be:

After editing a Markdown or Python file, the documentation says to run refresh_index with its path [1]. For the paths passed to it, the shown function selects those for which needs_update is true, calls index_file for each selected path, and returns the number selected [2].

Notice how each part can be checked. The instruction to run the function comes from the Markdown file. The description of what the shown function does comes from the Python excerpt. The labels [1] and [2] are useful because the application can associate them with their source paths and line ranges.

Now consider what the excerpts do not establish. They do not say whether refresh_index runs automatically, how needs_update makes its decision, or whether index_file always succeeds. An answer that confidently supplied those details would go beyond the available evidence—even if it cited [1] or [2]. A careful answer either leaves those details out or says that the shown sources do not resolve them.

This gives you a practical test for grounding: Could a reviewer point to the cited source text that supports each substantive claim? If not, the answer needs correction, better evidence, or an explicit statement of uncertainty.

---

Why retrieval helps—and why it can still fail

RAG addresses a specific problem: the LLM’s general training is not a dependable record of your current project files. Supplying relevant excerpts lets the answer reflect information that is private, project-specific, or newer than the model’s training. It can also give readers a route back to the evidence.

Yet the presence of retrieval does not certify the result. There are three separate opportunities for error:

• The source collection may be incomplete or stale. If refresh.py changed but the searchable collection still contains an older version, the answer may faithfully describe the old code.
• Retrieval may select the wrong material. A search could return a similarly named function from another module while missing the one the user meant.
• Generation may misuse good material. Even with the correct excerpt, the LLM might infer undocumented behavior, omit a qualification, or attach a citation that does not support its claim.

These are different failures. If the right text never reaches the model, asking the model to be more careful cannot supply the missing fact. If the right text does reach it but the answer overstates what it says, the generation and citation behavior need attention. This separation will matter when you debug your project.

There is also a responsible outcome when the available excerpts do not answer the question: say that the evidence is insufficient. For example, if the user asks whether needs_update checks a content hash, neither excerpt above answers that. “The supplied sources do not show how needs_update decides” is more useful than a confident guess.

---

Key takeaways

RAG lets an application retrieve external source material and supply it to an LLM when answering a question. In your project, the external material will be Markdown and Python files. Grounding means keeping the answer’s claims within what the retrieved passages support; citations help a reader verify that relationship. Retrieval can reduce unsupported answers, but source freshness, retrieval quality, and faithful generation all still matter.

Next, we’ll distinguish RAG from a standalone LLM, fine-tuning, and ordinary document search—approaches that can look similar from a user’s perspective but solve different parts of the problem.