Understanding RAG: Solving Knowledge Gaps and Hallucinations

Hello! Welcome to the first lesson in our module on giving agents a knowledge base.

In previous modules, we've discussed how to think about and design agentic systems. We've identified the key failure modes of LLMs and explored how to structure the information environment an agent needs to succeed. A recurring theme has been the challenge of providing agents with up-to-date, specific, and reliable knowledge.

This lesson tackles that challenge head-on. Our goal is to explain the retrieval-augmented generation (RAG) pattern and why it solves the critical problems of knowledge cutoff and hallucination. RAG is one of the most important architectural patterns in modern AI systems. Understanding it is fundamental to building the kind of robust, evidence-based agentic workflows you aim to deliver to clients.

Let's begin by examining the core problems that necessitate a solution like RAG.

1. The "Closed-Book Exam" Problem: Core LLM Limitations

A standalone Large Language Model, even a very powerful one, can be thought of as a student taking a "closed-book exam." Its knowledge is confined to what it learned during its training period—its "parametric memory." This leads to several significant limitations, especially in a business context where accuracy and currency are paramount.

To explore these issues, please watch the following segment from a Stanford lecture on Retrieval Augmented Language Models.

As the video highlights, standard LLMs struggle with:

• Knowledge Cutoff: The model's knowledge is static and becomes outdated. It knows nothing about events, data, or documents created after its training data was collected.
• Hallucination: The model can invent facts, sources, or details with high confidence, which is unacceptable for reliable business processes.
• Lack of Attribution: It's difficult to trace why the model generated a specific piece of information, making it impossible to verify its sources. This is a critical failure for any system that requires auditability.
• Difficulty of Updating: Modifying the model's internal knowledge to correct errors or add new information is a complex, expensive, and often impractical process.

You might wonder if we can't just solve these problems by continuously re-training the model or by simply providing all necessary information in the prompt. The next video explains why these seemingly straightforward approaches are not viable solutions.

These videos establish a clear set of challenges. We need a way to provide an LLM with current, relevant, and verifiable knowledge at the moment it's needed, without the costs and risks of retraining and without overwhelming it with irrelevant context. This is precisely the problem RAG was designed to solve.

2. The Solution: Retrieval-Augmented Generation (RAG)

RAG transforms the "closed-book exam" into an "open-book exam." Instead of relying solely on its pre-existing parametric memory, the LLM is given access to an external, up-to-date body of knowledge—a non-parametric memory—and is tasked with using that information to formulate its answer.

This approach was formally introduced in a 2020 paper from researchers at Facebook AI (now Meta AI). To get a precise, academic definition, please review the introduction of this foundational paper.

The core RAG pattern consists of three steps: Retrieve, Augment, Generate.

Here's a breakdown of the process:

1. Retrieve: When the system receives a query, it doesn't immediately pass it to the LLM. Instead, it first uses the query to search over an external knowledge base (e.g., a collection of company documents, product manuals, a customer support database, or even the live web). This search is performed by a component called a retriever. The goal is to find a small number of text snippets that are highly relevant to the user's query.

2. Augment: The relevant snippets of text retrieved in the previous step are then combined with the original user query to form a new, much more detailed prompt. This is the "augmentation" step. The prompt now contains not just the user's question but also the factual information needed to answer it.

3. Generate: Finally, this augmented prompt is passed to the LLM. The LLM is instructed to synthesize an answer to the user's query based on the provided contextual information.

The following video segment walks through this three-step process.

3. Why RAG is Effective

Now we can connect this architecture directly back to the problems we identified at the start.

Solving the Knowledge Cutoff Problem

The RAG pattern elegantly solves the knowledge cutoff issue by separating the agent's reasoning capability (the LLM) from its knowledge base (the retrievable documents).

• Knowledge can be updated continuously. To give the agent new knowledge, you don't need to retrain the multi-billion parameter LLM. You simply add, update, or remove documents from the external knowledge base. This process is fast, cheap, and can be done in real-time.
• Knowledge can be "hot-swapped." You can have multiple, distinct knowledge bases and point the agent to the relevant one for a specific task. An agent helping with HR queries can be pointed to a knowledge base of HR policies, while one helping with technical support can be pointed to a database of product documentation.

The authors of the original RAG paper demonstrated this powerfully. Take a look at their "Index hot-swapping" experiment.

Solving the Hallucination Problem

RAG mitigates hallucinations by grounding the LLM's output in retrieved evidence. Instead of being asked to recall a fact from its vast, opaque parametric memory, the LLM is asked to perform a more constrained task: synthesize an answer based on a small, relevant set of source texts.

• Evidence-based Generation: The model is explicitly prompted to use the provided context. A well-designed RAG system will instruct the LLM not to use outside knowledge and to state when the answer cannot be found in the provided text.
• Enables Attribution: Because the answer is generated from a specific set of retrieved documents, you can cite your sources. This is a game-changer for building trust with users and for creating auditable systems. If the RAG system provides an answer, it can also provide the link(s) to the source document(s) it used.

The following discussion provides a more nuanced take on hallucination and how RAG offers a way to control it.

This ability to produce factual, attributable generations is a key finding of the original RAG research and a primary reason for its widespread adoption. The authors note that human evaluators found RAG's generations to be significantly more factual than a state-of-the-art parametric-only model. This is mentioned in Sections 4.3 and the "Broader Impact" section of the paper (LINK).

Conclusion

In this lesson, we've established the conceptual foundation of Retrieval-Augmented Generation. You now have the architectural understanding to see why this pattern is so critical for building agents that are knowledgeable, trustworthy, and adaptable.

Here are the key takeaways:

• Core Problem: Standalone LLMs are like "closed-book" reasoners; their knowledge is static (knowledge cutoff), they can invent facts (hallucinate), and their outputs are unverifiable (no attribution).
• RAG as a Solution: RAG is an "open-book" architectural pattern that pairs a generator (the LLM's parametric memory) with a retriever that accesses an external knowledge base (non-parametric memory).
• Mechanism: The process involves (1) Retrieving relevant documents based on a query, (2) Augmenting the query with this information, and (3) Generating a response that is grounded in the retrieved evidence.
• Key Benefits: This architecture directly solves the knowledge cutoff problem by allowing the knowledge base to be updated independently of the LLM. It reduces hallucinations and enables attribution by forcing the model to base its answers on verifiable sources.

You've now answered the "why" and "what" of RAG. In our next lesson, we will dive into the "how" of the retrieval step, which is the heart of any RAG system. We will analyze the impact of different embedding models and distance metrics on retrieval performance and cost, which are core design decisions you'll make when architecting an agentic workflow for a client.