Building a RAG Knowledge Base: What to Include and Exclude

Hello! Welcome back to our module on giving agents a knowledge base.

In our last lesson, we explored the technical core of retrieval, analyzing how embedding models and distance metrics work together to translate text into a searchable vector space. We focused on the trade-offs between performance, cost, and accuracy at the vector level.

Now, we move from the how of retrieval to the more strategic question of what. Before you can embed anything, you must first decide what information to make available. This is not a simple data dump; it's a critical act of information architecture. The learning outcome for this lesson is to determine what business knowledge to include in a RAG knowledge base and what to exclude.

For your goal of delivering agentic AI solutions, designing the knowledge corpus is a foundational part of the service. It defines the boundaries of your agent's expertise and directly impacts its reliability, cost, and safety. Getting this right is the difference between an agent that is a genuine asset and one that is a source of noise and risk.

1. The Perils of an Uncurated Knowledge Base

It might seem intuitive to provide an agent with all available data. However, as with any information system, the principle of "garbage in, garbage out" holds true. More data is not always better. Including irrelevant, outdated, or low-quality information can actively degrade performance.

This performance degradation happens for a few reasons:

• Noise: Irrelevant documents can be retrieved alongside relevant ones, confusing the LLM and leading to less precise answers.
• Contradictions: Outdated or conflicting information sources can cause the LLM to hallucinate or produce hedged, unconfident responses.
• Cost: Embedding and storing vast amounts of unnecessary data incurs significant storage and computation costs, a key concern when designing client solutions.

Therefore, the first step in building a RAG system is intentional data curation. We need a framework for deciding what makes the cut.

2. A Framework for Knowledge Curation

We can approach the decision of what to include or exclude using a multi-faceted framework. Think of it as a set of gates that any piece of information must pass through before being admitted into the agent's "second brain."

The core idea is to move from a simple data repository to a structured, curated knowledge model.

Gate 1: Relevance and Utility

The most fundamental criterion is relevance. Does this information directly help the agent perform its intended function? To answer this, it's useful to apply the "agent-as-employee" mental model we discussed earlier. What documents, data, and institutional knowledge would you give a new human employee hired for this exact role?

A powerful way to manage this is through a "Domain-Driven" approach, where experts in a specific business area own the curation of knowledge for that domain.

The key insight here is that relevance is context-dependent. The information needed by an agent that explains product SLAs is different from one that describes system architecture. Distributing ownership to domain experts ensures that the knowledge base for each agent (or group of agents) is dense with high-signal, relevant information.

Gate 2: Quality and Trustworthiness

Once a document is deemed relevant, you must assess its quality. Including inaccurate, poorly written, or contradictory information is a direct path to unreliable agent behavior.

Establishing formal data governance practices is essential for ensuring the quality of your knowledge base.

For a client project, implementing such practices is a mark of a production-ready system. This could involve:

• Source Vetting: Prioritizing official documentation, wikis, and databases over informal sources like chat logs or old emails.
• Version Control: Ensuring that only the latest, approved versions of documents are included.
• Content Review: Having domain experts periodically review and attest to the accuracy of the information in the knowledge base.

Gate 3: Structure and Format

The format of your source data significantly impacts retrieval quality. While RAG can handle unstructured text, performance improves when the data is clean and well-structured.

This goes beyond simple text extraction. It involves transforming data into "LLM-readable" formats.

For a more advanced example of structuring complex knowledge, consider technical diagrams. A diagram contains rich information about system architecture or processes that is lost in a simple text dump. A sophisticated approach involves creating a textual representation of the diagram.

This technique of creating "synthetic" documentation from structured sources is a powerful tool for your arsenal, allowing you to include knowledge that isn't already in a prose format.

Gate 4: Sensitivity and Risk (What to Exclude)

Exclusion is just as important as inclusion. A primary task in knowledge base design is to identify and filter out data that poses a legal, ethical, or business risk.

To formalize this, we can again turn to the data governance framework.

This trade-off is a critical point of discussion with any client. Your role as the architect is not just to build the system, but to help the client navigate this risk-reward landscape. A highly restrictive policy may be appropriate for a public-facing chatbot, while a more inclusive one (with robust safeguards like data masking) might be necessary for an internal expert-assist tool.

3. The Result: A Curated Knowledge Model

By applying this four-gate framework, you move beyond a simple collection of files to a well-defined knowledge model. The "Domain-Driven RAG" article summarizes what this looks like in practice. It's not just the documents themselves, but a combination of components.

The inclusion of metadata is especially important. Metadata is data about your data—things like document source, creation date, version, author, or domain tags. As we'll see in later lessons on multi-agent systems, this metadata is invaluable for intelligently routing queries to the correct knowledge source or agent.

Conclusion

In this lesson, we established a strategic framework for designing the knowledge base of a RAG system. The goal is not to ingest everything, but to curate a high-quality, relevant, and safe corpus of information tailored to the agent's purpose.

Here are the key takeaways:

• Curation is Mandatory: Simply dumping all available data into a RAG system leads to noise, high costs, and reduced accuracy.
• Use a Multi-Gate Framework: Evaluate potential knowledge based on its relevance (is it useful for the task?), quality (is it accurate and trustworthy?), structure (is it in an LLM-friendly format?), and sensitivity (does it pose a risk?).
• Embrace Domain Ownership: Empowering subject-matter experts to curate knowledge for their domains is a scalable and effective strategy for ensuring relevance and quality.
• Exclusion is a Strategic Decision: Deciding what to exclude involves a critical trade-off between minimizing risk and maximizing the agent's utility. This requires careful consideration and discussion with stakeholders.
• Build a Knowledge Model: The final knowledge base is more than just documents; it's a model that includes curated content, rich metadata, and even structured data representations.

In our next lesson, we'll address the next step in the ingestion pipeline. Now that we've decided which documents to include, we'll learn how to select an appropriate chunking method and configure chunk size and overlap. This is a critical tuning step that determines the fundamental units of information your agent will retrieve.