Building a RAG Pipeline for External Knowledge

Introduction

In our last two lessons, we established the foundational components for building intelligent agents. First, we explored how text embeddings convert unstructured text into numerical vectors, enabling semantic search. Second, we analyzed the LLM's context window, recognizing it not as an infinite buffer but as a finite, performance-sensitive workspace that requires disciplined management—a practice we called context engineering.

We identified Retrieval-Augmented Generation (RAG) as a primary "selection" strategy within context engineering. It is the mechanism by which an agent can pull in precise, relevant information from vast external knowledge bases just in time to answer a query.

Today, we move from theory to architecture. The learning outcome for this lesson is to design a Retrieval-Augmented Generation (RAG) pipeline to provide agents with external knowledge. We will assemble the concepts of embeddings and context management into a coherent, end-to-end system. We'll start with a basic workflow and then progressively add the layers of sophistication required for a production-grade system suitable for demanding environments like financial services.

The Basic RAG Pipeline

At its core, RAG is a simple and powerful idea. Before an LLM answers a question, we first retrieve relevant information from a knowledge base and provide it to the model as context. This grounds the model's response in factual, up-to-date, or proprietary data.

The process consists of three steps: Retrieve, Augment, Generate.

The entire pipeline can be broken down into these two distinct phases:

1. Offline Phase: Data Ingestion and Indexing
   This is the preparatory work done before any user asks a question. The goal is to create a searchable knowledge base.

   • Load: Your source documents (e.g., underwriting policies, financial reports, regulatory filings in PDF format) are loaded.
   • Chunk: These documents are broken down into smaller, manageable pieces, or "chunks."
   • Embed: Each chunk is passed through an embedding model (as discussed in Lesson 1) to create a numerical vector.
   • Store: The chunks and their corresponding embedding vectors are stored in a specialized vector database.

2. Online Phase: Retrieval and Generation
   This phase happens in real-time when a user submits a query.

   • Embed Query: The user's query is converted into an embedding vector using the same model from the offline phase.
   • Retrieve: This query vector is used to search the vector database for the document chunks with the most similar embeddings (i.e., the most semantically relevant information).
   • Augment: The retrieved chunks are combined with the original user query into a structured prompt.
   • Generate: This augmented prompt is sent to the LLM, which generates a response grounded in the provided context.

To solidify these fundamentals and understand where RAG fits in the broader landscape of model customization, please watch the following video.

This basic pipeline is sufficient for simple demos. However, for enterprise applications in finance, where data is complex and accuracy is non-negotiable, this approach has critical weaknesses.

Designing a Production-Grade RAG Pipeline

The simple RAG model often fails in production because real-world data is messy (tables, headers, multiple versions) and user queries can be complex, requiring multi-step reasoning. A production system requires a more robust and sophisticated architecture.

The following video provides an excellent architectural blueprint for such a system.

Let's break down the components of this production architecture, referencing both the video and other enterprise-focused materials.

Stage 1: The Enterprise Ingestion Pipeline

The quality of your agent's knowledge base is determined entirely by the quality of your ingestion pipeline. As noted in the Informatica article "RAG Data Ingestion," traditional ETL (Extract, Transform, Load) pipelines are insufficient for RAG because they often strip the semantic context that is vital for accurate retrieval.

Building on this, a production ingestion pipeline involves several intelligent processing steps.

To summarize and expand on these critical points:

• Restructuring and Structure-Aware Chunking: Instead of splitting a document every N characters, a production pipeline first parses the document's structure (headings, paragraphs, tables). Chunking then respects these boundaries. For a financial report, this means a table of figures remains a single, coherent chunk, rather than being sliced into meaningless rows.
• Metadata Augmentation: For each chunk, you don't just store the text. You generate and store rich metadata, such as the document source, creation date, section title, and even AI-generated summaries or a list of questions the chunk can answer. This is invaluable for the next stage.
• Hybrid Storage: A simple vector store is not enough. You need a database that can handle both vector search and structured metadata filtering. For example, in an underwriting scenario, you might need to find information relevant to "commercial property loans" (semantic search) but only from policy documents updated in the last quarter (metadata filter). PostgreSQL with the pgvector extension is a common example of a system that supports this hybrid approach.

Stage 2: Advanced Retrieval and Reasoning

Once the data is ingested, the online query process also needs to be more sophisticated than a single vector search.

Key enhancements at this stage include:

• Hybrid Search: This combines semantic (vector) search with traditional keyword search. Semantic search is excellent for finding conceptually similar information ("work from home policy"), while keyword search is essential for exact matches on specific identifiers like a policy number (Form-A-1138), a client ID, or a legal term.
• Re-ranking: A common pattern is to retrieve a larger number of candidate chunks (e.g., 20) and then use a smaller, more specialized "re-ranker" model to score and select the top 3-5 most relevant chunks to pass to the main LLM. This improves signal-to-noise ratio in the final context.
• Reasoning and Orchestration: For complex queries like, "Compare the capital expenditure guidance for Company A and Company B for next year," a simple retrieval won't suffice. A "reasoning engine" or "orchestrator" first breaks the query into sub-problems:
  1. Find CapEx guidance for Company A.
  2. Find CapEx guidance for Company B.
  3. Synthesize the findings and generate a comparison.
     This is where RAG starts to become truly "agentic."

A Concrete Blueprint: RAG for Financial Analysis

To make this architecture tangible, let's examine a project that builds a RAG agent for analyzing financial conference calls—a use case highly relevant to your goals.

This project provides an excellent, modular implementation pattern:

• ingest.py & retriever.py: These handle the entire Ingestion Pipeline. They load PDFs, chunk them (RecursiveCharacterTextSplitter), create embeddings (HuggingFaceEmbeddings), and store them in a vector store (FAISS).
• prompts.py: This is the Augmentation step. It contains the master prompt template that instructs the LLM on how to behave and where to place the retrieved context and user query.
• websearch.py: This module implements a Tool Use / Fallback mechanism. If the answer isn't in the local documents, the agent can perform a targeted web search, making the system more robust.
• chatbot.py: This is the Orchestration Layer. It receives the query, calls the retriever, decides whether to fall back to web search, constructs the final prompt using the template, and invokes the LLM.
• Logging (log_interaction): Note the explicit inclusion of logging. For any financial application, creating a robust audit trail of queries, retrieved context, and final outputs is a critical governance requirement.

This modular structure is not just good software engineering; it is the practical embodiment of the architectural principles required to build a reliable and maintainable agentic system.

Conclusion

In this lesson, we have designed the complete architecture for a Retrieval-Augmented Generation pipeline, moving from a basic concept to a production-ready blueprint.

Key Takeaways:

• A RAG pipeline consists of two main stages: an offline ingestion phase to build a knowledge base and an online inference phase to answer queries.
• Production-grade RAG systems require more than a simple retrieve-augment-generate loop. Key enhancements include structure-aware chunking, metadata augmentation, and hybrid databases for storage.
• Advanced retrieval techniques like hybrid search (semantic + keyword) and re-ranking are crucial for improving the quality of the context provided to the LLM.
• For complex queries, a reasoning/orchestration layer is needed to break down the problem, execute multiple retrievals, and even use external tools (like web search) as a fallback.
• Designing with a modular architecture (as seen in the financial agent example) is a robust and practical approach to implementing these complex pipelines.

Preview of the Next Lesson:

We have now explored in depth how to provide an agent with external knowledge via RAG. This is one of three primary methods for modifying model behavior. In our next and final lesson of this module, we will "Differentiate between prompting, RAG, and fine-tuning for modifying model behavior." This will provide you with a strategic framework for deciding which technique—or combination of techniques—is appropriate for a given problem, completing your foundational toolkit for agent design.