Retrieval-Augmented Generation (RAG) Architecture Overview

The Motivation for RAG

Before we dive into the architecture, let's solidify our understanding of why RAG is such a critical pattern in modern AI engineering. As you know, large language models, despite their power, have inherent limitations. They can "hallucinate" when uncertain, their knowledge is frozen at the time of training, and they have no access to your private, domain-specific data.

Directly fine-tuning an LLM to incorporate new knowledge is one possible solution, but as you can imagine, it's a computationally expensive and slow process, ill-suited for rapidly changing information. RAG offers a more flexible and cost-effective alternative.

The following video provides an excellent overview of these challenges and introduces how RAG is designed to overcome them.

At its heart, RAG decouples the knowledge base from the LLM's internal parameters. Instead of trying to bake knowledge into the model itself, we provide it with relevant information on-the-fly, at the moment it's needed. This is the essence of retrieval-augmented generation: Answer = LLM( Query + Retrieved_Knowledge ).

The Two Halves of a RAG System: Indexing and Querying

From a system design perspective, a RAG application can be split into two primary processes: an Indexing Process and a Query Process.

1. Indexing Process (Offline/Build-Time): This is the preparatory stage where you build your knowledge base. You take your collection of documents, process them into retrievable chunks, and store their vector embeddings in a database. This is typically done offline and can be re-run whenever your source documents change.
2. Query Process (Online/Run-Time): This is the real-time stage that happens whenever a user submits a query. The system retrieves relevant information from the indexed knowledge base, combines it with the user's query to create an augmented prompt, and sends it to the LLM to generate a response.

This separation is fundamental. The heavy lifting of processing and embedding your entire document library is done ahead of time, ensuring that the run-time query process is as fast and efficient as possible.

The following diagram clearly illustrates this two-part structure.

A Step-by-Step Architectural Breakdown

Now, let's zoom in and examine the specific stages within these two processes. A standard RAG workflow can be broken down into four key stages: Ingestion, Retrieval, Augmentation, and Generation.

The following reading from a Microsoft developer resource provides a concise, high-level overview of these four stages.

Having established this four-stage framework, let's dive deeper into the engineering details of each step, connecting them to the concepts you've learned in previous modules.

Stage 1: Ingestion (The Indexing Process)

This stage corresponds to the "Index Process" and involves everything needed to create your searchable knowledge base. It's a pipeline of its own, often called the "data injection pipeline."

• Document Loading: The process begins by loading your raw data from various sources (e.g., PDFs, text files, web pages, database records).
• Preprocessing & Chunking: The raw text is cleaned and then split into smaller, semantically meaningful chunks. As you learned in Module 3, the chunking strategy (e.g., fixed-size, recursive) is a critical decision that directly impacts retrieval quality.
• Embedding Generation: Each chunk is passed through an embedding model (like the ones you used in Module 2) to convert its semantic meaning into a numerical vector.
• Vector Store Indexing: Finally, these embeddings, along with the original text chunks and any associated metadata, are loaded into a vector store (like ChromaDB from Module 4). The store uses an ANN algorithm (like HNSW) to build an efficient index for fast searching.

Stages 2, 3 & 4: Retrieval, Augmentation, & Generation (The Query Process)

This is the real-time pipeline that executes when a user interacts with the system.

• Query Processing: The user's query, which is a string of text, is first converted into a vector embedding using the same embedding model that was used during ingestion.
• Retrieval: This vector is used to search the vector store. The store performs a similarity search to find the top-k most similar document chunks from the knowledge base. This is the "Retrieval" in RAG. This step may also involve applying metadata filters, as you practiced in Module 4.
• Augmentation: The retrieved chunks of text (the "context") are then combined with the original user query. This is typically done by formatting them into a carefully constructed prompt. The prompt instructs the LLM to use the provided context to answer the user's question. This is the "Augmentation" step.
• Generation: The final augmented prompt is sent to the LLM. The model then generates a response that is "grounded" in the retrieved information, leading to a more factual and relevant answer. This is the "Generation" part of RAG.
• (Optional) Post-Processing: The LLM's raw output can be further processed to format citations, check for safety issues, or structure the output (e.g., into JSON).

The following article offers a fantastic, engineering-focused deep dive that aligns perfectly with our breakdown.

To see all these pieces in motion, the following video segment provides a clear whiteboard walkthrough of the entire end-to-end flow.

From Naive to Advanced: Types of RAG

The architecture we have just described is often called Naive RAG. It represents the simplest, most direct implementation of the pattern. While powerful, it serves as a baseline. As you move toward more complex, production-grade systems, you'll encounter more sophisticated architectures. It's useful to be aware of this landscape.

In Module 7, we will explore several techniques that fall under Advanced RAG, such as query rewriting and re-ranking, to further improve the quality of your system. For now, our focus in this module will be on mastering the implementation of the core, or "naive," RAG pipeline.

Conclusion

In this lesson, you've assembled all the individual concepts from the previous modules into a complete architectural blueprint for a RAG system. You now have a mental model of how data flows through the system, from raw documents to a contextually-aware AI response.

Here are the key takeaways:

• RAG's Purpose: To overcome LLM limitations by grounding them in external, up-to-date, or private knowledge without expensive retraining.
• The Two Core Processes: RAG architecture is split into an offline Indexing Process (preparing the knowledge base) and an online Query Process (answering user questions).
• The Four Key Stages: The end-to-end workflow consists of Ingestion (loading, chunking, embedding, indexing), Retrieval (searching the vector store), Augmentation (building the prompt with context), and Generation (getting the final answer from the LLM).
• Architectural Spectrum: The architecture we've described is a "Naive RAG," the foundational pattern upon which more advanced and modular systems are built.

You now have the complete map. In our next lesson, we will take the first step in building this system: you will implement the retrieval step that fetches relevant context for a user's query, putting your vector store knowledge into practice.