Building RAG with Vector Databases for Long-Term Memory

Part 1: The "Open-Book Exam" for LLMs - Understanding RAG

At its core, a Large Language Model (LLM) only "knows" what it learned during its training. It has no knowledge of your proprietary project data, recent market events, or specific deal terms. Retrieval-Augmented Generation (RAG) is a powerful technique that solves this problem.

Instead of just asking the LLM a question and hoping it knows the answer, the RAG process first retrieves relevant information from an external knowledge base and then provides this information to the LLM as context to generate a well-founded answer. Think of it as giving the LLM an open-book exam, where your knowledge base is the textbook.

The key components of this process are:

1. Indexing: Your documents (e.g., project finance PDFs, Word docs) are broken into smaller chunks. Each chunk is converted into a numerical representation called an embedding using a specialized model. These embeddings, which capture the semantic meaning of the text, are stored in a vector database.
2. Retrieval & Generation: When you ask a question, your query is also converted into an embedding. The vector database then performs a similarity search to find the document chunks with embeddings most similar to your query's embedding. These relevant chunks are retrieved and passed to the LLM along with your original question, giving it the specific context it needs to formulate a precise answer.

This workflow is visualized below.

To get a clearer picture of the underlying mechanics, let's turn to a great article from the CFA Institute.

Part 2: Building the Knowledge Base with a Vector Database

Now let's move to implementation. We will build a simple, local RAG system. The first step is to create the vector database that will serve as our agent's long-term memory. We'll use ChromaDB, a popular open-source vector database that is easy to set up and run locally.

The process involves these key steps:

1. Loading Data: Reading your source documents (for this example, a simple CSV).
2. Creating Document Objects: Structuring the data into a format LangChain can work with, including the main content and any associated metadata.
3. Embedding and Storing: Instantiating the vector store, which handles the embedding process and saves the resulting vectors to disk.

The following video provides a clear, step-by-step walkthrough of this entire process.

As you saw, the core logic involves preparing your text chunks and then using a command like Chroma.from_documents() to perform the embedding and storage in one go. The CFA Institute article you read earlier reinforces this pattern with a similar code snippet (LINK, Section 3).

Part 3: Retrieving Knowledge from the Vector Database

With our knowledge base created, we now need a way for our agent to query it. In LangChain, this is done using a Retriever. A retriever is an object that takes a query string as input and returns a list of relevant Document objects from the vector store.

Let's continue with the video to see how to create and use a retriever.

The retriever is the bridge between your long-term memory and your agent's reasoning process.

Part 4: Integrating RAG into an Agentic Workflow

Now we'll tie everything together. We will build a complete workflow that:

1. Takes a user's question.
2. Uses the retriever to find relevant context from our vector database.
3. Passes both the question and the context to an LLM to generate an answer.

The final part of the video demonstrates this integration.

Code Implementation

Here is a complete Python script that builds a RAG system for a simplified project finance use case. It sets up a vector store with information about a few sample renewable energy projects and then uses it to answer questions.

import os
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
from langchain_community.vectorstores import Chroma
from langchain_core.documents import Document

# Set your OpenAI API key
# os.environ["OPENAI_API_KEY"] = "your_api_key"

# 1. Define Sample Project Data
project_docs = [
    Document(
        page_content="Project 'Helios One' is a 50MW C&I solar project in Lagos, Nigeria. Its primary offtaker is a major manufacturing firm. The projected DSCR is 1.4x. The PPA term is 15 years.",
        metadata={"project_id": "H1", "country": "Nigeria", "technology": "Solar"},
    ),
    Document(
        page_content="Project 'Andes Wind' is a 120MW onshore wind farm in the Atacama region of Chile. It sells power to the spot market. The project relies on merchant pricing and has a high degree of revenue volatility.",
        metadata={"project_id": "AW1", "country": "Chile", "technology": "Wind"},
    ),
    Document(
        page_content="The 'Mekong Delta' project is a 30MW run-of-river hydro project in Vietnam. It has a 20-year PPA with the state-owned utility, EVN. The hydrology risk is considered the primary technical challenge.",
        metadata={"project_id": "MD1", "country": "Vietnam", "technology": "Hydro"},
    ),
]

# 2. Setup the Vector Store (Long-Term Memory)
embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
    documents=project_docs,
    embedding=embedding_model,
    persist_directory="./chroma_db_projects" # Persist the DB to disk
)

# 3. Create the Retriever
retriever = vectorstore.as_retriever(search_kwargs={"k": 2})

# 4. Define the RAG Prompt Template
template = """
You are a project finance analyst assistant. Answer the following question based ONLY on the context provided.
If the context does not contain the answer, state that you don't have enough information.

Context:
{context}

Question:
{question}
"""
prompt = ChatPromptTemplate.from_template(template)

# 5. Build the RAG Chain
llm = ChatOpenAI(model="gpt-4o")

rag_chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# --- Query 1: Ask a question that can be answered by the context ---
print("--- Query 1 ---")
query1 = "What is the PPA term for the Helios One project?"
response1 = rag_chain.invoke(query1)
print(f"Question: {query1}\nAnswer: {response1}\n")

# --- Query 2: Ask about a different project ---
print("--- Query 2 ---")
query2 = "What are the main risks for the Mekong Delta project?"
response2 = rag_chain.invoke(query2)
print(f"Question: {query2}\nAnswer: {response2}\n")

# --- Query 3: Ask a question where context is insufficient ---
print("--- Query 3 ---")
query3 = "What is the construction cost for the Andes Wind project?"
response3 = rag_chain.invoke(query3)
print(f"Question: {query3}\nAnswer: {response3}\n")

Run this code. You'll see how the chain successfully answers questions when the information is in our document set, and correctly states when it doesn't have the information, because we instructed it to rely only on the retrieved context. This prevents the LLM from hallucinating answers.

Part 5: Architecting for Production

While ChromaDB is excellent for local development, building production-grade agentic systems requires more robust and scalable solutions. Your CS background will be valuable in appreciating these architectural considerations. A common challenge is managing multiple types of memory (conversational, factual, procedural) efficiently.

A powerful pattern is to unify these different memory types in a single, scalable database like PostgreSQL, enhanced with extensions like pgvector for vector search.

The article "Building AI Agents with Persistent Memory" provides an excellent blueprint for this unified approach. It distinguishes between three memory types:

• Episodic Memory: The conversation history (short-term memory from our last lesson).
• Semantic Memory: The knowledge base of embedded documents (long-term RAG memory).
• Procedural Memory: Stored user preferences or agent configurations.

Adopting a unified database architecture with PostgreSQL and pgvector allows you to perform powerful queries that join across different memory types in a single transaction, reducing complexity and latency compared to managing separate databases for each memory function. This is a robust pattern for the kind of AI-native systems you aim to build.

Conclusion

In this lesson, you've unlocked a new dimension of agent capability by implementing long-term memory. Your agent can now learn from and leverage a dedicated knowledge base, moving it from a simple conversationalist to a knowledgeable assistant.

Key Takeaways:

• RAG is the Key to Knowledge: Retrieval-Augmented Generation (RAG) is the fundamental process of retrieving relevant information from an external source before generating an LLM response.
• Vector Databases are the "Brain": Vector databases like ChromaDB or PostgreSQL with pgvector store document embeddings and enable the fast semantic search required for RAG.
• The RAG Pipeline: The process is to chunk, embed, and store documents (indexing), then use a retriever to find relevant context for a given query to pass to the LLM (retrieval and generation).
• Production Systems Need Robust Architectures: For scalable and maintainable agents, consider unified database architectures that handle episodic, semantic, and procedural memory within a single system.

You now know how to manage both the flow of a conversation (short-term memory) and a vast library of external knowledge (long-term memory). But where does this external knowledge come from in the first place? A project finance professional deals with data from dozens of sources: market data APIs, government filings, news articles, legal documents, and more.

In our next lesson, "Map the diverse data sources for project finance," we will begin exploring the data engineering foundations required to feed your agent's long-term memory, starting by identifying and mapping the critical data sources in the project finance ecosystem.