Retrieval Evaluation for RAG Systems

Welcome back! In our last lesson, we implemented a cross-encoder re-ranking stage to improve the precision of our retrieval, ensuring the documents passed to the LLM are as relevant as possible. This optimization, along with others like query expansion, makes our RAG pipeline more sophisticated. But it also raises a crucial question for any engineer: "How do we prove these changes are actually improvements?"

Merely "eyeballing" a few example answers is not a scalable or reliable way to measure performance in a production system. This is where systematic evaluation comes in.

In this lesson, you will learn to evaluate the retrieval component of a RAG system using the core metrics of context precision and context recall. Mastering this skill is fundamental to metric-driven development, allowing you to objectively measure the impact of any change to your retrieval pipeline—be it a new chunking strategy, a different embedding model, or a re-ranking stage. This is a non-negotiable skill for a professional AI Engineer and a frequent topic in system design interviews.

Separating Retrieval from Generation

A RAG pipeline has two main stages, and a failure in one can be mistaken for a failure in the other:

1. Retrieval: The process of finding and ranking relevant context documents from your knowledge base.
2. Generation: The process where the LLM uses the retrieved context to synthesize a final answer.

A bad answer could be because the retriever failed to find the right information (a retrieval problem) or because the LLM misinterpreted or ignored the provided context (a generation problem). To debug effectively, we must evaluate them separately.

Today, we focus entirely on the retrieval stage. The core question we're trying to answer is: "How good is our system at finding the right evidence?"

The Prerequisite: A "Ground Truth" Test Set

To measure performance, we need a benchmark—a set of questions for which we already know the correct information. This is called a test set or evaluation dataset. At a minimum, each entry in this dataset needs:

• question: The user's query (e.g., "What is the capital of France?").
• ground_truth_context: A list of the specific text chunks from your database that contain the necessary information to answer the question. This is our "source of truth" for what the retriever should find.
• ground_truth: The ideal, factually correct answer. This will be used later to evaluate the generation step.

Creating this dataset can be time-consuming, but it's essential. For getting started, frameworks like ragas can help you generate a synthetic test set automatically, which is a great way to bootstrap your evaluation process.

The following video demonstrates how to use ragas to generate such a test set and prepare for evaluation.

The Core Retrieval Metrics: Precision and Recall

With our test set in hand, we can now measure the performance of our retriever using two classic information retrieval metrics: Context Precision and Context Recall.

The following diagram from the ragas framework shows how these metrics relate the Query, the retrieved Context, and the Ground Truth Answer.

To understand these metrics intuitively, it's helpful to think in terms of True Positives, False Positives, and False Negatives. The video below gives an excellent conceptual breakdown.

Now, let's dive into the definitions and why each metric is important.

From Theory to Code: Evaluating with ragas

Now let's apply these concepts. The ragas library makes it straightforward to run these evaluations. The general process is:

1. Prepare a test set with question and ground_truth_context columns.
2. For each question, run your RAG pipeline's retrieval step to get the list of retrieved_contexts.
3. Combine these into a single dataset.
4. Call the ragas.evaluate function, specifying the context_precision and context_recall metrics.

Here's a conceptual Python script showing how you would set this up. It assumes you have a retriever object (like the one we've built in previous lessons) and an evaluation dataset.

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import context_precision, context_recall




# 1. Assume you have your evaluation dataset loaded
# This would typically come from a file or a synthetic generator
eval_data = {
    'question': [
        "What is the difference between recursive and fixed-size chunking?",



        # ... more questions
    ],
    'ground_truth_context': [
        ["Fixed-size chunking involves splitting text into chunks of a specific character length...", "RecursiveCharacterTextSplitter tries to keep paragraphs, sentences, and words whole..."],



        # ... more ground truth contexts
    ]
}
eval_dataset = Dataset.from_dict(eval_data)




# 2. Run your retriever for each question to get the retrieved contexts
# Your retriever is the function/object from your RAG pipeline
# (e.g., the one that queries ChromaDB and uses the re-ranker)
retrieved_contexts = []
for question in eval_dataset["question"]:



    # This is your actual RAG system's retrieval function
    docs = your_retriever_function(question, k=5) 
    retrieved_contexts.append([doc.page_content for doc in docs])




# 3. Add the results to the dataset
eval_dataset = eval_dataset.add_column('contexts', retrieved_contexts)




# 4. Run the evaluation
result = evaluate(
    dataset=eval_dataset,
    metrics=[
        context_precision,
        context_recall,
    ],
)




# The 'result' object will contain the scores
print(result)



# Expected output might look like:
# {'context_precision': 0.85, 'context_recall': 0.90}

This workflow provides a repeatable, objective way to benchmark your retriever's performance.

Interpreting Scores and Driving Improvements

Getting a score is just the first step. The real value comes from using those scores to diagnose problems and guide improvements. This is a critical skill you'll be expected to demonstrate in an AI engineering role.

Here’s a practical troubleshooting guide based on your retrieval scores:

• If context_precision is low:

  • Symptom: Your retriever is returning a lot of irrelevant "noise" alongside the correct documents. This wastes context space and can confuse the LLM.
  • Potential Solutions:
    1. Add/Tune a Re-ranker: This is the most direct solution. A cross-encoder is specifically designed to improve precision. (We did this last lesson!)
    2. Improve the Embedding Model: Your model might not be capturing the semantic nuances of your domain. Consider fine-tuning it or switching to a domain-specific model.
    3. Optimize Chunking: If your chunks are too large, they may contain mixed topics, causing them to be retrieved for irrelevant queries. Try smaller chunk sizes.
    4. Implement Query Rewriting: Use an LLM to transform the user's raw query into a more precise search query for the vector database.

• If context_recall is low:

  • Symptom: Your retriever is failing to find all the necessary pieces of information. The final answer will likely be incomplete or incorrect.
  • Potential Solutions:
    1. Increase k: Retrieve more documents in the initial search (e.g., retrieve 20 instead of 10) to increase the chance of capturing all relevant chunks. This is often a trade-off with precision.
    2. Use Hybrid Search: Combine dense (vector) search with sparse (keyword) search like BM25. This helps find documents that match exact terms, which vector search can sometimes miss.
    3. Implement Query Expansion: Use an LLM to break a complex question into multiple sub-questions, run a retrieval for each, and combine the results.

Conclusion

You now have a robust framework for evaluating and improving the most critical part of your RAG system: the retriever. By moving from subjective spot-checking to metric-driven development, you are operating like a true AI Engineer.

Key Takeaways:

• Isolate and Evaluate: Always evaluate the retrieval and generation components of your RAG system separately to effectively diagnose issues.
• Precision vs. Recall: These two metrics provide a comprehensive view of your retriever's performance. Context Precision measures noise, while Context Recall measures completeness.
• Ground Truth is Key: Systematic evaluation is impossible without a well-defined test set containing questions and their corresponding ground truth contexts.
• Use Tools: Frameworks like ragas automate the calculation of these metrics, letting you focus on interpreting the results and improving your system.
• Scores Guide Action: Low precision or recall scores are not just numbers; they are signposts pointing you toward specific, actionable solutions like re-ranking, query expansion, or hybrid search.

In our next lesson, we will shift our focus to the other side of the pipeline. Now that we have a method for ensuring our retrieved context is high-quality, we'll learn how to evaluate the LLM's generated answer using metrics like faithfulness and answer relevance.