Evaluating RAG Generation with Faithfulness and Relevance

Welcome back! In the last lesson, we focused on evaluating the retrieval stage of our RAG pipeline. By using context_precision and context_recall, we established a systematic way to ensure that the information we pass to our Language Model is as relevant and complete as possible. This is a crucial first step, as a great generator can't salvage poor-quality context.

Now that we have a process to verify our retriever is doing its job, we must turn our attention to the second major component: the generator. Even with perfect context, the LLM can still produce a subpar answer. In this lesson, we will complete our RAG evaluation framework by learning to evaluate the generation component using the core metrics of faithfulness and answer relevance.

Mastering these metrics allows you to pinpoint whether a bad response is due to the LLM hallucinating, ignoring the context, or simply failing to answer the user's question directly. For an AI Engineer, this ability to diagnose and fix issues at every stage of the pipeline is essential for building trustworthy and effective production systems.

Why Evaluate the Generator?

The generator's job is to synthesize a coherent answer from the user's question and the retrieved context. However, this process can fail in several ways, even if the retrieved context is perfect.

As the article points out, the LLM might ignore key information, focus on the wrong details, or even disregard the context entirely in favor of its own internal knowledge. To systematically detect these issues, we rely on two key metrics: Faithfulness and Answer Relevance.

Faithfulness: Is the Answer Grounded in Facts?

Faithfulness measures the factual consistency of the generated answer against the provided context. In simple terms, it answers the question: "Does the answer stick to the context?" A low faithfulness score is a strong indicator of hallucination, where the model invents information that wasn't in the source documents. This is arguably the most critical metric for building user trust.

The core idea behind calculating faithfulness is to treat the LLM's output like a set of claims that need to be fact-checked against the retrieved context. The process is:

1. Decompose: Break down the generated answer into individual factual statements or claims.
2. Verify: For each claim, check if it is directly supported by the information present in the retrieval_context.
3. Score: The faithfulness score is the ratio of supported claims to the total number of claims.

(
\text{Faithfulness} = \frac{\text{Number of Supported Claims}}{\text{Total Number of Claims}}
)

A score of 1.0 means every part of the answer is backed by the context, while a score less than 1.0 indicates some degree of hallucination.

Answer Relevance: Does the Answer Help the User?

While faithfulness ensures the answer is factually correct based on the context, it doesn't guarantee the answer is actually useful. An answer can be 100% faithful but completely miss the point of the user's question.

Answer Relevance measures how well the generated answer addresses the user's original question. It answers: "Is the answer on-topic and helpful?"

The calculation for answer relevance is clever. It typically involves using an LLM to work backward:

1. Generate Questions: Given the generated answer, an LLM is prompted to create a few questions that this answer would be perfect for.
2. Compare: The semantic similarity between these newly generated questions and the user's original question is calculated.
3. Score: The average similarity score becomes the answer relevance score. A high score means the answer is a good fit for the original question.

These two metrics, Faithfulness and Answer Relevance, are considered "reference-free" in the sense that they don't require a pre-written ground_truth answer to be calculated. They only need the question, the retrieved contexts, and the generated answer.

From Theory to Code: "LLM as a Judge"

Under the hood, libraries like ragas and deepeval implement these metrics using a powerful pattern known as "LLM as a Judge." Instead of complex algorithms, they use another powerful LLM (like GPT-4) with a carefully crafted prompt to perform the evaluation.

The following video provides an excellent walkthrough of how you would build these evaluation functions from scratch, which is exactly what frameworks like ragas do for you. Since you're focused on building production systems, understanding this underlying mechanism is incredibly valuable.

As you saw, the logic is encapsulated in functions that take the input (question), output (generated answer), and retrieval_context as arguments and return a score. This "LLM as a Judge" approach is powerful because it can evaluate nuanced aspects of language that are hard to capture with traditional metrics.

Interpreting Scores and Driving Improvements

Just like with retrieval metrics, the scores for faithfulness and answer relevance are not just for reporting; they are diagnostic tools that guide engineering decisions.

Here's a summary of how to react to low scores:

• If faithfulness is low:

  • Symptom: Your answers contain made-up facts. The LLM is hallucinating.
  • Potential Solutions:
    1. Strengthen the Prompt: Make your instructions more explicit. Add phrases like: "Answer the question based only on the provided context. If the information is not in the context, say 'I don't know'."
    2. Use a Different Model: Some models are more prone to hallucination than others. A smaller, more "obedient" model might be better than a larger, more "creative" one for some RAG tasks.
    3. Implement Citations: Force the model to cite which part of the context supports each part of its answer. This makes it harder for the model to invent facts.

• If answer_relevancy is low:

  • Symptom: The answer is factually correct but off-topic.
  • Potential Solutions:
    1. Check Retrieval First: Low answer relevance is often a symptom of poor retrieval (context_precision). If the context is full of irrelevant noise, the LLM can get distracted. Always fix your retriever first.
    2. Improve Prompt Engineering: You can add instructions to the prompt to keep the model focused, such as "Your answer must directly address the user's question."
    3. Context Filtering: Before passing all retrieved chunks to the LLM, you could implement a filtering step to remove any chunks that seem less relevant to the original question.

Conclusion

Congratulations! You now have a complete, two-part framework for evaluating a RAG system. By measuring the retriever with context_precision and context_recall, and the generator with faithfulness and answer_relevance, you can move beyond anecdotal testing and adopt a true metric-driven development process. This is a critical skill that separates hobbyists from professional AI Engineers.

Key Takeaways:

• Generator evaluation is crucial: Even with perfect context, the LLM can fail by hallucinating (faithfulness) or providing irrelevant answers (answer_relevance).
• Faithfulness fights hallucination: It measures if the generated answer is factually grounded in the retrieved context.
• Answer relevance ensures utility: It measures if the answer actually helps the user by addressing their specific question.
• "LLM as a Judge" is the core pattern: Modern evaluation frameworks use a powerful LLM with specific prompts to score these nuanced metrics.
• Scores are diagnostic: Use low scores in faithfulness or relevance to guide specific improvements in your prompt engineering, model selection, or overall RAG logic.

In the next module, we'll take a significant step forward. We'll move beyond systems that just answer questions and start building autonomous AI agents that can reason and take actions in the world. Our first topic will be the foundational ReAct (Reasoning and Acting) pattern, which forms the basis for many modern agentic workflows.