Evaluating Agent Performance with Deterministic Test Cases

Recommended resources

The freely available resources this lesson will direct you to.

Hello! In our previous lessons, we've focused on building increasingly robust AI agents. We've designed them to reason and act, handle errors gracefully, and, most recently, to prevent them from getting stuck in costly infinite loops. Now that we have a stable agent, a critical question arises: how do we know if it's actually any good?

This lesson transitions from building to verifying. We will tackle the crucial topic of evaluation, focusing on how to measure an agent's performance in a systematic and repeatable way. You will learn how to evaluate an agent's autonomy and its ability to complete tasks by creating deterministic test cases. For you as a software engineer, this process will feel familiar, as it involves adapting the principles of unit and integration testing to the unique, non-deterministic nature of AI.

Why Agent Evaluation is a Unique Challenge

Evaluating a simple LLM application often boils down to checking the quality of its final text output. An agent, however, is a complex system with internal state, decision-making logic, and interactions with external tools. The final answer is only part of the story.

Consider the difference highlighted in the diagram below. Traditional LLM evaluation is linear: prompt in, text out. Agentic evaluation, on the other hand, requires us to inspect the entire process—the reasoning steps, the tool calls, the error corrections, and the checkpoints along the way.

This diagram contrasts the simple input-output focus of traditional LLM evaluation with the complex, multi-step process required for evaluating AI agents, which involves checking tool usage, reasoning, and intermediate checkpoints.

An agent could arrive at the correct answer through a wildly inefficient or expensive path. Conversely, it could fail a task for a very specific reason (e.g., a single malformed tool call). Without a proper evaluation framework, you're flying blind.

The Agent Testing Pyramid

To bring structure to this complexity, we can adopt a testing pyramid model, a concept that will be very familiar from traditional software development. This framework helps us organize our evaluation efforts from foundational checks to high-level quality assurance.

How to evaluate agents in practice

This video from Google Cloud Tech introduces a practical, three-tiered framework for agent testing. It's an excellent mental model for structuring your evaluation strategy.

Please watch the segment from the beginning, which outlines the three tiers: Component-level unit tests: Fast, automated checks of the smallest building blocks (e.g., tool selection). Trajectory-level integration tests: End-to-end tests of a full multi-step task. End-to-end human review: A final quality gate for assessing subjective qualities like helpfulness and tone.

In this lesson, we will concentrate on the first two tiers, as they form the basis of creating the deterministic test cases mentioned in our learning outcome. These are the tests you can automate and run as part of a CI/CD pipeline to prevent regressions and ensure reliability.

Tier 1: Unit Testing Agent Components with Mocking

At the base of the pyramid are unit tests. As the Google video explained, these test the "smallest building blocks in isolation." But how can you unit test a component that depends on a non-deterministic, slow, and potentially expensive LLM call?

The answer is the same as in traditional software engineering: mocking. We replace the live LLM client with a controlled, fake version that returns predictable responses. This makes our tests fast, free, deterministic, and repeatable.

Unit Testing AI Agents: Mocking LLM Calls for Fast, Deterministic Tests | CallSphere Blog

This article from CallSphere provides an excellent, code-level guide to mocking strategies for agent unit tests. It's written from a software engineering perspective and will be highly relevant.

First, read the introductory section, Why Unit Testing Agents Requires Special Patterns, to solidify your understanding of the problem. Next, read about two key implementation strategies: FakeLLM Classes: This pattern involves creating a simple, deterministic replacement for your LLM client. Study the Python FakeLLM dataclass example; this is a clean, dependency-free approach. Patching with unittest.mock: This uses Python's standard mocking library to intercept the API call directly, a flexible pattern you may have used in other contexts. Finally, and most importantly, read the section on Assertion Patterns. This is the crucial takeaway. Notice that the assertions are not checking the LLM's generated text. Instead, they verify the agent's behavior based on the mocked response: Did it select the correct tool? (assert "search_orders" in tool_names) Did it exhibit the correct error-handling logic, like retrying? (assert len(fake_llm.call_log) == 2)

By mocking the LLM, you can create a suite of deterministic unit tests for critical agent logic:

Tool Selection: Given a prompt, does the agent choose the search_database tool?
Argument Parsing: Does the agent correctly extract {"query": "open tickets"} to pass to the tool?
Error Handling (from our previous lessons): If the LLM returns malformed JSON for a tool call, does the agent's self-correction or retry mechanism engage?
Loop Prevention (from our last lesson): Does the agent correctly terminate after hitting the max_turns limit?

Tier 2: Evaluating Task Completion and Trajectory

Once we've verified the individual components, we move up the pyramid to integration-style tests. Here, the goal is to assess the agent's ability to complete a full, multi-step task successfully. This directly addresses the "task completion rate" and "autonomy" aspects of our learning outcome.

To do this, we need to evaluate two things: the final outcome and the path taken to get there.

These categories of metrics show the different dimensions of agent performance. Our focus in this section is on **Task Completion** (the outcome) and **Tool Interaction** (part of the trajectory).

The Concept of a "Golden Dataset"

To run these tests, we need a "golden dataset." This is a collection of test cases, each containing an input (e.g., a user query) and a corresponding expected output. However, for agents, the "output" is more than just a final string of text.

Beginner's Guide to Agent Evaluations

This video from LangChain clearly explains the components needed for evaluation and the different evaluation strategies.

First, watch the overview of the three strategies. Notice how they map to our pyramid: evaluating the final output, a single step (unit test), and the entire trajectory. Next, watch the explanation of the three things needed for an evaluation: a golden dataset, the application logic, and evaluators. Focus on how the composition of the golden dataset changes depending on what you're evaluating (final output vs. correct routing step vs. ideal trajectory).

For trajectory evaluation, your golden dataset would look something like this:

Input: "How many songs do you have by James Brown?"
Expected Trajectory: ["route_to_qa_agent", "call_tool:lookup_artist", "call_tool:count_songs", "format_response"]
Expected Final Answer: "We have 75 songs by James Brown."

Measuring Task Completion and Trajectory Correctness

With a golden dataset in place, we can run our agent against the inputs and compare its actual behavior to our expectations. This is where we measure our key metrics.

Task Completion Rate: This is the most straightforward metric. For a given set of test cases, what percentage of them did the agent successfully complete? "Completion" can be defined by comparing the agent's final answer to the expected answer in the golden dataset. Because LLM outputs are variable, this comparison often uses semantic similarity or an LLM-as-a-judge, but for deterministic tests, you can mock the final generation step as well.
Trajectory Correctness: This measures how closely the agent's path matched the ideal path defined in the golden dataset. A simple way to do this is to compare the sequence of tool calls.

Beginner's Guide to Agent Evaluations

This segment from the LangChain video provides a practical demonstration of trajectory evaluation.

Watch the detailed walkthrough starting from this section. Pay close attention to how they construct the golden dataset with an expected list of steps. The evaluators they build are simple but powerful Python functions that check for: Extra Steps: Did the agent take unnecessary actions? This indicates inefficiency. Unmatched Steps: Did the agent perform the wrong actions or the right actions in the wrong order? This indicates a flaw in its reasoning plan.

By combining these metrics, you gain a deep, quantitative understanding of your agent's autonomy and effectiveness. A high task completion rate with a low trajectory correctness score might indicate that your agent is functional but highly inefficient. A low score on both might point to a fundamental issue in its reasoning or tool-use capabilities.

Conclusion

In this lesson, you've learned how to move beyond simply building agents to rigorously evaluating them. This is a non-negotiable skill for any AI engineer aiming to build production-grade systems. By adapting the familiar testing pyramid from software engineering, you can create a robust, automated evaluation pipeline.

Key Takeaways:

Agent evaluation is multi-faceted: It requires assessing not just the final output, but the entire trajectory of reasoning and tool use.
The Testing Pyramid provides structure: Start with fast, deterministic unit tests for components, then move to integration tests for multi-step task trajectories.
Mocking is essential for deterministic tests: By replacing live LLM calls with FakeLLM objects or patches, you can create fast, repeatable, and free unit tests to verify core logic like tool selection and error handling.
Golden datasets are your source of truth: For trajectory evaluation, you need a dataset of inputs paired with the ideal sequence of steps and/or the final correct answer.
Key metrics are Task Completion Rate and Trajectory Correctness: These quantifiable measures tell you not only if the agent succeeded, but how it succeeded, revealing issues with efficiency and reasoning.

This lesson concludes our module on designing autonomous AI agents. You now have a comprehensive toolkit to build, stabilize, and evaluate them. In our next module, we will explore the Model Context Protocol (MCP), an emerging standard that offers a powerful and structured alternative for providing dynamic, real-time context to LLMs.