Crafting Comprehensive Evaluation Datasets

Hello! Welcome back to our module on "Evaluation and Evidence-Based Improvement."

In our previous lesson, we established the critical first step in evaluation: defining measurable success criteria and linking them directly to business outcomes. We now know what to measure. Today, we'll build on that by learning how to construct the tools for that measurement.

This lesson directly addresses the learning outcome: Design and construct an evaluation dataset with test cases covering normal operation, edge cases, and adversarial inputs. Building a high-quality evaluation dataset is the core practice that separates professional, reliable agentic systems from brittle prototypes. It's how you generate the "evidence, not vibes" that proves your system is ready for production and can deliver on its promises to a client.

We will cover the composition of a "golden dataset," a systematic approach to designing the three essential categories of test cases, and how to use advanced concepts to create sophisticated adversarial inputs that ensure your system is truly robust.

1. The "Golden Dataset": Your Source of Truth

At the heart of any rigorous evaluation process is a golden dataset. This is a curated collection of inputs, paired with their expected outputs or evaluation criteria, that represents the full spectrum of tasks your agentic system is expected to handle. It serves as the benchmark against which you will measure all improvements.

The diagram below illustrates how this dataset is central to the entire development and operational lifecycle.

Creating this dataset is an iterative process. It doesn't need to be perfect or exhaustive from day one.

As the video highlights, the size of your golden dataset evolves with your project's maturity:

• Initial Prototyping (Internal Testing): Start with 10-20 examples. This is enough to iterate quickly on your prompts and architecture and validate your core assumptions.
• Pre-Production (Confidence Building): Expand to 100+ examples. A larger dataset gives you the statistical confidence to move towards a production launch or a client-facing proof-of-concept.

Now, let's break down how to design the contents of this dataset.

2. Designing Your Test Cases: Three Essential Categories

A robust evaluation dataset must cover three distinct categories of inputs. Each serves a unique purpose in assessing the quality and reliability of your agentic system.

2.1 Normal Operation (The "Happy Path")

These test cases represent the standard, expected, and most frequent use cases for your system. They validate that the core functionality works as intended under ideal conditions.

• Purpose: To establish a baseline of correct behavior and catch regressions in fundamental capabilities.
• Characteristics: Unambiguous queries, complete information provided, common scenarios.

For the market research agent we discussed in the last lesson, a happy path test case might be: "Generate a 5-page report on the global electric vehicle market, focusing on North America for the year 2023."

2.2 Edge Cases

Edge cases test the boundaries and known limitations of your system. They are not necessarily malicious but are designed to probe situations where the agent might struggle or fail. Given your background in software engineering, you can think of this as analogous to testing with null inputs, boundary values (0, -1, max_int), or empty strings.

• Purpose: To uncover logical flaws, improve robustness, and define the operational boundaries of your agent.
• Characteristics: Ambiguous queries, requests with missing data, unusual combinations of parameters, inputs that test policy limits (e.g., a 30-day return policy tested on day 30).

The video you just watched provides a great example of an edge case with the "suspicious" transaction—a sequence of ATM withdrawals that isn't clearly fraudulent but is highly unusual. Another example from the "AI Evaluations" course (LINK at 23:20) was when the agent failed at simple math, comparing a 45-minute window to a 60-minute limit. This failure should immediately be codified as an edge case in the golden dataset to prevent future regressions.

For our market research agent, an edge case could be: "Compare the EV market in Antarctica and the Sahara desert." This tests how the agent handles queries for which little to no data exists.

2.3 Adversarial Inputs

Adversarial inputs are a step beyond edge cases. They are inputs crafted with malicious intent, specifically designed to exploit vulnerabilities and cause the system to fail in a predictable way. For client-facing systems, proving robustness against such attacks is a critical part of demonstrating production-readiness.

This is where your system design expertise can truly shine. We will approach this with the same rigor as designing a secure distributed system.

A Principled Framework: Adversarial Co-evolution

A powerful concept from software testing for generating robust tests is mutation testing. The core idea is to introduce small, artificial faults ("mutants") into your code and check if your test suite can "kill" (detect) them. A test suite that kills many mutants is considered highly robust.

We can apply a similar, more advanced principle to agentic systems using an adversarial loop.

The AdverTest paper illustrates a powerful mental model: you can design systems that help you find your own system's weaknesses. An adversarial agent's goal is to generate inputs that make your main agent fail. Every time it succeeds, you have found a valuable new test case to add to your golden dataset. The surviving "mutants" are your most valuable adversarial test cases.

A Taxonomy of Threats for Tool-Using Agents

To make this concrete, we need a classification of the types of adversarial attacks that tool-using agents are vulnerable to.

This paper provides a practical checklist for creating adversarial test cases for your agentic workflow. Let's summarize the key generation techniques:

• Deceptive Similarity: Create malicious tools that are textually similar to legitimate ones to fool the retriever.
  • Parameter Collision: Copy the argument schema of a valid tool.
  • Homoglyph Clone: Use visually similar characters in the tool name.
• Persuasion Cues: Add lexical markers to a tool's description to make it seem more appealing to the LLM selector (e.g., v2.0, official, recommended, all-in-one).

These techniques can be used to implement several types of attacks, which should form a core part of your adversarial test set:

1. Top-N Saturation: The adversary floods the tool pool with near-duplicates of a legitimate tool, pushing the correct one out of the retrieval slate.
2. Privilege Escalation: The agent is tricked into using a tool that requires higher permissions than it should have (e.g., a tool named get_public_stock_data that actually tries to access a private financial database).
3. Intent Shifting: The agent is persuaded to use a tool that performs a related but incorrect action, failing to satisfy the user's original intent.

By designing test cases that simulate these attacks, you can rigorously validate the security and reliability of your agent's tool selection and reasoning logic. For a client, being able to say "We have tested against privilege escalation attacks and can prove the agent will not access unauthorized data" is an incredibly powerful statement.

Conclusion

In this lesson, we've moved from the abstract concept of success criteria to the concrete practice of building the dataset needed to measure them. A well-designed evaluation dataset is not an afterthought; it is a strategic asset that guides development, proves reliability, and builds client confidence.

Key Takeaways:

• A golden dataset is your source of truth for evaluation, containing test cases for normal operations, edge cases, and adversarial inputs.
• The dataset should be treated as a living artifact, starting small for rapid iteration and growing in size and complexity as your system matures.
• Normal operation tests validate core functionality, while edge cases probe boundaries and known weaknesses.
• Adversarial testing is essential for building robust, secure systems. You can systematically design these tests using principled frameworks like adversarial co-evolution and threat taxonomies focused on agent vulnerabilities like tool selection.

You now have the framework to design and construct a comprehensive evaluation dataset. In our next lesson, we will put this dataset to use. We'll learn how to execute a benchmark by systematically running an agent against the evaluation dataset and collecting the outputs, turning our carefully crafted test cases into hard data.