Running Benchmarks

Hello! Welcome back to our module on "Evaluation and Evidence-Based Improvement."

In the last lesson, we focused on the meticulous process of designing and constructing a high-quality evaluation dataset—your "golden dataset"—complete with test cases for normal operation, edge cases, and adversarial inputs. You now have the what to test against.

Today, we'll address the learning outcome: Execute a benchmark by systematically running an agent against the evaluation dataset and collecting outputs. This is where we put your dataset to work, transforming it from a static collection of examples into a dynamic source of evidence. This process is the engine that powers data-driven improvement and provides the hard proof of performance your clients will demand.

We will explore the architectural workflow for benchmarking, detail the critical data points you need to collect beyond just the final answer, and discuss how to approach this process with the scientific rigor required for building scalable, reliable systems.

1. The Benchmarking Workflow: An Automated System

Executing a benchmark isn't an ad-hoc process of running a few scripts. For a production-grade system, you should design it as an automated, repeatable workflow. This ensures consistency and allows you to run evaluations continuously as you make changes to your agentic system.

A robust evaluation workflow can be conceptualized in four main steps.

As described in the AWS article, the workflow is:

1. Define Inputs: This is where we feed our golden dataset into the system.
2. Generate Metrics: The system automatically runs the agent on each input and uses an evaluation library to generate metrics. This is the core "execution" step.
3. Share Results: The collected outputs and metrics are stored in a structured format (e.g., in a database, S3 bucket, or a dashboard).
4. Analyze Performance: The results are then audited and monitored to detect performance changes and identify areas for improvement.

Today, we'll focus on steps 2 and 3: executing the run and collecting the right data for storage and subsequent analysis.

2. Executing the Run and Collecting the Right Data

The execution phase involves programmatically iterating through every test case in your evaluation dataset, invoking your agent, and, most importantly, capturing the results. The key to effective analysis later is to collect not just the final output, but a rich trace of the agent's entire execution process.

As the video demonstrates, the goal is to create a structured log for each test case run. For a system designer, the challenge is deciding what to log. A simple pass/fail is insufficient. We need a multi-layered approach to data collection.

Multi-Layered Data Collection

Let's use the mental model from the AWS resource (LINK) and think about data collection at different layers of the agentic system.

A. Trace-Level Data (The Execution Path)

This is the most granular data, capturing how the agent arrived at its answer. It provides the deep observability needed to debug failures. For each run, you should aim to collect:

• LLM Calls: The full prompt sent to the LLM (including system message, user input, and context), the raw model response, token counts, latency, and the specific model version used.
• Tool Interactions: The name of the tool called, the exact parameters passed to it, the data returned by the tool, and any errors that occurred.
• Reasoning Steps: If your agent uses a pattern like Chain-of-Thought or ReAct, capture the intermediate reasoning steps.

This level of detail is critical for complex, multi-step workflows, such as those involving router agents.

B. Output-Level Data (The Final Answer)

This is the agent's final deliverable for the given task.

• Final Response: The complete, user-facing answer.
• Structured Output: If the agent is supposed to produce structured data (e.g., a JSON object), capture that object.

C. Ground Truth Data (The Benchmark Standard)

This data comes directly from your golden dataset and serves as the reference against which the agent's performance is measured.

• Reference Answer: The ideal or expected final response.
• Reference Context/Actions: For RAG or tool-use, this could be the specific document chunks that should have been retrieved or the tool calls that should have been made. The video with James Briggs (LINK) referred to this as ground_truth and ground_truth_context.

D. Evaluation Metrics (The Scores)

During the benchmark execution, you can also run evaluators that compare the agent's output to the ground truth. This produces quantitative scores that summarize performance. The AWS article (LINK) provides an excellent "menu" of metrics you can collect.

By collecting data from all four of these categories for every test case, you create a rich dataset that allows for deep analysis, moving far beyond a simple accuracy score.

3. Storing and Visualizing Benchmark Results

Once your benchmark run is complete, you will have a large volume of structured data. The final step in the collection process is to store this data in a way that facilitates analysis and reporting.

• Storage: While simple CSV or JSON files are fine for initial experiments, a scalable solution might involve logging results to a database (like PostgreSQL) or a data lake (like Amazon S3 or Google Cloud Storage).
• Visualization: The ultimate goal of collecting this data is to make informed decisions. A dashboard that visualizes benchmark results is an incredibly powerful tool for communicating with both technical teams and non-technical stakeholders.

This dashboard view makes it immediately clear whether a change has improved or degraded the system, and on which specific dimensions. The diagram below shows how this all fits into a continuous loop.

4. Advanced Benchmarking: A Principled, Scientific Approach

Given your background, you're well-equipped to move beyond simple pass/fail benchmarking to a more scientific characterization of agent system performance. This involves designing your benchmark runs as controlled experiments to derive generalizable principles about your system's behavior.

This scientific approach to benchmarking allows you to answer much deeper questions for a client:

• Instead of "Does it work?" you can answer, "Under what conditions does this architecture outperform others, and by how much?"
• You can quantify the trade-offs. For example, you can state that "Centralized orchestration reduces error amplification by 4x compared to independent agents, but at the cost of 285% more coordination overhead (in tokens)."

This level of analysis, enabled by collecting the right data during a well-designed benchmark, is what distinguishes a top-tier agentic systems architect.

Conclusion

In this lesson, we have laid out the complete process for executing a benchmark. We've seen that it's an automated workflow that takes our evaluation dataset and systematically runs an agent against it to produce hard evidence of performance.

Key Takeaways:

• Benchmarking is an automated workflow: Design your evaluation process to be repeatable and systematic, treating it as a core part of your CI/CD pipeline.
• Collect rich, multi-layered data: Go beyond the final answer. Capture detailed traces of the agent's execution, including LLM calls, tool use, and intermediate reasoning, alongside the final output and ground truth.
• Generate quantitative metrics: During the benchmark, compute a suite of relevant metrics covering everything from final response quality to tool-use accuracy and safety.
• Store and visualize results: Use structured storage and dashboards to make the benchmark outputs accessible and interpretable for your team and your clients.
• Adopt a scientific mindset: For deep insights, design your benchmarks as controlled experiments, measuring system-level properties like coordination overhead and error amplification to truly understand how your system behaves at scale.

You now know how to generate the raw data needed for evidence-based improvement. In our next lesson, we will focus on the final step: Interpreting evaluation results to identify specific components (prompts, retrieval, tools) that need improvement. We will learn how to read the signals in the data you've just collected to make targeted, impactful changes to your agentic system.