Defining Success: Agentic Workflows & Business Outcomes

Recommended resources

The freely available resources this lesson will direct you to.

Welcome to the first lesson in our module on "Evaluation and Evidence-Based Improvement." In our last session, we built a crucial capability: tracing a request end-to-end through a multi-agent system. This gave us the observability to see what our system is doing, diagnose failures, and spot performance bottlenecks. Now, we shift from observing to judging. It's not enough for a system to run; it must run well and deliver tangible value.

This lesson tackles the foundational step in that process. Our learning outcome is to define measurable success criteria for an agentic workflow tied to business outcomes. This skill is the bedrock of building systems you can confidently improve, scale, and, most importantly, sell to a client. It's how you move from vibes ("it seems to work") to evidence ("I can prove this saves you $100k a year").

We will explore why evaluating agentic systems requires a new mindset, establish a framework for connecting technical metrics to business ROI, and outline a rigorous process for defining criteria that prove the value of your work.

1. The Evaluation Challenge: Beyond Traditional Metrics

Your experience in computer science provides a strong foundation in evaluating systems, from algorithmic complexity to software testing and traditional machine learning metrics like accuracy and F1-score. However, agentic AI introduces new challenges that render these approaches insufficient on their own.

Evaluating LLM-based Applications

To understand these challenges, let's watch a segment from a talk by Databricks on evaluating LLM-based applications. It clearly contrasts traditional ML evaluation with the complexities of generative systems.

Watch from 10:14 to 13:48. Pay attention to the three key reasons why old evaluation paradigms don't fit: 1) you don't control the training data, 2) outputs are often qualitative with no single 'correct' answer, and 3) agents are expected to handle a wide variety of tasks.

As the video explains, we can't simply measure accuracy against a known test set in the same way. The success of an agent that summarizes a document or negotiates a contract is inherently qualitative and context-dependent. Our evaluation framework must account for this ambiguity.

2. The Core Principle: Connecting Evals to ROI

An agentic workflow doesn't exist to be "faithful" or have high "relevance" scores; it exists to achieve a business goal. These technical metrics are only useful if they are reliable proxies for real-world value. Without this connection, you risk optimizing for metrics that your client doesn't care about.

LLM Evals Framework That Predicts ROI: A Step-by-Step Guide

This article from Confident AI drives this point home. It argues that the failure to connect evaluation metrics to business goals is the primary reason why many teams fail to get value from their evaluation efforts.

Read the first section, 'Why Your LLM Evals Must Correlate to ROI'. Focus on the distinction between development metrics (like correctness, tool use) and production outcomes (like cost savings, conversion rates).

The key takeaway is that our goal isn't to invent new business metrics but to establish a clear, demonstrable correlation between the technical metrics we can control during development and the production outcomes that define success for the business.

3. A Hierarchy of Success Criteria

To build this correlation, it's helpful to think of success criteria in a three-tiered hierarchy, moving from the client's high-level objectives down to the specific behaviors of individual agents.

Tier 1: Business KPIs (The "Why")

This is the top level, representing the client's goals. Defining these requires you to think like a business consultant. What pain point is the agentic workflow solving?

Efficiency & Cost Reduction: Reducing manual labor, decreasing time-to-completion.
Value Generation & Revenue Increase: Enabling new capabilities, improving sales conversion, increasing output quality.
Risk Mitigation: Ensuring compliance, reducing errors, improving security.

How Agentic AI Is Transforming Automation Workflows For Enterprises

For concrete examples of how agentic AI delivers on these KPIs, let's listen to Chris Halenbeck, SVP of AI at Boomi. He provides two powerful case studies that perfectly illustrate measurable business outcomes.

Watch from 11:25 to 13:46. Notice how he quantifies the outcomes: one is a direct efficiency gain ('saved 27,000 hours'), and the other is about unlocking new value ('something that was just physically not possible to be done before').

These are the outcomes you sell. The following images provide excellent, structured lists of such KPIs for agentic systems. They serve as a great reference when brainstorming with clients.

Core KPI Categories for Agentic AI — This image categorizes high-level KPIs for agentic AI into six key areas, from task completion and accuracy to operational ROI and business impact.

Multi-Agent System ROI Measurement Frameworks — This table provides a detailed framework for measuring the ROI of multi-agent systems, linking KPIs to specific measurement methods and even typical improvement ranges.

Tier 2: System-Level Metrics (The "What")

Once you have the business KPI, you need to translate it into measurable properties of the entire agentic system. The CLASSic Framework provides a robust and comprehensive structure for this.

LLM Evaluation Metrics, Best Practices and Frameworks - Aisera

The article 'LLM Evaluation Metrics, Best Practices and Frameworks' introduces this useful acronym. It helps ensure a holistic evaluation of your system's performance in a real-world enterprise context.

First, read the brief section 'The Rise of LLM Evaluation for AI Agents' to frame the problem. Then, read the section on the 'CLASSic Framework' to understand its five pillars. Finally, read 'Model Evaluations vs System Evaluations' to solidify the crucial distinction between evaluating a raw LLM and evaluating an LLM-powered system designed for a specific task.

The CLASSic Framework consists of:

Cost: API usage, token consumption, infrastructure expenses.
Latency: End-to-end response time for the entire workflow.
Accuracy: Precision of the final output or decision.
Stability: Robustness across different inputs and conditions.
Security: Resistance to prompt injection, data leaks, and other adversarial attacks.

These five pillars provide a bridge from a vague business goal like "improve customer support" to a concrete system goal like "resolve support tickets with <1 min latency, >95% accuracy, and at 30% of the current cost."

Tier 3: Agent-Level Metrics (The "How")

These are the granular, often technical metrics that you, the system designer, will use to iterate and improve the system. They measure the performance of individual components (LLM calls, tool use, agent outputs) that contribute to the system-level metrics.

Examples include:

Faithfulness / Groundedness: Does the agent's response stay grounded in the provided context? (Impacts Accuracy)
Answer Relevancy: Is the response relevant to the user's query? (Impacts Accuracy)
Tool Use Accuracy: Does the agent call the correct tools with the correct parameters? (Impacts Accuracy and Cost)
Toxicity/Bias: Does the agent produce harmful or biased content? (Impacts Security and Stability)
Response Format Adherence: Does the agent output JSON when requested? (Impacts Stability)

These are the development metrics we discussed earlier. Your task during development is to tune the system to improve these metrics, based on the hypothesis that they will, in turn, improve the system-level CLASSic metrics and ultimately deliver the business KPI. The next section describes how to validate that hypothesis.

4. The Process: From Business Goal to Validated Metric

Let's synthesize this into a step-by-step process for defining success criteria for a client project.

Scenario: A client wants to automate the process of generating market research reports. The current manual process is slow and expensive.

Step 1: Define the Business KPI (Tier 1)
First, work with the client to quantify their pain.

Business Goal: Reduce the cost and time required to produce a market research report while maintaining quality.
KPIs:
- Reduce cost per report by 50% (from an average of $2,000 in analyst time to $1,000 in compute and oversight).
- Reduce turnaround time from 2 weeks to 2 days.
- Maintain client satisfaction score of >8/10 on report quality.

Step 2: Translate to System-Level Metrics (Tier 2 - CLASSic)
Next, translate these KPIs into targets for your agentic workflow.

Cost: Total token cost < $500 per report.
Latency: End-to-end generation time < 48 hours.
Accuracy: All statistics cited in the report must be attributable to the source documents. The final report must pass a quality rubric (e.g., covers all required sections, contains no factual errors).

Step 3: Decompose into Agent-Level Metrics (Tier 3)
Identify the metrics for the individual agents (e.g., SearchAgent, AnalysisAgent, WriterAgent).

SearchAgent:
- Tool Use: Correctly formulates search queries.
- Relevance: Retrieves relevant documents (measured by precision/recall on a test set).
AnalysisAgent:
- Faithfulness: Extracts key statistics without hallucination.
- Correctness: Performs calculations accurately.
WriterAgent:
- Coherence: Writes a well-structured, readable report.
- Faithfulness: Ensures all claims are traceable to the analysis provided.

Step 4: Validate the Metric-Outcome Correlation
This is the most critical step. How do you ensure that improving, for example, the Faithfulness of the AnalysisAgent actually improves the client's satisfaction with the report?

This requires building an initial "golden dataset" and using human judgment to establish a baseline that connects your metrics to the desired business outcome.

LLM Evals Framework That Predicts ROI: A Step-by-Step Guide

Let's return to the 'LLM Evals Framework' article for a detailed guide on this validation process. It provides a practical, iterative method for ensuring your automated evals are aligned with real-world success.

Read the sections 'How to Build Your LLM Evals Metrics' and 'Validate Whether Your LLM Evals Predict Real Outcomes'. Focus on the core idea: humans should label outcomes (e.g., 'this report is good/bad for the client'), not metric scores. The goal is to tune your automated metrics until their pass/fail rates match the human judgment of the outcome.

By following this process, you create a system where you can confidently state: "Our evaluations show that a Faithfulness score of >0.9 correlates with a 95% human approval rate on report sections. By improving this metric, we are directly improving the quality of the final deliverable and ensuring we meet the client's satisfaction KPI."

Conclusion

In this lesson, we've laid the groundwork for a robust, evidence-based approach to agentic system development. You've learned how to translate vague business needs into a concrete, hierarchical set of measurable criteria.

Key Takeaways:

Evaluation is about proving value: The ultimate goal is to connect technical performance to business outcomes like ROI.
Use a metric hierarchy: Break down high-level Business KPIs into measurable System-Level Metrics (using a framework like CLASSic) and then into granular Agent-Level Metrics.
Start with the business outcome: Always begin by understanding and quantifying the client's goal or pain point.
Correlate, don't assume: The most critical step is to validate that your development metrics are reliable proxies for the desired business outcome, using human-labeled outcomes to ground your evaluation system.

You are now equipped to define what "success" means for any agentic workflow you design. This clarity is essential for guiding your development efforts and for communicating the value of your work to clients.

In our next lesson, we will take the logical next step. Now that we have our success criteria, we need to learn how to design and construct an evaluation dataset with test cases that can rigorously and reliably measure our system against those criteria.