Pinpointing Improvement Areas in LLM Components

Hello! Welcome to the next lesson in our module on "Evaluation and Evidence-Based Improvement."

In our last session, we focused on executing a benchmark run, systematically collecting a rich dataset of agent outputs, execution traces, and performance metrics against our golden evaluation dataset. You now have the raw data—the evidence—of your system's performance.

Today, we'll learn how to turn that data into insight. Our learning outcome is to interpret evaluation results to identify specific components (prompts, retrieval, tools) that need improvement. This is the crucial diagnostic step where you move from knowing that the system failed to knowing precisely why and where. For you, as an architect of agentic AI services, mastering this root cause analysis is what enables you to build reliable, production-grade systems and confidently deliver them to clients.

1. The Foundation: Systematic Error Analysis

Looking at aggregate scores like "75% accuracy" is rarely sufficient. A high-level score can hide critical, systemic failures. The key to effective diagnosis is error analysis: a methodical process of reviewing individual failures to build a qualitative understanding, which then informs quantitative measurement and targeted improvements.

Think of it as debugging a complex software system. You wouldn't just look at the final crash report; you'd analyze stack traces, log files, and memory dumps to pinpoint the faulty module. The same principle applies here.

The process outlined in the reading is a powerful loop:

1. Gather Traces: You've already done this by running your benchmark. The first step is to sample the traces that show failures. You can start with a random sample or use more efficient strategies like filtering by negative user feedback, sorting by metrics like high latency, or clustering queries to find problematic patterns.
2. Open Coding (Journaling): A domain expert (which will often be you) manually reviews each failed trace and writes open-ended notes describing the failure. For example: "The agent retrieved a document about the 2023 policy, but the query was about 2024," or "The agent called the search_products tool twice with the same irrelevant query."
3. Axial Coding (Creating a Failure Taxonomy): This is the most critical step for our goal. You group the open-ended notes into distinct categories. This is where you connect the dots, transforming individual observations into a "failure taxonomy." For instance, multiple notes about retrieving outdated documents might be grouped into a category called Retrieval_Stale_Content. Notes about incorrect tool usage could become Tool_Misuse_Incorrect_Parameters.
4. Count & Prioritize: By counting the instances in each category, you get a data-driven view of your system's most common failure modes. This tells you where to focus your improvement efforts for the biggest impact.

This process moves you from anecdotal evidence to a structured, quantifiable understanding of your system's weaknesses.

2. A Framework for Diagnosis: Mapping Failures to Components

Once you have a failure taxonomy from your error analysis, the next step is to map these failure categories to the specific components of your agentic system. To do this, we need a clear architectural model of the system itself.

As the paper details, we can attribute most agentic system failures to one of four pillars. Let's explore how to diagnose issues in each, using the data you collected in the previous lesson.

Pillar 1: The LLM (Reasoning and Instruction Following)

This pillar concerns the core reasoning capabilities of the agent. Failures here are often about the LLM not "thinking" correctly or not following your rules.

• Common Failure Modes:
  • Instruction Ignoring: The agent fails to adhere to constraints in your system prompt (e.g., "You must not give financial advice").
  • Policy Violation: The agent bypasses required steps in a workflow (e.g., terminating a resource without first checking a policy, as noted in Table 7 of the paper).
  • Flawed Reasoning: The agent's chain of thought is illogical or it makes incorrect deductions.
• How to Diagnose:
  • Review Execution Traces: Your primary tool is the trace log. Examine the sequence of LLM calls, the prompts sent, and the reasoning steps generated.
  • Check Against Ground Truth: Compare the agent's action sequence to the expected sequence defined in your evaluation test case. Did it skip a mandatory check_dependency step? The Tool sequence and Policy Adherence metrics in Table 3 of the paper are direct measures of this.
  • Use Judge LLMs: As discussed in the paper, you can use an LLM-as-Judge to score the Reasoning or Safety of an agent's trace, flagging logical gaps or policy breaches.

Pillar 2: Memory (Retrieval and Context)

For RAG-based agents, this is one of the most common sources of failure. The agent's final answer is highly dependent on the quality of the information it retrieves.

• Common Failure Modes:
  • Irrelevant Retrieval: The retrieved documents have nothing to do with the user's query.
  • Missing Context: The agent fails to retrieve the document that contains the correct answer, even though it exists in the knowledge base.
  • Conflicting Sources: The agent retrieves multiple documents with contradictory information, leading to a confused or incorrect answer.
• How to Diagnose:

The key is to interpret the relationship between retrieval metrics.

As the video explains, you can diagnose specific retrieval issues by looking at the combination of metrics:

• High Recall, Low Precision: This means you retrieved the right information, but it was buried in a lot of irrelevant noise.
  • Likely Cause: Your chunks are too large, or your retrieval query is too broad.
  • Component to Improve: Chunking strategy (try smaller chunks) or the retrieval query generation part of your prompt.
• Low Recall, High Precision: This means the chunks you retrieved were relevant, but you missed the most important ones.
  • Likely Cause: Your embedding model isn't capturing the semantic meaning of the query correctly, or your chunks are too small and lack context.
  • Component to Improve: Embedding model (try a different one) or chunking strategy (try larger chunks or add overlap).

The troubleshooting flowchart below provides a great visual summary of this diagnostic process.

Pillar 3: Tools (Action and Execution)

This pillar relates to the agent's ability to interact with the outside world via functions or APIs.

• Common Failure Modes:
  • Incorrect Tool Selection: The agent chooses the wrong tool for the task (e.g., calling get_weather for a stock price query).
  • Incorrect Parameter Mapping: The agent calls the right tool but with wrong or malformed parameters (e.g., get_stock_price(ticker='GOOGL', date='yesterday') when the tool expects a specific date format).
  • Looping: The agent gets stuck repeatedly calling the same tool without making progress.
  • Faulty Error Handling: The agent fails to recover when a tool call returns an error.
• How to Diagnose:
  • Analyze Tool Call Traces: The execution log is your source of truth. Look at the sequence of tool_name, tool_input, and tool_output for each step.
  • Use a Transition Failure Matrix: For complex, multi-step agentic workflows, it can be hard to see where things go wrong. A transition failure matrix is a powerful visualization for identifying failure hotspots.

By creating a matrix of last_successful_state vs. first_failure_state, you can immediately see patterns like "25% of all failures occur when transitioning from PlanGeneration to ToolSelection." This tells you to focus your debugging efforts on the Tool Selection component.

Pillar 4: System-Level Issues (Cost, Latency, Guardrails)

Beyond functional correctness, a production system must be efficient and safe. Your benchmark data can reveal these non-functional issues.

• Common Failure Modes:
  • Excessive Cost: The workflow uses too many tokens or makes too many expensive API calls.
  • High Latency: The agent takes too long to respond.
  • Guardrail Violations: The agent attempts an action that is explicitly forbidden.
• How to Diagnose:

• Cost Analysis: Your benchmark trace data should include token counts and tool costs. Aggregate this data. Are you using a flagship model like GPT-4 Turbo for a simple classification task that a smaller, cheaper model could handle? Are you embedding redundant data? This analysis points directly to model selection and input data processing as components to improve.
• Latency Analysis: Your traces should have timestamps for each step. By analyzing these, you can create a performance profile of your workflow and identify bottlenecks. Is a particular tool call slow? Is the LLM taking too long to reason?
• Guardrail Analysis: Your system logs should record any time a guardrail is triggered. A high frequency of violations for a specific rule suggests either the agent is consistently trying to "break the rules" (a prompt/instruction following problem) or the guardrail itself is too restrictive.

3. From Diagnosis to Prioritization

After conducting your error analysis and mapping failures to the four pillars, you'll have a prioritized list of issues. For example:

1. Memory/Retrieval (45% of failures): High recall but very low precision is causing noisy context. The likely culprit is the chunking strategy.
2. Tools (30% of failures): The update_crm tool fails frequently. The transition failure matrix shows the issue is in the ParameterExtraction step. The problem is in the prompt that extracts tool parameters.
3. LLM (15% of failures): The agent sometimes provides responses that are too verbose, violating a persona constraint. The problem is in the system prompt.
4. System (10% of failures): Latency is high during peak hours. Trace analysis shows the bottleneck is a specific external API call made by a tool. The problem is the tool's implementation.

This structured output allows you to move forward with targeted, evidence-based improvements. You're no longer guessing; you're operating with surgical precision.

Conclusion

In this lesson, we've bridged the gap between collecting evaluation data and taking action. You've learned how to move beyond simple pass/fail scores to a sophisticated diagnostic process that gives you a deep understanding of your agent's failure modes.

Key Takeaways:

• Error analysis is the core of interpretation: Use the Open Coding -> Axial Coding -> Taxonomy loop to systematically categorize and quantify failures.
• Use a structured diagnostic framework: The four pillars (LLM, Memory, Tools, System) provide a robust mental model for mapping failures to specific architectural components.
• Interpret metrics in context: Don't just look at a single number. The relationship between metrics (like high recall and low precision) is what reveals the root cause.
• Analyze the full execution trace: Your most powerful diagnostic tool is the detailed log of the agent's entire reasoning and action process.
• Use advanced tools for complex workflows: Techniques like transition failure matrices are essential for pinpointing failure hotspots in multi-agent or multi-step systems.

You are now equipped not only to measure your system's performance but to understand it on a fundamental level. This ability to diagnose and articulate the root cause of failures is a hallmark of an expert systems architect.

In our next lesson, we will look at how to use these evaluation insights proactively. We will learn how to incorporate a quality gate into an orchestrator that evaluates agent output against defined standards before proceeding, effectively building a self-monitoring mechanism into your agentic system.