Diagnosing Multi-Agent Failures with Logging & Tracing

Introduction

Welcome to the final lesson in our module on observability and debugging. In our previous lessons, we have meticulously built an observability stack, first by implementing distributed tracing to visualize request flows and then by adding metrics to monitor token usage and latency. You now have the instruments to see what is happening in your agentic system. Today, we put those instruments to the test.

This lesson is purely practical. We will shift from building tools to using them for their ultimate purpose: diagnosing and resolving failures. We will simulate a realistic multi-agent failure, and I will guide you through a systematic investigation process using the very traces and metrics you've learned to implement. This is the crucial skill that separates a prototype from a production-ready service, allowing you to maintain reliability and user trust.

A Taxonomy of Failure in Agentic Systems

Before we dive into our specific scenario, it's helpful to understand the common failure modes in multi-agent systems. When a request goes wrong, the cause usually falls into one of these categories:

• LLM-Level Failures: The model itself misbehaves. This includes generating factually incorrect information (hallucinations), failing to follow instructions, or producing malformed outputs like invalid JSON when a specific schema is required.
• Tool-Execution Failures: An agent successfully invokes a tool, but the tool fails. This is often due to external factors like network errors, API rate limits, authentication issues, or simply bugs in the tool's own logic.
• Orchestration & Logic Failures: The error lies in the "connective tissue" between agents. Examples include a supervisor agent delegating to the wrong worker, failing to propagate critical context, or getting stuck in a loop between two agents.
• Infrastructure Failures: The underlying platform causes a problem. This could be a serverless function timing out on a long-running task, a database connection being dropped, or out-of-memory errors.

Understanding these categories helps you form hypotheses as you begin your investigation.

The Scenario: A Faulty Research Report

Imagine a user submits a request to our system: "Generate a research report on Vercel, including a summary of their recent product announcements."

Our orchestrator routes this to a supervisor-agent, which delegates the sub-task "Find recent product announcements for Vercel" to a research-agent. The research-agent has a tool, search_web, which in turn calls an external search engine API.

After a long wait, the user receives a report that is incomplete and nonsensical. The summary is garbled, and there's no mention of product announcements. This is our starting point—the user-facing symptom.

The Investigation: From Symptom to Root Cause

As a senior developer, you know the first step is to reproduce the issue or find the exact request that failed. Let's assume the user provides a request ID, which we've wisely propagated as the traceId.

Step 1: Find and Analyze the Trace

We'll use our observability platform (e.g., Langfuse, which is shown in the images) to pull up the full trace for this request. The initial view might look something like this.

Upon examining the trace, we immediately look for visual cues of failure:

• Error Status: Spans marked in red indicate an explicit error was thrown and caught.
• Unusual Latency: A span that is unexpectedly long could indicate a timeout or a performance bottleneck. A span that is too short might mean a process exited prematurely.
• Missing Spans: If the research-agent was supposed to call the search_web tool, but there is no corresponding tool_call span, it points to a logic failure within the agent.

In our scenario, we scan the trace and see a tool_call span for search_web marked with an error.

Step 2: Drill Down into the Failed Span

Now that we've identified the point of failure, we click on the erroneous span to inspect its attributes, events, and logs. Here, we'd find attributes like:

• tool.name: search_web
• tool.arguments: {"query": "Vercel recent product announcements"}
• error.message: API rate limit exceeded. Please try again later.
• error.type: 429 Too Many Requests

We have found the root cause: the external search API used by our tool is rate-limiting our requests. The research-agent received this error, couldn't complete its task, and returned a poorly formed or empty result to the supervisor-agent. The supervisor, not having a robust strategy for handling failed sub-tasks, attempted to synthesize a report from incomplete data, leading to the garbled output the user saw.

Mitigating Failures with Durable Workflows

This type of transient failure—rate limits, network blips, temporary service unavailability—is extremely common in distributed systems. While we could implement custom retry logic with exponential backoff for every tool, a more robust and scalable solution is to use a durable workflow engine.

The Vercel Workflow Development Kit, which is built to integrate with the AI SDK, is designed precisely for this. It allows you to transform your agentic logic into durable, resumable, and observable workflows.

By wrapping our tool calls and agent invocations in useStep, the workflow engine automatically handles state management and retries. If our search_web tool fails with a 429 error, the engine can automatically wait and retry the step.

Using a durable workflow engine transforms our debugging process. Instead of just identifying the failure, the trace might now show a failed step followed by a successful retry, with the overall workflow completing correctly. The problem fixes itself.

Hardening the Observability Layer

Our investigation was successful, but a production system requires a production-grade observability layer. Simply logging everything naively can introduce security risks and even cause application crashes.

This article provides critical, battle-tested patterns for instrumenting agent swarms safely.

By adopting these patterns, you ensure that your observability system is not only powerful but also secure, stable, and cost-effective.

Conclusion

In this lesson, you've put your observability knowledge into practice, transforming from a builder of tools to a skilled investigator. You've learned how to follow the breadcrumbs from a high-level symptom down to the root cause buried deep within a multi-agent interaction.

Key Takeaways:

• Systematic Debugging: The process of diagnosing failures is systematic: start with the symptom, find the corresponding trace, identify anomalies (errors, latency), and drill into the specific span attributes to find the root cause.
• Traces Tell the Story: A well-instrumented trace is a narrative of your request. It's the single most powerful tool for understanding the complex, distributed behavior of an agentic system.
• Durability Solves Transient Failures: Many common errors (rate limits, network issues) can be automatically mitigated by building on a durable workflow engine like the Vercel Workflow DevKit.
• Production-Grade Observability is Hardened: A robust observability pipeline must be secure (redacting PII), stable (handling circular JSON), and cost-effective (avoiding high-cardinality data).

You have now completed the Observability and Debugging module. You are fully equipped to not only build a complex agentic system but also to confidently operate it in a production environment. In our next module, we will begin the final phase of our course: End-to-End System Integration, where we will assemble all the components we've built into a single, cohesive, multi-tenant application.