Troubleshooting Agent Information Gaps

Introduction

In our last lesson, we designed a "compiler" for context—a systematic pipeline to select, compress, and organize information into a final prompt for the LLM. This assembly strategy is crucial for managing token limits and maximizing the signal-to-noise ratio. However, this process assumes that the source information is complete and accurate. What happens when it isn't?

This lesson addresses that critical question, directly fulfilling the learning outcome: Identify gaps in an agent's information environment that would cause poor or unreliable outputs. We will shift from context assembly to context diagnostics. Think of this as root cause analysis for agentic systems. When an agent fails, your first question shouldn't be "Is the model dumb?" but rather, "What information was it missing?" Your background in computer science, particularly in debugging and systems analysis, provides a strong foundation for this process.

1. The "Context Failure" Paradigm

A fundamental shift in perspective is necessary when building agentic systems. Failures are rarely arbitrary. More often than not, they are logical consequences of a flawed information environment.

This idea—that failures stem from the context—is our guiding principle. An agent's output is a function of its model and its context. Since we treat the model as a fixed component (a "black box" of sorts), our primary lever for improving reliability is engineering the context. Identifying gaps in that context is the first step in the debugging cycle.

2. A Taxonomy of Information Gaps

When an agent produces a poor or unreliable output, the "gap" in its information environment typically falls into one of three categories. Understanding these categories helps you quickly narrow down the source of the problem.

1. Knowledge Gaps: The information the agent needs simply does not exist in any of its accessible sources (e.g., its RAG database, toolset, or memory). The agent is being asked a question it fundamentally cannot answer.
2. Contextual Gaps: The information does exist in an accessible source, but it was lost, distorted, or buried during the context assembly process. The context "compiler" we designed in the last lesson has a bug.
3. Implicit Knowledge Gaps: The information exists only as an unwritten rule or assumption within the business process. It's the "tribal knowledge" that humans use but which was never explicitly documented for the agent.

The image below illustrates how these gaps can lead to one of the most common failure modes: hallucination.

Let's explore how to diagnose each of these gap types.

3. Diagnosing Contextual Gaps: When Information Gets Lost

Contextual gaps are often the most common and are directly related to the assembly strategies we discussed previously. The information was available, but it didn't make it into the final prompt effectively.

A primary cause of this is the "lost in the middle" phenomenon, where an LLM fails to attend to information buried deep within a long context. Your CS background will appreciate that this isn't just a quirk; it has architectural roots in the transformer model itself.

As the video explains, even with large context windows, there's no guarantee the model will use all the information equally. This leads to several common failure modes you can diagnose:

• Failure Symptom: The agent ignores a piece of retrieved information or a detail from earlier in the conversation.

  • Diagnostic Check: Examine the final assembled prompt. Where was the critical information placed? If it was sandwiched between large, less relevant blocks of text (e.g., verbose tool logs or other RAG results), it was likely "lost in the middle." This points to a failure in your reranking or prioritization strategy from the last lesson.

• Failure Symptom: The agent seems unaware of a crucial detail from a long document or a lengthy conversation history.

  • Diagnostic Check: Review your context compression steps. Did a summarization module create a lossy summary, omitting the key detail? As the "Context Engineering" article notes, "summaries are only as good as the model producing them, and important details can occasionally be lost." This indicates your compression strategy is too aggressive.

• Failure Symptom: An agent retrieves a relevant fact, but the fact is incomplete or lacks necessary surrounding context.

  • Diagnostic Check: Examine your RAG chunking strategy. It's likely that a critical piece of information was split across two chunks, and the retrieval system only pulled one. The article on Context Engineering discusses this in the "Chunking Strategies" section, warning that if chunks are too small, "important context might get split up and not retrieved together."

4. Diagnosing Implicit Knowledge Gaps: The Unwritten Rules

This is often the most challenging category to diagnose because the failure is not in the code or the data, but in the gap between the documented process and the real-world process. The agent follows its instructions perfectly, yet the output is naive or incorrect from a business perspective.

Diagnosing this requires moving beyond analyzing logs and traces to analyzing the business logic itself. This is where your value as a system designer, rather than just an implementer, becomes paramount.

Failure Symptom: The agent produces an output that is technically correct according to its system prompt but is practically useless or violates a business norm. For example, a sales outreach agent drafts an email that is grammatically perfect but has the wrong tone for a C-level executive.

Diagnostic Process:

1. Confirm Technical Correctness: First, verify that the agent did follow all of its explicit instructions. If it did, the problem isn't a bug in the agent's reasoning but a flaw in its world model.
2. Engage a Domain Expert: This is a Human-in-the-Loop process. You must work with someone who understands the business domain to identify the "unwritten rule" the agent violated.
3. Translate Implicit to Explicit: The goal is to articulate this rule and add it to the agent's static context (system prompt). In the sales example, you might add a rule like: "When writing to C-level executives, adopt a formal and concise tone. Avoid using emojis and casual language."

For your client-facing work, identifying these implicit knowledge gaps is a high-value activity. It forces clients to clarify their own processes and turns "tribal knowledge" into a documented, automated asset.

5. A Formal Framework for Failure Analysis

To make this diagnostic process more systematic, especially in complex multi-agent systems, we can use a formal taxonomy of failures. This moves us from ad-hoc debugging to a structured audit.

The MASFT provides a checklist for your root cause analysis. When an agent system fails, you can analyze the execution trace and map the failure to one of these categories:

• Specification & System Design Failures: This category directly corresponds to our Implicit Knowledge Gaps. A failure here, such as an agent disobeying a constraint it was never given, means the initial specification was incomplete.
  • Example: A coding agent generates insecure code because its prompt didn't include security requirements.
• Inter-Agent Misalignment Failures: These are often Contextual Gaps that occur during handoffs. The article's example of one agent knowing the login format but not telling another is a classic case of an inadequate scoping of context during delegation.
  • Example: A router agent correctly identifies a user query as a "billing issue" but only passes the query text to the billing agent, omitting the user's ID from the conversational history.
• Task Verification & Termination Failures: This happens when an agent lacks the information needed to judge its own work. The example of an agent checking if code compiles but not if it works is a gap in its verification context. It lacks the criteria for success.
  • Example: A research agent is asked to find a company's revenue. It finds the first number it sees and reports it, without the contextual knowledge to verify if that number is from the correct fiscal year or currency.

Using this framework, you can methodically pinpoint the type and location of the information gap, which is the first step toward a targeted fix.

Conclusion

In this lesson, we developed a diagnostic mindset for identifying why agentic systems fail. We've established that the root cause is often not a mysterious model flaw, but a tangible gap in the agent's information environment.

Key Takeaways:

• Think Context First: When an agent fails, start with the assumption of a "context failure," not a "model failure."
• Categorize the Gap: Failures can be traced to Knowledge Gaps (missing data), Contextual Gaps (flawed assembly), or Implicit Knowledge Gaps (unwritten rules).
• Diagnose Systematically: Use agent traces and output analysis to detect issues like "lost in the middle," over-compression, or poor chunking.
• Make the Implicit Explicit: A key design task is to work with domain experts to uncover and codify the unwritten rules of a business process into the agent's instructions.
• Use Formal Frameworks: Taxonomies like MASFT provide a structured, evidence-based checklist for performing root cause analysis in complex multi-agent systems.

Preview of the Next Lesson

We now know how to assemble context and how to diagnose gaps when the source information is flawed or missing. The next logical step is to focus on the format of the information we provide. Even if the right information is present, presenting it in a confusing or ambiguous way can lead to unreliable outputs.

In our next lesson, we will learn how to structure input data formats to maximize agent comprehension and output consistency. This will cover techniques for formatting user inputs, tool outputs, and retrieved data to make them as clear and machine-readable as possible.