LLM Failure Modes and Agent Design

Introduction

Welcome to our third lesson. In the previous two lessons, we explored the fundamental probabilistic nature of Large Language Models (LLMs) and the practical levers we can use—like temperature, token limits, and context engineering—to guide their behavior. We've learned how to control the machine.

Now, we must learn how to distrust it.

No matter how well you craft a prompt or tune parameters, an LLM's core architecture leads to inherent, unavoidable failure modes. Building a robust agentic system isn't just about elegant design; it's about anticipating and engineering defenses against these failures. This is the difference between a clever demo and a production-ready system you can sell to a client.

Today's learning outcome is to identify the key failure modes of LLMs (hallucination, drift, and instruction ignoring) and their implications for agent design. We will dissect each failure, understand its root cause, and, most importantly, explore the architectural patterns and design principles required to build resilient agentic workflows.

A Framework for Failure: Cognitive Deficits

Before diving into specific failures, it's useful to have a mental model for why they occur. LLMs are not databases or conventional programs; they are complex systems with emergent, sometimes counter-intuitive behaviors.

Andrej Karpathy offers a compelling analogy. He describes LLMs as "people spirits"—entities trained on vast amounts of human text that exhibit a human-like psychology, complete with both superhuman abilities and significant cognitive deficits.

Karpathy's "cognitive deficits" provide a useful taxonomy:

• Hallucination: The model invents facts. This is the most well-known failure mode.
• Jagged Intelligence: The model is superhuman at some tasks but makes elementary mistakes on others. This inconsistency is a major challenge for reliability.
• Anterograde Amnesia: The model cannot natively form new long-term memories. Its "memory" is limited to its context window, which is constantly being refreshed. As we will see, this is a primary driver of a more subtle failure mode called drift.

With this framework in mind, let's dissect these failures from a system design perspective.

The Unholy Trinity of LLM Failures

We will focus on three categories of failure that have the most significant impact on agentic system design: hallucination, instruction ignoring, and drift.

1. Hallucination: The Plausible Lie

A hallucination is the generation of information that is plausible and grammatically correct but factually wrong or nonsensical.

• Root Cause: LLMs are autoregressive models. Their fundamental objective is to predict the next most probable token in a sequence, not to be truthful. When the training data contains conflicting information, or when the model lacks specific knowledge, it may generate a statistically likely but factually incorrect sequence.

• Implications for Agent Design: You cannot "fix" hallucination, but you can build systems that are resilient to it. The key principle is grounding.

  1. Retrieval-Augmented Generation (RAG): This is the primary strategy. Instead of relying on the model's parametric memory, you provide relevant, factual information in the prompt's context at inference time. We previewed this in our last lesson as "semantic memory" and will dedicate Module 6 to its architecture.
  2. Tool Use: An agent should not be expected to "know" things like the current price of a stock or the details of a flight. Instead, it should be given tools (APIs) to look up this information from authoritative sources. This outsources the need for factual recall to a deterministic system.
  3. Verification Steps: In a multi-agent workflow, one agent's task can be to verify the output of another. For example, a "Fact-Checking Agent" could take the output of a "Writer Agent" and use a search tool to validate its claims before the content is finalized.

2. Instruction Ignoring: The Disobedient Agent

This failure occurs when an agent fails to follow specific constraints, roles, or formatting requirements laid out in its system prompt. This can range from minor formatting errors to critical safety violations.

• Root Cause: This is often a function of prompt complexity and conflicting signals. An LLM weighs all parts of its context. If a user's request appears to contradict a system instruction, or if an instruction is ambiguous, the model may "choose" to follow the path of least resistance based on patterns in its training data, rather than your explicit rule.

• Implications for Agent Design: A recent research paper introduces a comprehensive taxonomy for failures in multi-agent systems (MAS). It correctly frames many failures not as intrinsic model limitations, but as flaws in system design.

As the paper highlights, "Disobey task specification" (FM-1.1) and "Disobey role specification" (FM-1.2) are critical failure modes. The key design implications are:

1. Constraint-Based Prompting: Instructions should be unambiguous. Use clear language, and explicitly state what the agent should not do (negative constraints). We will explore this in Module 4.
2. Structured Outputs: Forcing the model to generate output that conforms to a specific JSON schema is one of the most powerful techniques for ensuring reliability. If the output doesn't validate against the schema, you know a failure has occurred and can trigger a retry or escalation.
3. Role Specialization: In a multi-agent system, creating highly specialized agents with narrow, well-defined roles reduces ambiguity and the likelihood of instruction-ignoring. A "Code-Writing Agent" should only write code; a "Reviewing Agent" should only review it.

3. Drift: The Slow Decay

Agent drift is the progressive degradation of an agent's behavior, decision quality, or coherence over long interaction sequences. It's a subtle, long-term failure mode that is invisible in short tests but catastrophic for production systems.

• Root Cause: Drift is an emergent property arising from the interplay of an LLM's "anterograde amnesia" and the autoregressive nature of agentic loops. A theoretical paper on this topic provides an excellent framework for understanding its mechanisms.

The key mechanisms causing drift are:

1. Context Window Pollution: As an agent's conversation history grows, its context window fills with stale, irrelevant information, degrading the signal-to-noise ratio for its current decision.
2. Distributional Shift: The agent operates in a narrow domain, but its outputs gradually shift back toward the broader distribution of its training data.
3. Reinforcement through Autoregression: Small errors or stylistic quirks in an agent's output become part of its future input (via conversation history), creating a feedback loop that amplifies the deviation over time.

• Implications for Agent Design: Combating drift requires moving beyond simple prompt-and-response and building a robust monitoring and maintenance infrastructure.

1. Systematic Evaluation: You must have a "golden dataset" of test cases to run your agent against periodically. By comparing outputs over time, you can detect regressions and drift before they impact production. The LINK video touches on this in its discussion of evaluation and monitoring (from 49:53).
2. Sophisticated Memory Management: A simple conversation history is not enough. You need systems that can summarize, prune, and selectively retrieve information from long-term memory to prevent context pollution.
3. Behavioral Anchoring: Periodically re-grounding the agent with its core instructions or few-shot examples of ideal behavior can break the autoregressive feedback loop and reset accumulating drift.

From Identification to Mitigation

Recognizing these failure modes is the first step. The next is designing systems that can handle them. This involves building in layers of orchestration, verification, and human oversight. Andrej Karpathy terms this "partial autonomy," where we keep the fallible AI "on a leash."

When a failure does occur, a production system needs a process to identify, log, and analyze it. This is where failure attribution becomes critical.

The MAST paper (LINK) provides an extensive, fine-grained taxonomy of 14 failure modes that can serve as a detailed checklist during the design and testing of your agentic systems. For example, their list includes failures like "Premature termination," "Information withholding," and "Reasoning-action mismatch"—all subtle but critical issues you must anticipate.

Conclusion

In this lesson, we confronted the inherent fallibility of LLMs and framed it not as a roadblock, but as a core system design challenge. A successful agent architect doesn't just build for success; they engineer for failure.

Key Takeaways:

• LLM Failures are Systematic: Hallucination, instruction ignoring, and drift are not random bugs but predictable consequences of the underlying technology.
• Defense is Architectural: You cannot eliminate these failures, but you can mitigate their impact through deliberate design choices: grounding with RAG, tool use, structured outputs, specialized agents, robust monitoring, and sophisticated memory systems.
• Design for Partial Autonomy: The most robust agentic systems today operate with a human in the loop, using an "autonomy slider" to manage the trade-off between speed and reliability. Your job as a designer is to build the validation and oversight mechanisms that make this collaboration effective.

Preview of the Next Lesson:

We've now identified what an agent is (Lesson 1), what we can control (Lesson 2), and what can go wrong (Lesson 3). But how does this translate into a coherent system? In our next lesson, we will begin Module 2 by answering the question: How do we distinguish between using an LLM as a tool versus as an autonomous agent? This will establish the fundamental building block for designing agentic workflows.