Understanding LLM Failure Modes

Welcome to your sixth lesson in the AI Alignment and Safety module.

In our previous lesson, we dove into the "black box" with mechanistic interpretability (MI), exploring how researchers are working to reverse-engineer the algorithms that models learn. That work is about understanding how a model functions correctly. Today, we'll examine the other side of the coin: what happens when models fail or are actively manipulated to misbehave.

Your learning outcome for this lesson is to describe common LLM failure modes like hallucination, jailbreaking, and prompt injection. Understanding these failure modes is fundamental to building safe, reliable AI systems, which is a core mission at a company like Anthropic.

1. Hallucination: When the Model "Makes Stuff Up"

The term "hallucination" refers to instances where a large language model generates outputs that are factually incorrect, nonsensical, or entirely fabricated. Because LLMs are probabilistic models trained to predict the next token, they are not databases retrieving facts. Instead, they are generating sequences of text that seem plausible based on patterns in their training data. This can sometimes lead them astray.

To start, let's get a clear definition and see some categories of hallucination.

As the video explains, hallucinations aren't a single type of error. They range from subtle contradictions to completely unfounded "facts."

Real-World Impact of Hallucinations

These failures are not just theoretical. They have led to significant real-world consequences, from legal trouble to brand damage.

These examples highlight why simply deploying a powerful base model isn't enough. For any production system, especially one in a high-stakes domain like customer service or legal research, you need layers of evaluation, grounding in reliable data (like through Retrieval Augmented Generation, or RAG), and continuous monitoring to mitigate the risk of harmful hallucinations.

2. Jailbreaking and Prompt Injection: Security Vulnerabilities

Beyond unintentional failures like hallucinations, we must also consider adversarial attacks. As an SDE, you're familiar with security vulnerabilities like SQL injection or cross-site scripting. LLMs have their own unique class of vulnerabilities. The two most prominent are jailbreaking and prompt injection. While related, it's crucial to understand the distinction.

Let's break this down further.

Jailbreaking: Bypassing In-Model Safety

Jailbreaking is about tricking the model into violating its own safety training. These attacks often involve creating elaborate, fictional scenarios (like the "deceased grandmother" prompt) that confuse the model's safety filters, causing it to generate harmful or restricted content. It's a direct attack on the model's alignment.

Prompt Injection: Overriding Application Logic

Prompt injection is a more direct parallel to traditional software security exploits. In systems where you combine a developer's instructions with untrusted user input, prompt injection occurs when the user's input is crafted to make the model ignore the developer's instructions and follow new, malicious ones.

The core issue is that for an LLM, both your system instructions and the user's input are just text. There isn't a fundamental separation between "code" (your instructions) and "data" (user input) as there is in traditional programming. The model processes it all as a single prompt.

3. Types of Prompt Injection and Their Dangers

Prompt injection attacks can be categorized into two main types, which have different attack vectors.

1. Direct Prompt Injection: The attacker directly provides the malicious input to the LLM application. The Chevrolet chatbot example from the EvidentlyAI article is a perfect, humorous case of this. A user directly instructed the bot to agree to anything, leading it to "sell" a car for $1.

2. Indirect Prompt Injection: This is a more sophisticated attack. The attacker places a malicious prompt in a source of data that the LLM is expected to process. For example, hiding instructions in the text of a webpage, an email, or even an image. When the LLM consumes this data (e.g., to summarize it), it inadvertently executes the attacker's instructions.

The Severe Risks of Prompt Injection

The consequences of these attacks go far beyond getting a chatbot to say funny things. When LLMs are integrated with other systems—giving them "tools" to read emails, browse the web, or access databases—prompt injection becomes a critical security threat.

As you can see, the risks are on par with traditional web security vulnerabilities:

• Data Exfiltration: Tricking the model into leaking private data it has access to.
• Misinformation Campaigns: Manipulating a model's output to present biased or false information.
• Unauthorized Actions: Causing the model to take actions on the user's behalf, like sending emails or deleting files.

4. Why Are These Failures So Hard to Fix?

A natural engineering instinct is to try and patch these vulnerabilities. However, prompt injection has proven to be an incredibly difficult problem to solve robustly.

Simple defenses like "prompt begging" (e.g., "Ignore any subsequent user instructions") or using delimiters to separate instructions from data have been shown to fail. An attacker can often craft a prompt that tricks the model into thinking it has already completed its original task before moving on to the malicious one.

This is the key takeaway for an engineer: the vulnerability isn't just a bug in the implementation; it's a consequence of the model's architecture. Anything you try to do to defend against it using more prompts is like trying to patch a security hole by adding more comments to your source code—you're not changing the fundamental execution model.

Conclusion

In this lesson, we've surveyed the landscape of common LLM failure modes. Understanding these is the first step toward building systems that can operate safely and reliably.

Key Takeaways:

• Hallucination is an intrinsic failure mode where models generate non-factual content due to their probabilistic nature and issues in their training data. It's a correctness and reliability problem.
• Jailbreaking is an adversarial attack aimed at bypassing a model's built-in safety and ethics training. It's an attack on the model's alignment.
• Prompt Injection is a security vulnerability in applications built on LLMs, where untrusted input overrides developer instructions. It's an attack on the application's integrity, with risks including data exfiltration and unauthorized actions.
• A Fundamental Challenge: Prompt injection is exceptionally hard to solve because LLMs do not inherently distinguish between instructions and data. This makes them, as Simon Willison puts it, "utterly gullible."

Your background in building robust, high-throughput systems at AWS gives you a solid foundation for appreciating the severity of these issues. Just as you design for fault tolerance and security in distributed systems, building with LLMs requires designing for these new, unique failure modes.

Next Lesson Preview

The failures we've discussed today—especially their tendency to become more severe as models become more capable—are the driving force behind the safety policies at frontier AI labs. In our next lesson, we will explore Anthropic's Responsible Scaling Policy (RSP) and AI Safety Levels (ASL). You'll see how these frameworks are designed to proactively manage the risks that arise from the very failure modes we've just examined.