Welcome to the sixth lesson of our foundational module. In our last session, we explored embeddings and vector similarity, the core technologies that allow us to perform semantic search and implement the Retrieval-Augmented Generation (RAG) pattern. You now understand the mechanism for providing an LLM with external, real-time knowledge.
This lesson addresses the "why" behind such patterns. While powerful, LLMs have inherent limitations and failure modes that can be problematic, especially in the production systems you aim to build. Today, we will identify and analyze three common LLM failure modes: stale knowledge, hallucinations, and prompt injection. Understanding these weaknesses is crucial for designing robust, reliable, and secure AI agents.
The Nature of LLM Risks
Large Language Models, by their very design, are susceptible to certain risks. They are autoregressive token predictors trained on vast but static datasets, and they process instructions and data in the same "channel"—natural language. These characteristics are the source of their power but also their primary vulnerabilities.
Let's break down these risks, starting with the most straightforward one.
1. Stale Knowledge: The Frozen Worldview
The knowledge of a large language model is not perpetually current. It is "frozen" at the point in time when its training data was collected. This is known as the knowledge cutoff. For instance, a model trained in early 2023 will have no knowledge of events, discoveries, or data that emerged later that year.
For your goal of building logistics agents, this is a critical limitation. Such an agent, relying solely on its internal knowledge, would be unaware of:
- New shipping regulations enacted last month.
- A recently opened port that could optimize a route.
- The real-time status of a specific shipment, which is currently delayed in transit.
This is precisely the problem that the RAG pattern, which we discussed in the previous lesson, is designed to solve. By retrieving fresh, relevant information from external sources (like a database or a live API) and providing it to the model as context, we overcome the stale knowledge problem. Stale knowledge is less a flaw to be "fixed" in the model and more a constraint to be managed through system architecture.
2. Hallucinations: Confident Untruths
A more subtle and challenging failure mode is hallucination. This occurs when an LLM generates text that is plausible, coherent, and grammatically correct but is factually incorrect or nonsensical. Crucially, the model presents this fabricated information with the same level of confidence as it would a known fact.
The term "hallucination" can be slightly misleading; the model isn't "seeing things." As a token-predictor, it's simply generating the most statistically likely sequence of words based on the patterns in its training data. When it lacks sufficient information or a clear signal for a given query, it fills the gaps by extrapolating, which can lead to fabrications.
To get a better sense of this, let's start with a high-level overview.
Why Large Language Models Hallucinate
This video from IBM Technology, "Why Large Language Models Hallucinate," provides a concise introduction to the concept and its common forms.
Please watch from the beginning of the discussion on hallucinations at 01:43. Focus on the different categories the speaker introduces, such as factual contradictions and prompt contradictions.
While that gives us a general understanding, building production systems requires a more rigorous, engineering-focused perspective. For that, we'll turn to a more in-depth analysis.
LLM Hallucinations: Why They Happen and How to Reduce Them [2026]
This article from Atlan provides a fantastic, enterprise-oriented breakdown of hallucinations. It reframes the problem from a "model flaw" to a "context failure," which is highly relevant for agent developers.
First, read the section Why LLM hallucinations happen. Pay close attention to the three root causes identified, especially the third one: The context quality problem. Next, read the section Two typologies. This is the most important part. You need to understand the difference between the Standard Hallucination Types (like factual vs. faithfulness) and the Context Failure Types (like stale or conflicting context). The latter category is what you, as an system architect, can directly address.
The key insight from the Atlan article is that for enterprise applications, most harmful hallucinations are not a result of a "broken" model but a broken data-flow. When a RAG system retrieves stale, conflicting, or poorly described information, the model will faithfully use that bad context to generate a confident, but wrong, answer. This highlights a critical principle for your future work: the reliability of your AI agent depends directly on the quality and governance of the data you provide to it.
3. Prompt Injection: Hijacking the Conversation
The third major failure mode, prompt injection, is a security vulnerability. It exploits the fact that LLMs don't have a separate, privileged channel for instructions. Both system instructions (e.g., "You are a helpful assistant") and user data (e.g., "Summarize this email") are processed as text in the same context window. An attacker can craft input that causes the model to ignore its original instructions and follow new, malicious ones.
Let's see a classic example of this in action.
What Is a Prompt Injection Attack?
This IBM Technology video, "What Is a Prompt Injection Attack?," opens with a memorable real-world example of prompt injection.
First, watch the introductory scenario about the car dealership chatbot from the beginning. Then, skip ahead to this segment, which explains why LLMs are vulnerable to this: the distinction between instructions and input is "a lot murkier" than in traditional programming.
This "murky distinction" is the heart of the problem. Unlike SQL injection, where an attacker injects code into a data channel, with LLMs, there is only one channel. An attacker's goal is to make their malicious instructions more salient to the model than your original system prompt.
Prompt Injection vs. Jailbreaking
You will often hear the term "jailbreaking" used in this context. While related, there is a subtle distinction that is useful to understand.
In practice, jailbreaking is often accomplished via prompt injection techniques (e.g., using role-playing scenarios), so you can think of it as a specific subtype of prompt injection.
Defense Against the Dark Arts
As a developer, you need to know the attack vectors and the corresponding defenses. The OWASP Foundation, known for its work on web application security, has compiled an excellent guide.
LLM Prompt Injection Prevention Cheat Sheet
The OWASP cheat sheet is the definitive guide for developers on this topic. It's dense but directly applicable.
Please review the following sections: Start with the Introduction and Anatomy to solidify your understanding of the core vulnerability. Skim the Common Attack Types section. You don't need to memorize all of them, but pay special attention to Direct Prompt Injection and Remote/Indirect Prompt Injection. The indirect vector, where malicious content is hidden in a document or webpage your agent processes, is a significant threat for RAG-based systems. Carefully read Primary Defenses, focusing on "Input Validation and Sanitization" and "Structured Prompts with Clear Separation." These are fundamental techniques you will implement. Finally, read about the Model-Based Guardrails, particularly the dual-LLM pattern. This is a powerful architectural defense where one LLM handles untrusted data and passes only structured, sanitized information to a second, privileged LLM that can take action. This is a key pattern for building secure agents.
The primary defense against prompt injection is to create as clear a separation as possible between what you intend as instructions and what you process as data. Techniques like using XML tags, JSON objects, or other delimiters to encapsulate user input can help the model distinguish data from commands. Building input filters and monitoring outputs are also crucial layers in a defense-in-depth strategy.
Conclusion
Today we've confronted the three most common and critical failure modes of LLMs. Understanding these weaknesses is the first step toward building systems that are resilient to them.
Here are the key takeaways:
- Stale Knowledge is a fundamental constraint of pre-trained models. It is managed through architecture, primarily with the RAG pattern that provides up-to-date context at inference time.
- Hallucinations are confident-sounding fabrications that arise from the model's probabilistic nature. While they can't be eliminated completely, they can be drastically reduced by grounding the model in high-quality, well-governed external data using RAG.
- Prompt Injection is a security vulnerability that hijacks the model's behavior. It is mitigated by a combination of defensive prompt engineering, input/output filtering, and robust architectural patterns like model-based guardrails.
Notice the common thread: the solutions to these problems often lie not in tweaking the model itself, but in the systems you build around it. Your role as an agent developer is as much about data governance and security architecture as it is about writing prompts.
In our next lesson, we will transition from these foundational concepts to the practical matter of choosing a model. We will compare major providers like Google (Gemini), OpenAI, and Anthropic, evaluating them on criteria like tool-calling support, context size, and cost—all of which are critical for agent development.