Understanding LLM01: Prompt Injection in OWASP Top 10 for LLM

Hello! Welcome to the first lesson of Module 3.

In the last module, we successfully built our vulnerable target application. We confirmed it was functional for legitimate queries and, more importantly, exploitable through both direct and indirect prompt injection attacks. We've seen that it can be broken. Now, it's time to build a formal framework to understand how and why it breaks, which will be essential for structuring your demo.

This lesson addresses the learning outcome: Summarize the OWASP Top 10 for LLM Applications, focusing on the LLM01: Prompt Injection category. We'll start with a high-level overview of the most critical security risks in AI and then dive deep into prompt injection, the central theme of your project. This framework will help us classify the 10 attack payloads you'll be building in the upcoming modules.

The Landscape of LLM Vulnerabilities

In traditional web development, the OWASP Top 10 provides a crucial, industry-recognized list of the most critical security risks. Given your background in SRE and platform engineering, you're likely familiar with its importance for threat modeling and security audits. The AI community, through OWASP, has created a similar resource for Large Language Models.

Let's start by getting a complete overview.

To quickly get up to speed on what each of these categories means, please watch the following video. It provides a concise explanation of all ten vulnerabilities.

To solidify this, here's how you can think about these vulnerabilities in the context of your SRE/DevOps experience:

• LLM01: Prompt Injection: Analogous to SQL Injection, where untrusted input is executed as a command.
• LLM02: Sensitive Information Disclosure: A classic data leak, amplified by the LLM's tendency to be overly helpful or reveal training data patterns.
• LLM03: Insecure Supply Chain: Similar to vulnerabilities in third-party libraries (npm, pip) or Docker base images, but here it applies to pre-trained models, datasets, and plugins.
• LLM04: Data and Model Poisoning: Can be compared to tampering with configuration data or build artifacts to compromise a system's integrity.
• LLM05: Improper Output Handling: Like failing to sanitize the output of an internal API before passing it to a shell, database, or web frontend, which can lead to XSS, CSRF, or RCE.
• LLM06: Excessive Agency: The equivalent of granting an application overly broad IAM permissions (e.g., s3:*), allowing it to perform unintended, high-impact actions.
• LLM07: System Prompt Leakage: Similar to leaking environment variables, configuration files, or other sensitive infrastructure manifests.
• LLM08: Vector and Embedding Weaknesses: A new attack surface, but you can think of it as manipulating a database index to trick the system into retrieving and processing malicious data.
• LLM09: Misinformation & Hallucination: A reliability and integrity issue where the system generates incorrect or harmful output, undermining its function.
• LLM10: Unbounded Consumption: A resource exhaustion or Denial of Service (DoS) attack, where an attacker crafts inputs that lead to extreme computational or financial costs.

A Deep Dive into LLM01: Prompt Injection

While all ten categories are important, your demo focuses on LLM01: Prompt Injection. It's listed first for a reason: it is the most fundamental and prevalent vulnerability in LLM applications.

The official OWASP project makes a direct comparison that should resonate with your technical background.

As the video states, prompt injection is about making the system do things it was not intended to do. The vulnerability exists because, by design, LLMs process instructions and data in the same context, without a strong separation. Your malicious input is seen by the model as just another set of instructions to follow.

Formal Definition and Attack Anatomy

Let's get a more formal definition and look at the anatomy of the vulnerability. The OWASP Cheat Sheet for prompt injection is an excellent technical resource for this.

The key takeaway is that direct concatenation of a system prompt and untrusted user input creates the vulnerability. The model has no reliable way to distinguish which part is the inviolable instruction and which part is the data to be processed.

A Taxonomy of Prompt Injection Techniques

"Prompt Injection" is a broad category. To prepare your 10 attack payloads, you need to understand the different techniques that fall under this umbrella. These techniques are what we will be implementing in Modules 4 and 5.

The attack you performed in the previous lesson to poison the RAG data source is a classic example of Indirect Prompt Injection. The attacker doesn't provide the input directly; instead, they place it in an external resource they know the LLM will consume.

The OWASP resources provide a detailed list of these attack patterns. The following reading will be the foundation for the attacks we will build.

From this reading, you now have a catalog of techniques. For your demo, your 10 planned payloads will be concrete implementations of these patterns:

1. Instruction Override: A form of Direct Prompt Injection.
2. Jailbreaking (e.g., DAN): A specific Jailbreaking Technique.
3. Prompt Leaking: An example of System Prompt Extraction.
4. Few-shot Injection: A clever way to manipulate the model by providing malicious examples.
5. Delimiter Injection: A Context Manipulation attack.
6. Goal Hijacking: Can be a Direct or Indirect attack with high business impact.
7. Payload Splitting: An advanced technique to bypass simple filters.
8. Obfuscation (e.g., Base64): An Encoding and Obfuscation Technique.
9. Indirect Injection via RAG: An example of RAG Poisoning or Remote/Indirect Prompt Injection.
10. Multi-turn Social Engineering: A form of Multi-Turn Attack.

We now have a shared vocabulary and a clear structure for the hands-on work ahead.

Conclusion

In this lesson, we established a formal framework for discussing LLM vulnerabilities. You now have a high-level understanding of the OWASP Top 10 for LLM Applications and a much deeper appreciation for the nuances within the LLM01: Prompt Injection category.

Key Takeaways:

• The OWASP Top 10 for LLM Applications provides a standard, comprehensive list of key security risks, analogous to the classic Top 10 for web apps.
• LLM01: Prompt Injection is a fundamental vulnerability stemming from the architectural fusion of instruction and data in an LLM's context window.
• Prompt injection is not a monolithic threat; it encompasses a wide range of techniques, including direct injection, indirect injection, jailbreaking, obfuscation, and poisoning data sources.
• We have now mapped the 10 payloads for your demo to this formal taxonomy, giving us a clear plan for the upcoming modules.

Next Up:

In the next lesson, we will dedicate our time to solidifying the most important distinction within prompt injection: direct versus indirect attacks. We'll explore more concrete examples and discuss why understanding this difference is critical for both threat modeling and designing effective defenses.