Recommended resources
The freely available resources this lesson will direct you to.
Hello! Welcome to your next lesson.
In our last session, we conducted a threat modeling exercise on our future application, analyzing its architectural weaknesses against the OWASP LLM01 criteria. We now have a clear "vulnerability report card" that explains why our application will be susceptible to prompt injection.
Today, we'll build on that analysis to create a concrete attack plan. Your goal for the final demo is to showcase 10 distinct attack payloads. This lesson is dedicated to defining those payloads and mapping each one to its underlying technique and the attacker's intent. This process is the bridge between understanding vulnerabilities and actually exploiting them.
This lesson directly addresses the learning outcome: Map each of the 10 planned attack payloads to its relevant prompt injection technique and intent.
By the end of this session, you will have a comprehensive "attack matrix" that will serve as our roadmap for the hands-on payload crafting modules that follow.
A Framework for Classification: Technique and Intent
In security analysis, it's crucial to distinguish between how an attack is carried out and why. For our purposes:
- Technique: The specific method or "trick" used to make the LLM misbehave. This is the how. Examples include role-playing, obfuscating instructions, or poisoning data.
- Intent: The attacker's ultimate goal, which we can classify using the familiar CIA triad: Confidentiality, Integrity, or Availability. This is the why.
Let's explore these two dimensions.
1. Attack Techniques
Attackers have developed a diverse arsenal of techniques to bypass LLM defenses. The following article provides an excellent catalog of these methods, written from the perspective of a red teamer.
20 Prompt Injection Techniques Every Red Teamer Should Test
Please read the following sections from the article '20 Prompt Injection Techniques Every Red Teamer Should Test'. This will give you a practical vocabulary for describing the 'how' of our attacks.
Read the sections titled '1. Direct Instruction Overrides', '2. Encoding & Obfuscation Tricks', '3. Jailbreaks & Role-Play', '4. Indirect Prompt Injection', and '5. Contextual Exploits'. You don't need to memorize all 20, but focus on understanding the core idea behind each category.
These techniques can be broadly grouped, as shown in this visual aid.
2. Attacker Intent (The CIA Triad)
The ultimate goal of an injection attack isn't just to make the chatbot say something funny; it's to compromise the system's security objectives. The CIA triad provides a robust framework for categorizing these goals.
To deepen your understanding of how prompt injection maps to the CIA triad, the following academic paper provides detailed, real-world examples for each category.
[PDF] Trust No AI: Prompt Injection Along The CIA Security Triad
This paper, 'Trust No AI', is a fantastic resource that systematically ties prompt injection exploits to their impact on Confidentiality, Integrity, and Availability. Read the highlighted sections to see how researchers classify real-world attacks.
Skim the introduction (Section I and II). Then, read the first few paragraphs and the 'Scenario' sub-headings in Section III ('Loss of Confidentiality'), Section IV ('Loss of Integrity'), and Section V ('Loss of Availability'). This will give you a clear picture of how different attacks serve different intents.
Here’s a quick summary of how these intents apply to LLMs:
- Confidentiality: The attacker's goal is to steal information. This could be the system prompt, user data from the context window, API keys, or proprietary information from a RAG database.
- Integrity: The attacker's goal is to make the LLM produce false, biased, or malicious content. This could be generating misinformation, creating a phishing email, or manipulating a business-critical recommendation.
- Availability: The attacker's goal is to disrupt the service. This could involve tricking the model into an infinite loop, making it refuse to answer prompts, or causing it to consume excessive resources.
The 10 Planned Attack Payloads
Now, let's define the 10 attack payloads we will build for your demo. These are designed to provide a broad and compelling showcase of the risks.
Direct Injection Payloads (Module 4):
- Instruction Override: The most basic attack. Make the LLM ignore its system prompt and obey a new, malicious instruction.
- Jailbreak (Role-Playing): Use a persona like "DAN" (Do Anything Now) to trick the model into violating its safety guidelines.
- Prompt Leaking: Coax the model into revealing its own system prompt.
- Few-Shot Injection: Provide a few malicious examples in the prompt to teach the model a new, undesirable behavior.
- Delimiter Injection: Confuse the model's parsing of the prompt by inserting characters it uses to separate instructions from data, making it treat data as an instruction.
Advanced & Indirect Payloads (Module 5):
6. Goal Hijacking: Redirect the chatbot from its intended purpose to a malicious one, such as generating a fake financial recommendation.
7. Payload Splitting: Break a malicious instruction across multiple conversational turns to evade simple filters that only inspect one message at a time.
8. Obfuscation: Hide a malicious payload using an encoding like Base64 to bypass naive keyword filters.
9. Indirect Injection (RAG Poisoning): Plant a malicious instruction in an external document that the RAG pipeline will retrieve and feed to the model.
10. Multi-turn Social Engineering: Use a series of conversational prompts to gradually build trust or a specific context that leads to the model performing an unsafe action.
Mapping Payloads to Technique and Intent
This is the core of our lesson. The following table maps each of our 10 planned payloads to its primary technique and the attacker's intent. This "attack matrix" will be our guide for the next two modules.
| # | Payload Name | Primary Technique | Attacker Intent (CIA) | Rationale |
|---|---|---|---|---|
| 1 | Instruction Override | Direct Instruction Override | Integrity | The primary goal is to make the model do something other than its intended function, corrupting its behavior. |
| 2 | Jailbreak (Role-Playing) | Jailbreaking / Role-Play | Integrity | Bypasses safety filters to generate prohibited content, violating the system's operational integrity. |
| 3 | Prompt Leaking | Direct Instruction Override | Confidentiality | The explicit goal is to exfiltrate the (supposedly secret) system prompt. |
| 4 | Few-Shot Injection | Contextual Exploit (Few-Shot Hijack) | Integrity | Manipulates the model's in-context learning ability to produce consistently incorrect or biased outputs. |
| 5 | Delimiter Injection | Contextual Exploit | Integrity / Confidentiality | Confuses the model into treating data as instructions, which can lead to prompt leakage (Confidentiality) or altered behavior (Integrity). |
| 6 | Goal Hijacking | Direct Instruction Override | Integrity | A more sophisticated integrity attack that redirects the LLM's purpose towards a specific, malicious business outcome. |
| 7 | Payload Splitting | Evasion Technique | Integrity / Confidentiality | The technique is primarily for evasion, but the intent is usually to achieve an Integrity or Confidentiality breach that would otherwise be blocked. |
| 8 | Obfuscation (Base64) | Encoding / Obfuscation | Integrity / Confidentiality | Like payload splitting, this is an evasion technique. The underlying goal is to sneak a payload past filters to attack integrity or confidentiality. |
| 9 | Indirect Injection (RAG) | Indirect Injection (Poisoned Data Source) | Integrity / Confidentiality | The classic indirect attack. A poisoned document can leak other data from the context (Confidentiality) or manipulate the final output (Integrity). |
| 10 | Multi-turn Social Eng. | Social Engineering | Integrity | Gradually manipulates the model's state over a conversation to make it perform an action it would have refused if asked directly. |
This matrix provides a structured, top-down view of our attack plan. It's the kind of deliverable you would create in a professional security engagement before beginning the hands-on testing phase.
Conclusion
In this lesson, we transitioned from analyzing our application's weaknesses to creating a formal plan of attack. We defined a framework for classifying payloads by technique (the how) and intent (the why), and then used it to build a comprehensive attack matrix for the 10 payloads you'll demonstrate.
Key Takeaways:
- Prompt injection attacks can be categorized by their technique (e.g., role-playing, obfuscation, indirect injection) and their intent (Confidentiality, Integrity, Availability).
- A structured attack matrix is an essential tool for planning and executing a security assessment, ensuring comprehensive coverage of different threat vectors.
- Direct attacks manipulate the prompt in a single interaction, while indirect attacks poison the data the LLM will consume later.
- Evasion techniques like obfuscation and payload splitting are not attacks in themselves but are used to bypass defenses to achieve a C, I, or A goal.
Next Up:
Our planning is complete. In the next lesson, we will finally get our hands dirty. We will begin crafting the payloads from our attack matrix, starting with the most fundamental attack: executing a basic Instruction Override to make the LLM ignore its system prompt.