Prompt Injection: In-context Learning and Few-shot Prompting

Hello! Welcome back to our course.

In the last lesson, we established the fundamental architectural pattern for building LLM applications: separating the developer's instructions (system prompt) from the end-user's query (user prompt) within the model's context window. This separation is our first line of defense.

However, as we noted, LLMs process instructions and data as a unified stream of tokens. This lesson dives into the core mechanism that makes this architectural separation so fragile. We will explore how an LLM's greatest strength—its ability to learn on the fly—is also the primary vector for prompt injection.

By the end of this lesson, you will be able to explain how in-context learning and few-shot prompting are the mechanisms that enable prompt injection attacks. Understanding this is critical to grasping why these attacks are not just possible, but often deceptively simple to execute.

1. The Power of On-the-Fly Learning: In-Context Learning (ICL)

At their core, LLMs are incredibly sophisticated pattern-matching and text-completion engines. They don't "reason" in the human sense; they predict the next most probable token based on the entire sequence of tokens they've seen so far—the context. This leads to a remarkable emergent capability called In-Context Learning (ICL).

ICL is the model's ability to learn how to perform a task simply by seeing examples within the prompt itself, without requiring any changes to its underlying weights (i.e., no retraining or fine-tuning).

Think of it in terms of your experience with automation scripts. A script's behavior can be drastically altered at runtime by the environment variables or configuration files it reads on startup. The script itself isn't recompiled, but its execution changes based on the context provided. The LLM's prompt serves a similar function: it's a dynamic, runtime configuration that guides the model's behavior for a single execution.

To get a formal understanding of ICL and its variants, the following article provides an excellent guide.

As the article explains, ICL manifests in several ways, primarily distinguished by the number of examples provided:

• Zero-shot: No examples are given. You just state the task.
• One-shot: One example is provided to clarify the task or format.
• Few-shot: Two or more examples are given to establish a clear, generalizable pattern.

This ability to teach a model a new pattern on the fly is what makes few-shot prompting so powerful for legitimate use cases.

2. The Attack Vector: Exploiting In-Context Learning

The same mechanism that allows you to teach an LLM to translate French or generate code in a specific style can be co-opted by an attacker. If the examples in the prompt can define the rules, what happens when an attacker injects their own malicious examples into the user input?

The model, lacking a true separation between trusted and untrusted input, will treat the attacker's examples as part of the pattern to be learned and replicated.

Let's watch a video that illustrates this concept with a simple, yet effective, example.

In the video's example, the user ZetaTwo's input, "LiveOverflow broke the rules, LiveOverflow wrote illegal comments...", acts as a malicious "shot". It creates a strong pattern within the context that suggests "LiveOverflow is a rule-breaker". The model, seeking to complete this pattern, follows the new instruction and ignores the fact that your original comment ("I like trains") was benign.

This is the essence of a few-shot injection attack: the attacker provides fabricated examples within their prompt to manipulate the model's behavior.

3. A Real-World Attack: PII Extraction via Augmented Few-Shot Learning

This concept isn't just theoretical. Security researchers have demonstrated highly effective attacks that weaponize few-shot prompting. A prominent example is using this technique to extract Personally Identifiable Information (PII) from a model's training data.

The following academic paper details such an attack. While it's a technical read, we will focus on the high-level concepts and the structure of the attack prompts, which are directly relevant to the demo you're building.

The key takeaway from the paper is how the attack exploits the model's Memorization and Association capabilities. By providing examples in a specific format ({"name": ..., "email": ...}), the attacker creates a strong contextual pattern. This pattern "convinces" the model that the task at hand is to complete another PII record, leading it to recall and output sensitive data from its training set.

This directly informs several of the attack payloads you'll be building, particularly Prompt Leaking and Few-Shot Injection.

A Quick Look at Another Advanced Example

Even beyond PII leakage, few-shot ICL can be used for sophisticated data exfiltration. In a notable exploit against Microsoft Copilot, an attacker used in-context learning to teach the model how to encode sensitive data into hidden Unicode characters before sending it to an attacker-controlled server.

This demonstrates the core threat: any sufficiently advanced LLM that can learn from examples in its context is susceptible to being taught malicious behaviors in that same context.

Conclusion

In this lesson, we've connected the dots between an LLM's core functionality and its vulnerability to prompt injection. The very mechanism that makes LLMs so flexible and powerful is what attackers exploit.

Key Takeaways:

• In-Context Learning (ICL): An emergent property of LLMs that allows them to learn tasks from examples provided within the prompt's context window, without any permanent changes to the model.
• Few-Shot Prompting: A specific, powerful application of ICL where providing 2-5 examples (shots) can reliably guide the model's output format, tone, and behavior.
• The Attack Mechanism: Prompt injection works by exploiting ICL. An attacker crafts a user prompt containing malicious instructions or examples. The model interprets these as part of the pattern to learn from, leading it to bypass its original instructions.
• Relevance to Your Goal: The techniques you will use to craft attacks like "few-shot injection," "goal hijacking," and "prompt leaking" are all practical applications of weaponizing in-context learning.

You now understand the "how" and "why" behind the fundamental vulnerability. We've seen how attackers can manipulate the model directly through the user prompt.

In our next lesson, we will expand the threat landscape by exploring Retrieval-Augmented Generation (RAG). You will learn how to diagram a RAG pipeline and, crucially, identify the new attack surfaces it introduces—especially for indirect prompt injection, where the malicious payload comes from an external data source, not the user's input.