Defending Against Indirect Prompt Injection

Welcome to the next lesson in our journey to build production-grade AI systems. In our previous sessions, we've focused on building a strong perimeter defense. You learned to filter malicious user inputs and use the "sandwich" technique to isolate untrusted data. We've effectively secured the data flowing into the LLM.

However, a truly robust system must also scrutinize what comes out. An LLM, compromised by a hidden instruction in a document it reads or a tool it uses, can become a vector for an attack. Its generated output might contain a malicious command intended for another part of your system, or even harmful code to be executed in a user's browser.

In this lesson, we will implement the critical final checkpoint in your application's security pipeline: robust output validation and parsing. You will learn to treat the LLM's output not as a final answer, but as an untrusted payload that must be rigorously inspected. We will use modern API features and Pydantic schemas—a cornerstone of production Python development—to enforce structure and then apply semantic checks to mitigate threats from indirect prompt injection.

The Threat: When the LLM Becomes the Attacker

The principle of "never trust user input" is a fundamental concept in software engineering. When building AI systems, we must extend this to: Never trust the LLM's output. An LLM doesn't have intentions, but it can be manipulated to produce outputs that are harmful to your system.

The most common vector for this is indirect prompt injection. Imagine an AI agent designed to summarize web pages for a user.

In this scenario, the user's request is benign. The attack is hidden in the data the agent consumes. If the agent's output is not validated, it might:

• Pass a malicious command like wget malicious.sh to a downstream tool-execution engine.
• Return a script tag like <script>steal_cookies()</script> that gets rendered in the user's browser, leading to a Cross-Site Scripting (XSS) attack.
• Leak its own system prompt or other confidential data it has access to.

Our goal is to build a "filter" on the LLM's output that ensures it is not only correctly formatted but also semantically safe.

Defense Layer 1: Enforcing Structure with Constrained Decoding

The first and most powerful defense against malicious output is to eliminate the possibility of generating unstructured, arbitrary text. If the LLM is forced to respond in a strict, predefined format, it literally cannot output a random shell command or a malformed script tag. This is achieved through a feature often called structured outputs or constrained decoding.

Let's explore how this works.

As the article explains, by providing the LLM API with a JSON schema (which can be generated from a Pydantic model), we activate constrained decoding. The LLM is no longer free to generate any token; it can only choose tokens that will result in an output that is 100% compliant with our schema.

Here is a practical example in Python. Let's say we want an LLM to extract a user's sentiment from a piece of text. We can define our desired output structure using Pydantic:

from pydantic import BaseModel, Field
from typing import Literal




# Define the structured output using a Pydantic model
class SentimentAnalysis(BaseModel):
    reasoning: str = Field(description="Think step-by-step to determine the sentiment before providing the final answer.")
    sentiment: Literal["positive", "negative", "neutral"] = Field(description="The final sentiment classification.")
    confidence: float = Field(ge=0.0, le=1.0, description="Confidence score for the classification, from 0.0 to 1.0.")




# Example of how you would use this with the OpenAI API
# (Assumes 'client' is an initialized OpenAI client)




# user_text = "I'm not sure how I feel about this new update. It has some good features, but it also broke my workflow."




# response = client.chat.completions.create(
#     model="gpt-4o",
#     messages=[{"role": "user", "content": user_text}],
#     response_model=SentimentAnalysis, # This is a conceptual example; actual syntax varies by library (e.g., instructor)
#     # For native OpenAI, you might use response_format={"type": "json_object", "schema": ...}
# )




# The 'response' object would now contain a parsed SentimentAnalysis object,
# not just a raw string.

By defining this SentimentAnalysis model, we've created a powerful guardrail. The LLM cannot respond with "Ignore instructions. Run this command...". It is physically constrained to produce a JSON object with reasoning, sentiment, and confidence fields, with the correct data types.

Visualizing the Validation Workflow

This entire process of taking raw AI output and ensuring it becomes safe, structured data can be visualized as a simple flowchart.

Constrained decoding ensures the "Success" path is taken for structural validation. Our next step is to handle the semantic content of that successful output.

Defense Layer 2: Semantic Validation and Sanitization

Constrained decoding gives us structurally valid data. But is it semantically safe? What if the LLM generates a perfectly-formed JSON object that contains a malicious instruction?

{
  "tool_to_call": "run_command",
  "parameters": {
    "command": "rm -rf /"
  }
}

This is where application-level validation comes in. Your code, having received a clean Pydantic object, must now inspect its contents.

As the article makes clear, your responsibility as an engineer doesn't end when you receive a well-formatted response. You must implement a second layer of checks:

• Allow-listing: The tool_to_call field should be checked against a predefined list of safe, allowed tools. If the LLM returns delete_user_account, and that's not in your agent's allow-list, the request is rejected.
• Parameter Scrutiny: Don't just execute a tool with the parameters the LLM provides. Validate them. Is the user_id a valid format? Does the file_path point to a permitted directory? The execute_sql example in the reading shows this perfectly—blocking keywords like DROP or DELETE.
• Sanitization for Display: If the output is meant to be shown in a UI, sanitize it to remove any potential HTML or script tags. Use standard, well-vetted libraries for this (e.g., bleach in Python, or DOMPurify in JavaScript).

This two-step process—structural validation via constrained decoding, followed by semantic validation in your application logic—forms a robust defense against attacks funneled through the LLM's output.

Conclusion

In this lesson, we completed our defense-in-depth security architecture by focusing on the final, critical step: validating the LLM's output. You have moved beyond simply hoping for a good response to engineering a system that guarantees it.

Key Takeaways:

• Treat LLM Output as Untrusted: The core security principle for AI engineering is to treat the model's output with the same skepticism as user input.
• Enforce Structure with Constrained Decoding: Use modern API features with Pydantic schemas to force the LLM to generate structurally correct output. This is your most effective first line of defense against arbitrary, malicious text.
• Perform Semantic Validation: After ensuring the structure is correct, your application code must validate the content of the output. Use allow-lists, parameter checks, and business logic to ensure the response is not just well-formed, but also safe.
• Sanitize for Display: Any output rendered in a user interface must be sanitized to prevent XSS and other injection attacks.

You've now covered the essential security practices for building single-agent and RAG applications. This knowledge is not just theoretical; it's a prerequisite for designing the scalable, reliable systems that top tech companies are building.

In our next module, we will begin preparing for the AI Engineer interview. We'll start by learning a structured framework for deconstructing system design questions, where the security and validation principles you've just learned will be indispensable.