Deliberate Vulnerabilities in the Target App

Hello! Welcome to the fourth and final lesson of our second module.

In our last session, you successfully built a complete Retrieval-Augmented Generation (RAG) chatbot and exposed it through a FastAPI endpoint. This application is the foundation for our entire security demonstration.

Today, we shift our perspective from a builder to a security analyst. This lesson directly addresses the learning outcome: Identify and document the deliberate vulnerabilities in the target app: unvetted RAG data, no input validation, no output filtering, and a permissive system prompt.

Instead of adding new features, we will perform a high-level security audit of the application we just built. We'll break down exactly why it's vulnerable and document these weaknesses in a structured way, just as you would for a professional security assessment. This is a crucial step before we begin crafting attack payloads in the upcoming modules.

The Four Pillars of Vulnerability in Our App

Our application, while functional, lacks any meaningful security controls. This is by design, to create a perfect "vulnerable" target for our demo. We can categorize its weaknesses into four main areas, which correspond to missing layers in a "defense-in-depth" security strategy:

1. Unvetted RAG Data: We implicitly trust the content of our knowledge base.
2. No Input Validation: We blindly accept and process any user input.
3. No Output Filtering: We return the LLM's response directly to the user without inspection.
4. A Permissive System Prompt: We give the LLM minimal instructions on how to behave securely.

Let's analyze each of these in detail.

1. Vulnerability: Unvetted RAG Data (Enabling Indirect Injection)

What it is: Our RAG application reads the company_policy.txt file, chunks it, and places it into the vector store without any sanitization or verification. The application assumes the data source is trustworthy.

Why it's a problem: This creates a classic vector for Indirect Prompt Injection. An attacker doesn't need to interact with the chatbot directly. Instead, they can "poison" the data that the RAG system relies on. When a legitimate user asks a question, the RAG system retrieves the poisoned data, which contains malicious instructions, and feeds it to the LLM as trusted context.

To understand this concept more deeply, please watch the following short video.

This attack flow is also visualized clearly in the diagram below. "Step 1" is poisoning the data source; "Step 2" is the LLM retrieving and executing the malicious instruction during a normal user interaction.

Code Evidence:

In rag_app.py, the vulnerability is born the moment we load the document without scrutiny:

# rag_app.py




# ...
# 1. We load the document from a file path. What if this file was
#    uploaded by a user or sourced from a compromised web page?
loader = TextLoader("./data/company_policy.txt")
docs = loader.load()




# 2. We split and embed the document's raw content. Any malicious
#    instructions within it are converted into embeddings and stored.
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=100)
splits = text_splitter.split_documents(docs)

vectorstore = Chroma.from_documents(documents=splits, embedding=embeddings)



# ...

Documentation Entry for Your Demo

As you build your portfolio, documenting your findings is key. Here’s a template you can adapt:

Vulnerability ID: VULN-001
Category: Indirect Prompt Injection via Unvetted RAG Data (OWASP LLM-01)
Description: The application's RAG pipeline ingests documents from the local filesystem (./data/) without performing any validation or sanitization. It implicitly trusts the content of the knowledge base.
Impact: An attacker could poison the source documents (e.g., company_policy.txt) with hidden instructions. When a legitimate user's query retrieves the poisoned content, the LLM may execute the attacker's commands, leading to data exfiltration, misinformation, or goal hijacking.
Location: rag_app.py, lines 13-22.

2. Vulnerability: No Input Validation (Enabling Direct Injection)

What it is: The user's question, received at the /chat endpoint, is passed directly to the rag_chain.invoke() method without any checks, filtering, or sanitization.

Why it's a problem: This enables Direct Prompt Injection. An attacker can craft a malicious query designed to manipulate the LLM. In traditional web security, this is analogous to passing raw user input into an SQL query, leading to SQL Injection. Here, the "payload" is natural language, but the principle of not validating untrusted input is the same.

Code Evidence:

In main.py, the user's question is taken directly from the JSON request body and used to invoke the chain.

# main.py




# ...
@app.post("/chat")
async def chat_endpoint(request: ChatRequest):
    """
    Receives a question, passes it to the RAG chain, and returns the answer.
    """
    try:



        # The 'request.question' is from an untrusted user and is passed
        # directly to the LLM pipeline without any validation.
        response = rag_chain.invoke(request.question)
        return {"answer": response}
    except Exception as e:
        return {"error": str(e)}

For contrast, a secure application would implement validation checks. To see what that looks like, take a look at the "Query Sanitization" section in the article below.

Our application does none of this, leaving the front door wide open.

Documentation Entry for Your Demo

Vulnerability ID: VULN-002
Category: Direct Prompt Injection via Missing Input Validation (OWASP LLM-01)
Description: The /chat API endpoint accepts a user-provided question and passes it directly to the LangChain RAG pipeline without any form of sanitization, validation, or pattern matching.
Impact: An attacker can submit a prompt that contains "jailbreaking" instructions, commands to ignore the system prompt, or other manipulative language. This can cause the LLM to bypass its safety instructions, leak confidential context, or perform unauthorized actions.
Location: main.py, line 21.

3. Vulnerability: No Output Filtering (Enabling Insecure Output)

What it is: The response generated by the LLM is immediately packaged into a JSON object and returned to the user. There is no intermediate step to check if the output is safe, appropriate, or contains sensitive information.

Why it's a problem: This is classified as Insecure Output Handling. The fundamental mistake is treating the LLM as a trusted actor. An LLM's output is a direct result of its input. If an injection attack is successful, the output will contain the malicious result. By failing to filter the output, we are essentially delivering the attacker's payload for them.

The following video segment explains the severe consequences of this vulnerability.

Code Evidence:

The vulnerability is a single, simple line in main.py. The response from the rag_chain is sent back without a second thought.

# main.py




# ...
@app.post("/chat")
async def chat_endpoint(request: ChatRequest):



    # ...
    try:
        response = rag_chain.invoke(request.question)



        # The 'response' from the LLM is returned directly. If the LLM was
        # tricked into leaking the system prompt or generating harmful content,
        # that content is sent straight to the user.
        return {"answer": response}



    # ...

Documentation Entry for Your Demo

Vulnerability ID: VULN-003
Category: Insecure Output Handling (OWASP LLM-02)
Description: The application forwards the raw string output from the LLM directly to the user in the API response. There is no output filtering, encoding, or validation layer to check the content before it is delivered.
Impact: If a prompt injection attack is successful, the application will dutifully return the malicious output. This could include leaked system prompts, exfiltrated data from the RAG context, generated hate speech, or links to malicious websites.
Location: main.py, line 22.

4. Vulnerability: A Permissive System Prompt

What it is: The prompt template we designed in rag_app.py is minimal. It tells the LLM to answer based on context but gives it no defensive instructions.

Why it's a problem: The system prompt is your primary tool for instructing the LLM on its core mission and rules of engagement. A well-designed, robust system prompt acts as a defense-in-depth layer. It can explicitly instruct the model to disregard instructions found in the retrieved context or to refuse to answer questions that seem manipulative. Our prompt is too simple and trusting.

The prompt is a composite of many inputs, but the system prompt is your chance to set the foundational rules.

Code Evidence:

This is the prompt template in rag_app.py.

# rag_app.py




# ...
# This template is too simple. It doesn't tell the model what to do
# if the context contains conflicting or malicious instructions.
template = """Answer the question based only on the following context:
{context}

Question: {question}
"""
prompt = ChatPromptTemplate.from_template(template)



# ...

To see what a more secure prompt looks like, review the example in the article we referenced earlier.

The difference is stark: the secure prompt is defensive, anticipating potential attacks and instructing the model on how to respond.

Documentation Entry for Your Demo

Vulnerability ID: VULN-004
Category: Insufficient Prompt Engineering for Security
Description: The system prompt template is overly simplistic. It instructs the model to use the provided context but fails to include defensive instructions, such as explicitly telling the model to ignore any commands or instructions found within that context.
Impact: The lack of a robust system prompt makes the LLM more susceptible to both direct and indirect prompt injection. The model has no "immune system" or guiding policy to help it differentiate between legitimate user questions and embedded malicious commands.
Location: rag_app.py, lines 31-36.

Conclusion

In this lesson, we put on our security hats and performed a threat analysis of our target application. You now have a clear, documented understanding of the four major vulnerabilities we intentionally built into it. This analytical process is just as important as the attacks themselves, especially in a professional consulting context where explaining the "why" is paramount.

Key Takeaways:

• Unvetted RAG Data opens the door to Indirect Prompt Injection.
• No Input Validation allows for Direct Prompt Injection.
• No Output Filtering results in Insecure Output Handling, ensuring malicious payloads are delivered.
• A Permissive System Prompt fails to provide a basic layer of defense through instruction.

We have a fully functional—and fully vulnerable—target application. We understand its weaknesses and have documented them. We are now perfectly positioned to start our attack phase.

In the next module, we will begin with "Prompt Injection Taxonomy and Threat Modeling." We'll learn how to classify different prompt injection attacks based on the OWASP Top 10 for LLMs and map them to our planned payloads, setting a clear strategy for the hands-on exploits to come.