Building RAG Chains with ChatPromptTemplate

The Anatomy of a Chat Model Prompt

When you interact with a chat model API directly, you typically don't send a single string. Instead, you provide a sequence of messages, each with an assigned role. This structure allows the model to understand the context of the conversation and its own expected behavior. The three fundamental message types are:

• System Message: This sets the high-level instructions for the AI. It defines the model's persona, its objective, constraints, and any other rules it should follow. This message is usually set by the developer and is not visible to the end-user.
• User Message: This represents the input from the end-user. In our RAG case, this will be the user's question.
• Assistant Message: This represents the AI's previous responses. It's used to build a history of the conversation, allowing for follow-up questions and contextual memory.

While our previous ChatPromptTemplate.from_template was functional, it bundled everything into a single message. For a robust RAG application, we need to separate the system instructions from the user's query.

Constructing a Multi-Message Template

LangChain provides ChatPromptTemplate.from_messages to create these structured prompts. This method accepts a list of tuples, where each tuple contains the message role ("system", "user", or "ai") and the corresponding template string.

Let's define a proper RAG prompt. Our goal is to create a prompt with two parts:

1. A system message that instructs the model on how to behave. It will tell the model to act as a Q&A assistant and use the provided context.
2. A user message that contains the actual question from the user.

The following article from Aurelio AI provides a clear explanation of this structure and its implementation in LangChain.

As you saw in the reading, the implementation is straightforward. We define a string for our system prompt and then pass it along with the user prompt to ChatPromptTemplate.from_messages.

Let's update our src/lcel_quickstart/main.py file to incorporate this more robust prompt structure. Notice how the setup_and_retrieval chain we built in the last lesson already produces the exact dictionary ({"context": ..., "question": ...}) that this new prompt template expects.

import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate # Updated import
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnableLambda, RunnableParallel, RunnablePassthrough




# Load environment variables
load_dotenv()
if "OPENAI_API_KEY" not in os.environ:
    raise ValueError("OPENAI_API_KEY not found in .env file")

def fake_retriever(query: str) -> str:
    """A simple fake retriever for demonstration purposes."""
    print(f"--- [DEBUG] Retrieving documents for query: '{query}' ---")
    if "lcel" in query.lower():
        return "LCEL stands for LangChain Expression Language, a declarative way to compose chains."
    return "No relevant documents found."

def main():
    """
    Demonstrates a RAG chain with a multi-message prompt template.
    """
    model = ChatOpenAI(model="gpt-4o-mini")
    parser = StrOutputParser()
    retriever = RunnableLambda(fake_retriever)




    # 1. Define the system prompt
    system_prompt = (
        "You are an assistant for question-answering tasks. "
        "Use the following pieces of retrieved context to answer the question. "
        "If you don't know the answer, just say that you don't know. "
        "Keep the answer concise."
        "\n\n"
        "Context: {context}"
    )




    # 2. Create a ChatPromptTemplate using from_messages
    prompt_template = ChatPromptTemplate.from_messages(
        [
            ("system", system_prompt),
            ("user", "{question}"),
        ]
    )




    # 3. The parallel step to prepare data for the prompt (from previous lesson)
    setup_and_retrieval = RunnableParallel(
        {"context": retriever, "question": RunnablePassthrough()}
    )




    # 4. The complete chain, now with the new prompt template
    chain = setup_and_retrieval | prompt_template | model | parser




    # Invoke the chain
    question = "What is LCEL?"
    print(f"--- Invoking chain with question: '{question}' ---")
    response = chain.invoke(question)
    
    print("\n--- Final Response ---")
    print(response)

if __name__ == "__main__":
    main()

Run this script with uv run src/lcel_quickstart/main.py. The output should be functionally identical to the previous lesson, but under the hood, we are now sending a much better-structured prompt to the LLM, giving us more control.

Preparing for Conversation: MessagesPlaceholder

Your goal is to build a conversational chatbot. This means our chain must eventually handle conversation history. We can prepare our prompt template for this right now by adding a placeholder for chat history messages.

LangChain provides the MessagesPlaceholder for this exact purpose. It dynamically inserts a list of messages into the prompt sequence. The standard structure for a conversational RAG prompt is:

1. System Message (instructions)
2. MessagesPlaceholder (for past user/AI messages)
3. User Message (the latest question)

This ensures the model gets its instructions, is reminded of the conversation so far, and then receives the new query. Let's add this to our prompt. We will need to import MessagesPlaceholder.

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

Then, we modify the prompt_template definition:

    # ... inside main() ...




    # The system prompt remains the same
    system_prompt = (
        "You are an assistant for question-answering tasks. "
        "Use the following pieces of retrieved context to answer the question. "
        "If you don't know the answer, just say that you don't know. "
        "Keep the answer concise."
        "\n\n"
        "Context: {context}"
    )




    # Update prompt_template to include a placeholder for chat history
    prompt_template = ChatPromptTemplate.from_messages(
        [
            ("system", system_prompt),
            MessagesPlaceholder(variable_name="chat_history"),
            ("user", "{question}"),
        ]
    )

By adding MessagesPlaceholder(variable_name="chat_history"), we've told the prompt template to expect an additional input variable named chat_history, which should be a list of messages.

The create_history_aware_retriever and RunnableWithMessageHistory components, which we will cover in Module 3, are designed to automatically manage and populate this variable. For now, to make our chain runnable, we must provide this input manually. We can add it to our setup_and_retrieval step.

Let's modify the main.py code one last time.

# full main.py with MessagesPlaceholder

import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI



# Import MessagesPlaceholder
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnableLambda, RunnableParallel, RunnablePassthrough




# ... load_dotenv and fake_retriever function ...

def main():
    model = ChatOpenAI(model="gpt-4o-mini")
    parser = StrOutputParser()
    retriever = RunnableLambda(fake_retriever)

    system_prompt = (
        "You are an assistant for question-answering tasks. "
        "Use the following pieces of retrieved context to answer the question. "
        "If you don't know the answer, just say that you don't know. "
        "Keep the answer concise."
        "\n\n"
        "Context: {context}"
    )

    prompt_template = ChatPromptTemplate.from_messages(
        [
            ("system", system_prompt),
            MessagesPlaceholder(variable_name="chat_history"),
            ("user", "{question}"),
        ]
    )




    # We need to add 'chat_history' to the dictionary being passed to the prompt
    # For now, we pass an empty list via a RunnableLambda.
    setup_and_retrieval = RunnableParallel(
        {
            "context": retriever,
            "question": RunnablePassthrough(),
            "chat_history": RunnableLambda(lambda x: []), # Pass an empty history
        }
    )

    chain = setup_and_retrieval | prompt_template | model | parser

    question = "What is LCEL?"
    print(f"--- Invoking chain with question: '{question}' ---")
    response = chain.invoke(question)
    
    print("\n--- Final Response ---")
    print(response)

if __name__ == "__main__":
    main()

Now, our chain expects chat_history and we provide it—an empty list for now. When you run the script, the result will be the same, but our prompt template is now fully prepared for the conversational features we'll implement in Module 3.

Conclusion

In this lesson, you have elevated your prompt engineering skills within LCEL. You've moved from a simple, flat string template to a structured, multi-message prompt that gives you far more control over the model's behavior.

Here are the key takeaways:

• Structured Prompts: Chat models are designed to work with a sequence of messages, each having a role (system, user, ai).
• ChatPromptTemplate.from_messages: This is the standard LCEL method for creating role-based prompts, which is a best practice for building reliable LLM applications.
• System Prompts: You can provide high-level instructions, persona, and constraints to the model via the system message, separating the task definition from the user's specific query.
• MessagesPlaceholder: This component creates a slot in your prompt for a list of messages, making your chain ready for conversational memory, which we will explore in depth later.

You now have a solid data flow and a robust prompt structure. The next logical step is to replace our fake_retriever with a real one. In the upcoming lesson, you will dive into the "Retrieval" part of RAG: loading documents from a source, splitting them into optimized chunks, and generating vector embeddings.