Augmented Prompting for Grounded LLM Responses

The "Generate" Stage: Synthesizing a Grounded Answer

You've done the hard work of retrieving relevant information and preparing it for the LLM. Now, the model takes over for the "Generate" stage. Think of this process like an open-book exam. In the previous lesson, you acted as the diligent student who found the correct pages in the textbook (retrieval) and highlighted the relevant paragraphs (augmentation). Now, you hand that open textbook to the LLM and ask it to write the answer based only on what's on those pages.

The video below provides this exact analogy and clarifies the role of the generation step within the broader RAG architecture.

The key idea is that the LLM is not using its vast, pre-trained knowledge to answer the question. Instead, it's using its language and reasoning capabilities to synthesize an answer from the specific evidence you've provided. The augmented prompt is the only thing the model sees.

This diagram perfectly illustrates the final package sent to the LLM. It includes the system instructions you wrote, the context you retrieved and formatted, and the original user question. The LLM processes all of this as a single input to produce the final, grounded output.

From Prompt to Response: Calling the LLM

With the augmented prompt string ready, the final step is to send it to an LLM via an API call. As a software engineer, you're already familiar with this process from Module 1. The mechanics are the same, but the payload—our rich, augmented prompt—is far more powerful.

We'll define a Python function to handle this interaction. While you could use any LLM provider, we'll look at an example using Ollama, a popular tool for running open-source models locally. This approach is excellent for development and for applications that require data privacy.

The article below provides a clear breakdown of the "Generate" step and offers a robust Python function for calling a local LLM.

Here's a simplified version of a function to generate a response, which you can integrate into your project. This function assumes you have an OpenAI-compatible client, which many services like Ollama or Anyscale provide.

import os
from openai import OpenAI




# It's good practice to use environment variables for API keys and base URLs
# For Ollama, the base_url is typically "http://localhost:11434/v1"
# The api_key can be a dummy string like "ollama"
client = OpenAI(
    base_url=os.environ.get("OPENAI_API_BASE"),
    api_key=os.environ.get("OPENAI_API_KEY"),
)

def generate_response(prompt: str, model: str = "llama3") -> str:
    """
    Sends the augmented prompt to an LLM and returns the response.
    """
    try:
        response = client.chat.completions.create(
            model=model,
            messages=[



                # The augmented prompt is passed as a single "user" message
                # The system instructions are already embedded within our prompt string
                {"role": "user", "content": prompt}
            ],
            temperature=0.0, # Set to 0.0 for deterministic, fact-based responses
        )
        return response.choices[0].message.content
    except Exception as e:
        print(f"An error occurred: {e}")
        return "Sorry, I was unable to generate a response."




# --- Example Usage ---
# final_prompt = create_augmented_prompt(retrieved_nodes, query) # From previous lesson
# answer = generate_response(final_prompt)
# print(answer)

In this function, we set temperature=0.0. This is a crucial parameter for RAG. A low temperature encourages the model to be more deterministic and factual, sticking closely to the information provided in the context. A high temperature would make the model more creative, increasing the risk of hallucination.

Orchestrating the Full Pipeline

You now have all the necessary components: a retriever, a prompt augmentation function, and a response generation function. Let's assemble them into a single, cohesive workflow that takes a user query and returns a grounded answer.

def query_rag_pipeline(query: str) -> dict:
    """
    Orchestrates the full RAG pipeline from query to answer.
    """



    # 1. RETRIEVE: Fetch relevant context chunks
    # This assumes you have a `retriever` object initialized as in previous lessons
    retrieved_nodes = retriever.retrieve(query)




    # 2. AUGMENT: Construct the augmented prompt
    augmented_prompt = create_augmented_prompt(retrieved_nodes, query)




    # 3. GENERATE: Get the final response from the LLM
    answer = generate_response(augmented_prompt)
    



    # Also return sources for verification
    sources = [node.metadata.get("file_name") for node in retrieved_nodes]

    return {"answer": answer, "sources": list(set(sources))}




# --- Example Usage ---
# user_query = "What are Python's naming conventions for constants?"
# result = query_rag_pipeline(user_query)
#
# print("Answer:")
# print(result["answer"])
# print("\nSources:")
# for source in result["sources"]:
#     print(f"- {source}")

This orchestration function encapsulates the entire logic. It’s the engine of your RAG application.

For a live demonstration of these final steps, the following video shows the prompt being sent to the LLM and the final response, along with its sources, being generated.

The Importance of "Constrained Generation"

The term for what we're doing—forcing the model to base its answer only on provided context—is constrained generation. This is an advanced form of prompt engineering that is fundamental to building reliable RAG systems. By providing explicit negative constraints (e.g., "Do not use your own knowledge") and positive constraints (e.g., "Use ONLY the provided context"), you are building guardrails that prevent the model from hallucinating.

The following resource provides a strong theoretical foundation for this concept.

As you can see, the prompt engineering strategies you learned in the previous lesson are the primary mechanism for achieving this constrained behavior and producing a truly grounded response. This is also a great example of a more complex RAG workflow, as shown in the L25 Grounding Workflow flowchart. Our simple pipeline maps directly to the "Vector Search" -> "Select Prompt" -> "LLM Generate" path in that diagram.

Conclusion

Congratulations! You have now implemented a complete, albeit basic, Retrieval-Augmented Generation pipeline. By connecting the retrieval, augmentation, and generation steps, you've created a system that can answer questions using a private knowledge base.

Here are the main takeaways from this lesson:

• The "Generate" step synthesizes an answer. The LLM uses its reasoning skills to create a response based on the evidence provided in the augmented prompt.
• Grounded responses are the goal. Through "constrained generation," enforced by strong prompting, we ensure the model's output is faithful to the retrieved context, minimizing hallucinations.
• The full pipeline is modular. The three steps—Retrieve, Augment, Generate—are distinct components that you can individually tune and improve. Setting temperature=0.0 is a key first step in tuning the generation phase for factual accuracy.

You've built the core engine. However, what happens if your retriever finds no relevant documents for a user's query? In the next and final lesson of this module, we will address this critical edge case to make our pipeline more robust. After that, we will move on to Module 6, where you'll learn how to evaluate the performance of your RAG system.