Recommended resources
The freely available resources this lesson will direct you to.
- RAG vs Agentic AI: How LLMs Connect Data for Smarter AIwww.youtube.com
- Context Engineering vs. Prompt Engineering: Smarter AI with RAG & Agentswww.youtube.com
- What is retrieval-augmented generation?www.redhat.com
- Retrieval augmented generation (RAG) and indexes in Microsoft Foundry - Microsoft Foundry | Microsoft Learnlearn.microsoft.com
Hello again. Last time, you saw why a local language model cannot inspect your document library merely because the files are on your computer: it generates from the text supplied in its current request. The missing piece is how the application finds the right text—and what it does after finding it.
This lesson distinguishes four capabilities that are often bundled together under the label “AI assistant”: semantic retrieval, retrieval-augmented generation (RAG), tool use, and autonomous agent behavior. We’ll follow a question about a bicycle-repair receipt through each one. By the end, you should be able to tell whether a feature needs a search index, a model-generated answer, a callable action, or a system that decides its own next step.
Semantic retrieval finds relevant material
Suppose your library contains a March 2025 receipt headed “Cycle Service,” with a line for “rear derailleur adjustment.” You search for “bike gear repair bill.” A keyword search might miss the receipt because your wording does not appear in it. Semantic retrieval aims to find material with a similar meaning, despite differences in wording.
One common implementation represents document passages and the query as numerical embeddings. A search index compares those representations and returns ranked matches. The immediate output is not an answer such as “The total was $86.40.” It is something more like: Cycle Service receipt, page 1, matching passage. The application can show that result directly, without asking a language model to write anything.
Read Red Hat’s short comparison for the distinction between finding relevant data and generating a response from it:
What is retrieval-augmented generation?
Read Red Hat’s “What is retrieval-augmented generation?” for its explanation of semantic search as a retrieval capability used in many RAG systems.
In the “RAG vs. semantic search” subsection, start where the text defines semantics and read the comparison through its closing statement about RAG. Focus on what semantic search returns rather than treating it as answer generation.
Semantic similarity is useful for finding the repair receipt, but it is not a verdict that the result is correct. Another receipt might discuss a similar repair. Conversely, an exact date, invoice number, or amount may be easier to find with keyword search or a metadata filter than with embeddings. Semantic retrieval is one way to search; it need not be the only one. Later, you’ll build an index that can preserve file and page information alongside searchable text, then combine semantic and exact matching where appropriate.
RAG uses retrieved material to generate an answer
Now change the request from “Find my bike gear repair bill” to “What was the total on my March 2025 bicycle-repair receipt?” A ranked list of files might be useful, but the user has asked for a fact. In retrieval-augmented generation, the application retrieves relevant source material, places it in the model’s input, and asks the model to generate an answer using that material.
Microsoft Learn’s overview gives a compact account of the moving parts:
Read Microsoft Learn’s “Retrieval augmented generation (RAG) and indexes in Microsoft Foundry” to connect retrieval, model context, and source citations.
In “What is RAG?”, read the explanation and key concepts. Then, in “How does RAG work?”, continue through “What is an index and why you need one”: read the three stages and index discussion. Notice that the index may support keyword, semantic, vector, or hybrid search; RAG does not require one particular retrieval method.
For our receipt example, imagine the retrieved passage reads “Total paid: $86.40” and retains the label “Cycle Service, March 2025, page 1.” The model can then write, “The total paid was $86.40,” while the application supplies the file-and-page citation. Retrieval supplies candidate evidence; generation turns supplied evidence into an answer. RAG does not require retraining the language model on your receipts.
The diagram below shows these two jobs side by side. It depicts a particular database and optional caching features; neither is a requirement for the local application you’ll build.
Recall the context-window limit from the previous lesson. RAG helps by selecting a small amount of relevant text instead of sending the whole library to the model. It does not, by itself, guarantee a correct answer. The search might retrieve an older receipt, the model might misread a total, or a citation might point to a passage that does not support the claim. A trustworthy assistant must keep the source reference attached to each passage and check that the final answer is supported. If no retrieved passage establishes the total, it should say that the evidence is insufficient.
A useful boundary follows: a search-results page can use semantic retrieval without being RAG, while a RAG application can retrieve with keyword search, semantic search, or both.
Tools let the application perform defined actions
Retrieval and generation describe what information is found and how an answer is written. Tool use describes how a model-driven application can request a defined operation. A tool is a callable function with a specified purpose and inputs—for example, search_documents(query, document_type), preview_page(document_id, page_number), or open_managed_file(document_id).
Consider “Show me the March 2025 repair receipt.” The application might expose a document-search tool. A model could request a search with a date and document type, receive a list of permitted results, and ask for a preview of one page. The model does not gain direct access to the filesystem simply by naming a tool. Application code implements the operation and decides what the tool is allowed to return. For private files, that distinction matters: an open_managed_file tool should accept a managed document identifier, not an arbitrary path supplied by a model.
Tools do not necessarily imply an agent. A fixed RAG application may always call search once, add the results to a prompt, and generate an answer. Its code controls that routine even though search is implemented as a callable operation. Alternatively, a model might choose whether to call preview_page before answering. That is model-selected tool use, but one choice alone does not make an extended, autonomous workflow.
IBM Technology’s comparison helps separate a RAG pipeline from an application that chooses actions:
RAG vs Agentic AI: How LLMs Connect Data for Smarter AI
Watch IBM Technology’s “RAG vs Agentic AI: How LLMs Connect Data for Smarter AI” to hear the distinction between an action-selecting agent and retrieval followed by generation.
In the agent discussion, watch the agent cycle for the idea of choosing an action and checking its result. Then skip to the RAG stages for document indexing, retrieval, and model input. Treat the examples as architectural patterns; a private local assistant does not need the enterprise services mentioned in the video.
An agent decides whether another action is needed
Imagine a broader request: “Compare what I paid for bicycle repairs in March and June, and show me both receipts.” The first search finds only March. A fixed workflow might report that June was not found. An agentic workflow can use that observation to choose another permitted action: perhaps search with a different description, inspect a candidate’s date, and then decide whether it has enough evidence to compare the two receipts.
The distinguishing feature is a decision loop. The system gives a model a goal and a limited set of tools; the model selects an action, receives its result, and decides what to do next or when to stop. The application still controls the available tools and must impose limits. “Autonomous” here means making some intermediate choices without a person specifying each one—not unrestricted access to files, indefinite searching, or permission to take consequential actions.
Nor does every multi-step request require an agent. If the application can reliably parse “March and June,” run two searches, verify two totals, and format a comparison using a predetermined workflow, that may be simpler and easier to test. An agent is more justifiable when results or ambiguity genuinely determine the next search. You’ll examine bounded workflows and agent orchestration later in the course; for now, the question is who chooses the next action—application code or the model?
| Capability | What it contributes | Receipt-library example |
|---|---|---|
| Semantic retrieval | Finds meaning-related passages or documents | Finds “rear derailleur adjustment” for “bike gear repair” |
| RAG | Generates an answer using retrieved passages | Answers the total question from the receipt and cites its page |
| Tool use | Invokes a defined, application-controlled operation | Requests a search or a permitted page preview |
| Agent behavior | Chooses subsequent actions based on observed results | Searches again for June after an initial search finds only March |
These capabilities can coexist. An agent may call a semantic-search tool and then use the returned passages in a RAG answer. Equally, a useful document finder may have semantic search and page previews but no generated answers or agent at all. Calling every such system an “agent” hides important differences in complexity, cost, and control.
Putting the distinctions to work
For a private document assistant, start by asking what the user needs. If they want a file, return search results with clear provenance. If they want a fact from a file, retrieve evidence and generate a grounded answer—or report that the evidence is missing. If they need an operation, expose a narrowly defined tool. If the request requires choosing further actions in response to uncertain results, consider a bounded agent rather than assuming one is necessary.
The key distinction is between finding, answering, acting, and deciding what to do next. Keeping those responsibilities separate will make the eventual React and Python application easier to reason about. Next, you’ll examine the privacy implications of those responsibilities: sensitive information can appear not only in source files, but also in extracted text, embeddings, temporary files, logs, and model outputs.