Recommended resources
The freely available resources this lesson will direct you to.
Hello! We’ll start by mapping the jobs that make up a local LLM system. Across this course you’ll run models on your own hardware, connect them to applications and documents, adapt an existing model, and train a small one from scratch. Those projects become easier to plan once you can answer a basic question: Are you changing the model, changing what you send it, or simply running it?
This lesson distinguishes five terms—pretraining, inference, prompt engineering, retrieval-augmented generation (RAG), and fine-tuning—using your planned coding, document, and RPG assistants as examples. Allow about 40 minutes, including the short videos.
Pretraining builds a model; inference runs it
Imagine downloading a Llama-family model and asking it to explain a Python function. Two very different kinds of computation lie behind that interaction. The model acquired its general language and code abilities during pretraining, before you downloaded it. When you ask your question, your machine performs inference: it uses the trained model to generate an answer.
Watch this brief explanation before we sharpen that distinction.
Large Language Models explained briefly
Watch “Large Language Models explained briefly” by 3Blue1Brown for a visual account of how a model is trained and then used to produce text.
Watch generation for the idea of repeatedly predicting what comes next in a response. Then watch pretraining, focusing on how examples from a large text collection are used to adjust the model's numerical parameters. The video sometimes says “word”; the more precise unit for our course is a token, which we’ll examine next lesson.
A model’s parameters, often called weights, are the numerical values learned during training. In pretraining, a system starts with a model architecture and adjusts those weights across a very large collection of text and other training material so that its next-token predictions improve. The result is a pretrained model: a reusable starting point, not necessarily a polished chat assistant. Additional training may subsequently make it better at following instructions.
Inference runs in the opposite mode. Your local runtime loads model weights, receives input, calculates likely next tokens, selects one, and continues until the response is complete. Ordinary inference does not update the weights. If a model gives a better answer after you clarify your question, it has responded to different input; it has not learned a permanent lesson from that exchange.
This difference sets the scale of two later projects. Your computers can run suitable existing models locally, and you will train a small Llama-style model to understand the mechanics. Pretraining a full-scale Llama-family model from scratch is a much larger data-and-compute undertaking. The small model is a learning project, not a prerequisite for building useful local assistants.
Prompt engineering changes the request, not the model
Suppose your coding assistant returns a broad essay when you need a focused review of a function. Before changing any software or training data, you can change the request: state the task, provide the relevant code, define what counts as a problem, and specify the output format. That is prompt engineering—designing the input to elicit a more useful response during inference.
RAG vs Fine-Tuning vs Prompt Engineering: Optimizing AI Models
Watch the prompt-engineering portion of IBM Technology’s “RAG vs Fine-Tuning vs Prompt Engineering: Optimizing AI Models.” It shows how clearer context and instructions can change a response without retraining.
Go to prompt engineering. Focus on the contrast between a vague code-security question and a more specific request, and on the fact that the underlying model remains unchanged.
For example, “Is this code secure?” leaves the model to guess your priorities and desired answer. A more useful request might be:
Review the Python function below for ways untrusted input could reach a shell command. Report only plausible issues, quote the relevant line for each, and propose a minimal fix. If the snippet is insufficient to decide, say what additional code you need.
The revised prompt narrows the task and defines a standard for the answer. You could also include an example of a well-formed review. All of that information is available for this request; it is not written into the model’s weights.
There is an important limit. Prompt engineering can guide a model’s existing abilities, and you can put a new fact directly in a prompt for it to use. But clever wording cannot make the model know a private repository file it has never been shown, nor does mentioning that file teach the weights its contents permanently. A lengthy prompt pasted by hand might work for one request; systematically finding the right material for each request is the job of RAG.
RAG supplies external evidence at answer time
Consider a private document assistant asked, “What does our current deployment runbook say about rolling back the service?” The answer may be in a document changed yesterday. Even a capable model cannot reliably reconstruct that particular revision from its training. Retrieval-augmented generation (RAG) addresses the information gap by finding relevant material and supplying it alongside the question during inference.
RAG vs Fine-Tuning vs Prompt Engineering: Optimizing AI Models
Return to the IBM Technology video for its explanation of retrieval, augmentation, and generation. Watch for the boundary between the document-search component and the model that writes the answer.
Watch the RAG workflow. The discussion moves from finding relevant files, to adding their contents to the request, to generating an answer. It also introduces embeddings as one way to find passages whose wording differs from the question; you do not need to implement embeddings yet.
In a typical system, you prepare a collection of documents so that passages can be searched. For each question, a retrieval component selects likely relevant passages. The application then places those passages in the model’s input with instructions to answer from them, ideally identifying the source of each claim. The model still performs inference to write the answer. RAG does not require changing its weights.
The diagram’s key boundary is between searching for evidence and generating language from that evidence. A local RAG setup might keep the runbooks, search index, embedding model, and answering model on machines you control. “Using a local answering model” alone is not a privacy guarantee if another part of the application sends documents to an external service.
Retrieval is not the same thing as understanding, and it is not automatically accurate. A search step can miss the decisive passage; a model can overlook or misstate a passage it receives. Embeddings are one search method, not a defining requirement of RAG—keyword search or a combination can also retrieve context. Later, you’ll build and evaluate these parts separately, rather than assuming that a plausible-sounding answer came from the right document.
RAG and prompt engineering work together. Retrieval supplies the current runbook passage; the prompt tells the model to answer the rollback question from that passage and cite its source.
Fine-tuning changes learned behavior
Now picture an RPG assistant that knows the campaign rules supplied in its input but repeatedly ignores your desired character-sheet format. A prompt and a few examples may solve this. If the same behavioral failure persists across many situations, fine-tuning becomes a candidate: continue training an existing pretrained model on examples of the inputs and responses you want.
RAG vs Fine-Tuning vs Prompt Engineering: Optimizing AI Models
The IBM Technology video contrasts fine-tuning with supplying information at request time. Watch for what changes during training and why maintaining a trained specialization differs from updating a document collection.
Watch fine-tuning. Focus on the starting point—an existing model—and on the training examples used to adjust its behavior. Treat its speed and data-size comparisons as scenario-dependent, not guarantees for every local setup.
Fine-tuning is another training phase, but it is not pretraining from scratch. The model already has general abilities; the additional examples push its predictions toward a particular task, style, or output convention. For the RPG assistant, examples could show how a player action should be turned into a consistent narration and structured character-state update.
There are several ways to make that change. Full fine-tuning updates the model’s existing weights. A parameter-efficient method such as LoRA can leave the original weights fixed while training a comparatively small set of additional parameters. In either case, the resulting system’s learned behavior changes. Later in the course, you’ll use parameter-efficient fine-tuning rather than attempting to retrain a large model from scratch on your 12 GB GPU.
Fine-tuning is usually a poor substitute for a current, checkable fact store. Training an assistant on today’s campaign notes would not guarantee faithful recall of every inventory item, and the notes will change as the game proceeds. Keep changing world and character state in external records supplied when needed; consider fine-tuning for a stable response convention that prompting alone cannot achieve reliably. The same distinction applies to a coding assistant: repository contents belong in supplied context or retrieval, while a consistently required review style might justify fine-tuning.
Choose by locating the missing piece
These five terms are not five competing model types. They describe different stages and components that can coexist in one application.
| Term | Role in the system | Are learned parameters changed? |
|---|---|---|
| Pretraining | Builds broad model capabilities from a large training collection | Yes |
| Inference | Uses a trained model to produce a response | No |
| Prompt engineering | Shapes the instructions and context given at inference time | No |
| RAG | Retrieves external material to include at inference time | No; it maintains external data or an index |
| Fine-tuning | Further trains an existing model for a desired behavior or task | Yes, including when only adapter parameters are trained |
When deciding what to build, diagnose the failure before choosing a technique. If the assistant knows enough but responds in the wrong format, improve the prompt first. If it lacks a private file or up-to-date rule, supply that information; use RAG when the system must find the right passage repeatedly. If it has the necessary information yet persistently fails a well-defined behavior across many cases, evaluate whether fine-tuning on representative examples helps. Pretraining a model from scratch serves a different purpose here: learning how models acquire their initial capabilities.
Every version of those assistants still uses inference to produce an answer. A fine-tuned document assistant might also retrieve passages and receive carefully written instructions. What changes is where each capability comes from: learned parameters, external evidence, or the current request.
Takeaways
Pretraining creates the model’s starting capabilities; fine-tuning adapts an existing model through further training. Inference runs a trained model without updating it. Prompt engineering changes what you ask and how you ask it, while RAG finds external evidence to include with the request. For a local project, distinguishing these roles helps you avoid training to fix a missing-document problem—or building a search system to fix an unclear instruction.
Next lesson, we’ll look inside the model’s input and output: how tokens, parameters, and context length shape the next-token predictions on which these workflows depend.