Recommended resources
The freely available resources this lesson will direct you to.
Hello! This course will take you from choosing a local model to running it, building assistants around it, adapting it, and finally training a small model yourself. We’ll start with a decision that shapes all of those projects: what actually needs to change when an existing model does not meet a requirement?
By the end of this lesson, you should be able to classify a requirement as inference, prompt engineering, retrieval-augmented generation (RAG), fine-tuning, or pretraining from scratch, and explain why that choice fits. The distinction will help you avoid training a model to solve a search problem—or building a search system to solve a behavior problem.
First, separate using a model from changing it
Suppose you download a model, run it on your Mac or desktop, and ask it to explain a function. It generates an answer using its existing parameters. That is inference: applying an already-trained model to an input. Nothing about that request necessarily calls for customization.
Watch this short explanation of the training–inference distinction:
AI Inference: The Secret to AI's Superpowers
In “AI Inference: The Secret to AI's Superpowers,” IBM Technology distinguishes learning a model's parameters from using them. This gives us a baseline for the other four categories.
Watch the two phases, then weights and inference. Focus on which phase changes the model and which phase produces an output from a new input.
Inference is not mutually exclusive with the other choices. A RAG assistant, for instance, retrieves documents and then runs inference to answer. For classification, use inference when the requirement is simply to run or integrate an existing model and no additional knowledge, behavior change, or training is called for.
Consider a private coding assistant. If its requirement is “run a local model that explains code pasted into the chat,” inference may be enough. If it must answer questions about files elsewhere in your repository, you now have a separate question: how will the relevant files reach the model?
Three ways to improve an existing model’s answers
An existing model can produce a poor answer for different reasons. The request may be underspecified; the model may lack a needed document; or it may repeatedly fail to follow a desired pattern. These are different engineering problems.
Prompt engineering changes the instructions and examples supplied with a request, without changing model parameters. You might tell an RPG assistant to answer as a concise game master, state the rules it should apply, and show two examples of suitable responses. It is a sensible first choice when the model already has the necessary capability and information, but needs clearer direction. Supplying a known passage directly in a prompt also supplies information without training; it does not, by itself, require a retrieval system.
RAG addresses a different need: finding relevant information in an external collection and supplying it to the model at answer time. For a document assistant, the application searches your files for passages relevant to the question, includes those passages in the model’s input, and asks for an answer grounded in them. The model’s parameters remain unchanged. RAG is especially useful when the information is private, changes regularly, spans too many documents to paste into every request, or needs source references.
Fine-tuning starts with an existing model and performs additional training to adapt its parameters—or a small set of added parameters—to examples of desired behavior. It is a candidate when prompting has not reliably achieved a repeatable style, format, task performance, or domain-specific behavior, and you can assemble suitable training examples. It is generally a poor substitute for retrieving an evolving set of facts: changing a document is easier than retraining a model, and trained-in facts do not provide dependable citations.
The following video develops the contrast. Treat its descriptions of training cost and sample counts as illustrations, not minimum requirements for every fine-tuning method; later in the course we’ll work with parameter-efficient methods on constrained hardware.
RAG vs Fine-Tuning vs Prompt Engineering: Optimizing AI Models
In “RAG vs Fine-Tuning vs Prompt Engineering: Optimizing AI Models,” IBM Technology compares the three approaches through their mechanisms and trade-offs. Watch it to connect each method to the kind of problem it solves.
Begin with the three choices. In RAG in action, track what gets searched and what enters the model's input. Watch fine-tuning for what training changes, then prompt engineering for what it changes without training. Finish with using them together; the methods need not be exclusive.
Here is the RAG path in one view. The useful distinction is that retrieved information becomes input context for generation, rather than being written into the model’s parameters.
Retrieval does not guarantee a correct answer. The search may miss the right passage, or the model may make a claim that the passage does not support. For your private document assistant, “uses local files” therefore describes an architecture, while “answers with accurate citations” is a requirement you will still need to test.
When does the answer involve training from scratch?
Pretraining from scratch means beginning with newly initialized model parameters and training a model on a corpus, rather than adapting an existing pretrained model. It calls for choices about the training data, tokenizer, architecture, and training process. This is fundamentally different from fine-tuning, which inherits the capabilities of an existing model.
A requirement might justify pretraining from scratch if available pretrained models cannot meet essential constraints and the project has sufficient data, compute, and expertise to build its own foundation model. On your hardware and six-week schedule, that is not a practical route to a general-purpose Llama competitor. It is a worthwhile route to understanding the machinery: later, you’ll train a small Llama-style model on a bounded dataset and evaluate what it learns. The objective in that case is learning and experimentation, not replacing a production-scale pretrained model.
AWS’s guidance offers a useful resource-conscious starting point. Its examples use managed cloud services, but the decision criteria also apply when every component runs locally.
GENOPS05-BP01 Learn when to customize models - Generative AI Lens
Read the “Implementation guidance” in AWS's Generative AI Lens for its progression from prompting through retrieval and training. Focus on the conditions for escalating, rather than the AWS products mentioned later on the page.
Under “Implementation guidance,” begin at the paragraph advising a start with the least resource-intensive option. Read the decision criteria through the conditions for building a custom foundation model. Notice which options need task examples, external knowledge, or control over model creation.
“Start with the least resource-intensive option” is a good default, not a rule that every project must try every option in order. If the central requirement is to cite a changing private manual, prompting alone cannot fetch that manual. Conversely, if the assistant already has all needed information but consistently fails a particular output convention, adding document retrieval may not address the failure.
A diagnostic for choosing—and justifying—a method
A useful justification has three parts: the missing ingredient, the mechanism that supplies it, and a reason not to choose a more involved alternative yet.
Begin by asking what is missing:
| If the requirement is primarily… | First classification to consider | Why |
|---|---|---|
| Run an existing model privately on supplied input | Inference | The existing model can perform the task without another information or training step. |
| Clarify a task, constrain an answer, or demonstrate a format | Prompt engineering | Instructions and examples can guide behavior without changing parameters. |
| Answer from a collection of files, especially changing files, with source references | RAG | Retrieval supplies relevant evidence at request time. |
| Make a recurring behavior or task performance more reliable across many inputs | Fine-tuning | Training examples can adapt an existing model when prompting is insufficient. |
| Build a model beginning with new parameters | Pretraining from scratch | The requirement explicitly calls for creating the underlying model, not adapting one. |
This table gives a first classification, not a promise of success. For example, a strict JSON output requirement often starts with prompting and application-side validation. Fine-tuning becomes a stronger candidate if measured failures persist across representative inputs. Similarly, RAG may supply the right documents while fine-tuning helps a model learn how to use them. Both can be appropriate if each addresses a demonstrated failure.
Meta’s discussion is helpful here because it frames RAG and fine-tuning as potentially complementary rather than rival brands of “customization.”
To fine-tune or not to fine-tune
Read the “Fine-tuning and RAG” subsection of Meta AI's article for diagnostic questions about knowledge, behavior, changing data, and transparency. Use the questions to sharpen a requirement before selecting a technique.
Within “Comparison with other techniques for domain adaptation,” find the “Fine-tuning and RAG” subsection. Start with its opening discussion of why a simple RAG-versus-fine-tuning rule is inadequate. Read the comparison, ending after the paragraph about measuring the benefit of using both. Pay particular attention to the questions about external knowledge, custom behavior, data freshness, and references; you can skip the acknowledgements below.
There is one more constraint your projects share: local and private describes where components run and where data goes, not which customization category they belong to. A RAG system is not private if document embeddings or queries are sent to an external service. A fine-tuned model is not automatically safe to distribute merely because training ran locally; its training data may contain private material. Keep deployment and data boundaries in the justification alongside the method.
Applying the diagnostic to your projects
For the coding assistant, start by specifying its reach. “Explain this pasted function” is primarily local inference, perhaps with a prompt specifying review criteria. “Answer where this repository handles authentication” requires supplying repository evidence: selecting a few files manually may suffice for a small task, while searching a larger repository suggests RAG. Training a model on the repository would not be the first way to obtain an accurate answer about today’s code.
For the document assistant, “answer from my collection and show which passages support the answer” points to RAG. Its central need is accessible, traceable source material. If the assistant cites a retrieved passage but repeatedly ignores a specialized summarization convention, you could first revise the prompt and then evaluate whether fine-tuning adds value. The citation requirement still calls for retrieved evidence.
For the RPG assistant, a defined narrator voice can begin with prompt engineering. If it must remember the current inventory and world state, that state must be supplied or retrieved when needed; fine-tuning is not a dependable record of events that occurred after training. If you later collect many examples and the model repeatedly mishandles a particular dialogue style despite clear prompts, fine-tuning becomes a candidate for that behavioral gap.
Finally, “train a custom model on another dataset” is not yet precise enough to classify. If you mean “adapt an existing model to produce answers like these examples,” that is fine-tuning. If you mean “initialize a small model and learn its language patterns from this corpus,” that is pretraining from scratch. The distinction is the starting point of the model, not merely the fact that a dataset is involved.
In each case, classification improves when you can name an observable failure: missing documents, inconsistent format, stale facts, or inadequate capability. That failure becomes the basis for comparing the simplest plausible solution with more involved ones.
Key takeaways
Inference runs an existing model. Prompt engineering changes its instructions. RAG supplies retrieved information at request time. Fine-tuning adapts an existing model through training. Pretraining from scratch builds a newly initialized model. These categories describe different interventions, and an application may combine them.
Choose by identifying what is missing, then justify why the proposed intervention supplies it at an acceptable cost and within your privacy boundary. In the next lesson, we’ll look inside an inference request: how a prompt becomes tokens, how the model predicts the next token, and why context length matters for every approach introduced here.