Training vs. Inference: Understanding the Difference

Good to see you again. In the last lesson, you followed a prompt through inference: the model processes its input, estimates likely next tokens, and generates an answer token by token. This lesson draws the essential boundary behind that process: training changes the model; inference uses the model as it currently is.

That distinction matters for RAG. When a RAG system retrieves a company policy or product manual and places it in the prompt, it is not teaching the LLM permanently. It is supplying temporary evidence for one inference request.

By the end of this lesson, you should be able to describe what happens in training, what happens when an LLM answers a user, and explain why adding a document to a RAG system is different from retraining or fine-tuning a model.

---

Two phases with very different jobs

An LLM has a large collection of numbers called parameters or weights. These numbers determine how the model transforms input tokens into probabilities for the next token. They are not a database of readable sentences; rather, they encode many learned statistical patterns about language, concepts, and response behavior.

There are two broad phases:

Phase	Main purpose	Are model weights changed?	Typical input	Output
Training	Learn or improve behavior from examples	Yes	Large datasets, usually with known target text	An updated model
Inference	Produce an answer or prediction for a new request	No	A live prompt, including user question and optional RAG context	A response

The simple mental model is:

• Training: adjust the model’s internal numerical settings.
• Inference: hold those settings fixed and use them to respond to a new prompt.

The diagram includes practical deployment steps such as optimization and hosting. Those are important in real systems, but the conceptual dividing line for this course is whether the model’s learned parameters are being updated.

---

What actually happens during training?

Training begins with many examples of text. In the basic pre-training task, the model is given an earlier part of a sequence and asked to predict the next token. Unlike a live chat, training has access to the text that actually came next.

Suppose an example in the training data is:

RAG retrieves relevant documents before generating an answer.

At one position, the model may receive:

RAG retrieves relevant

and predict probabilities for the next token. The training data tells it that the desired continuation is documents.

Initially, a model’s predictions are poor because its parameters begin as random or inadequately adapted values. Training repeatedly compares its prediction with the actual next token. This comparison produces a numerical measure of error called a loss.

A simplified training cycle has five parts:

1. Select a batch of training text. The system takes many text examples from its dataset.
2. Make predictions. A forward pass processes the tokens and produces next-token probabilities.
3. Measure error. The model’s predicted distribution is compared with the known target token.
4. Compute how to reduce the error. A procedure called backpropagation calculates how each parameter contributed to the error.
5. Update parameters. An optimization algorithm makes tiny numerical adjustments intended to lower future loss.

Conceptually, the update looks like:

$$\text{new parameters}=\text{old parameters}-\text{small adjustment based on error}$$

This process is repeated over enormous quantities of data. No one manually writes a separate fact or grammar rule into each weight. Instead, optimization gradually changes many parameters so that useful next-token predictions become more likely across the training examples.

Training is expensive because it requires more than generating predictions. The system must retain information needed to calculate gradients, perform backpropagation, and update potentially billions of parameters across many iterations. Large-scale training can take substantial computing time and specialized hardware.

Training is not only pre-training

“Training” can refer to several kinds of parameter-updating work:

• Pre-training develops broad language capability from a very large text corpus using next-token prediction.
• Fine-tuning updates an existing model on a smaller, selected dataset to adapt its behavior, style, or competence for a particular task.
• Post-training or alignment further updates a model to better follow instructions, produce useful formats, or meet safety requirements.

All three involve changing some model parameters. The amount and type of data differ, but the defining feature remains the same: the model is being modified based on examples or feedback.

---

What happens during inference?

Inference begins after the model has been trained and deployed. A user sends a question, such as:

What is the refund period for this product?

The application constructs the full prompt. It may contain system instructions, prior conversation, a user question, and — in a RAG application — retrieved source passages.

The LLM processes that prompt using its already-learned, fixed parameters. As covered in the previous lesson, it performs a forward computation to assign probabilities to possible next tokens, chooses a token, then repeats until it reaches a stopping condition.

The key difference is what is missing:

• There is no known “correct next token” supplied to the model during the request.
• There is no loss calculation for the purpose of learning.
• There is no backpropagation.
• There is no parameter update.

Inference is therefore commonly described as forward-only and read-only with respect to the model weights.

This comparison captures the central contrast:

Question	Training	Inference
Does the model see a target answer?	Yes, through training examples or feedback	No, it only receives the current prompt
Does it calculate a learning error?	Yes	No
Does it use backpropagation?	Yes	No
Do weights change?	Yes	No
Does it serve a live user answer?	Usually no	Yes
Main concern	Learning quality and cost	Latency, reliability, throughput, and cost per request

---

Where RAG fits: external context, not model learning

Now connect the distinction directly to RAG.

Imagine a support assistant has access to this newly written policy:

Source: Warranty policy, revised May 2026
A replacement request must be filed within 45 days of delivery.

A user asks:

How long do I have to request a replacement?

A RAG application can retrieve that policy passage and place it alongside the user’s question in the prompt. During inference, those tokens influence the model’s next-token probabilities. A properly instructed model can answer:

You must file a replacement request within 45 days of delivery.

The model did not absorb this policy into its weights. If the same model later receives a prompt without the policy passage, it has not thereby gained a dependable permanent memory of the “45 days” rule.

This distinction has practical consequences:

Change you make	What changes?	Requires training?
Add a current policy document to a RAG knowledge base	The external evidence available for retrieval	No
Retrieve different chunks for each question	The current prompt	No
Rewrite system instructions	The current prompt	No
Fine-tune on many ideal customer-support responses	Some model parameters	Yes
Train a new base model	The model’s parameters at large scale	Yes

A RAG system often has an offline ingestion workflow in which it cleans documents, splits them into chunks, creates embeddings, and stores them in a search index. Despite happening before a user asks a question, that is still not LLM training. It prepares external material so it can be retrieved later.

When the user asks a question, the online RAG workflow retrieves relevant passages and supplies them to the LLM at inference time. The LLM then generates an answer using both:

1. its general capabilities learned during training, and
2. the specific evidence in the current prompt.

---

Common misconceptions to avoid

“The model learns from every chat message”

Usually, no. A deployed model normally does not update its weights while responding to your request. A provider may log interactions or later use selected data in a separate training process, subject to its policies and settings, but that would be a separate workflow — not live inference.

“Putting a document in the prompt trains the model on it”

No. It gives the model temporary context. Once the request is complete, the model’s parameters are unchanged.

“RAG makes the LLM know the document forever”

No. RAG makes relevant document text available when the retriever finds it and the prompt includes it. Its benefit is grounded, updatable context rather than permanent model learning.

“Inference is just looking up a stored answer”

Not exactly. The model computes a new token sequence based on the prompt and its fixed weights. It can use a supplied source passage, but it still generates the final wording token by token.

A useful diagnostic question is:

After this process, have the model’s parameters changed?

If yes, it was training or fine-tuning. If no, it was inference, even if the prompt contained many documents.

---

Key takeaways

Training and inference both involve a model making next-token predictions, but they have opposite purposes:

• Training compares predictions with known examples, calculates error, and updates parameters so the model improves.
• Inference uses fixed parameters to generate an answer for a new prompt.
• Inference is forward-only: it does not backpropagate errors or modify weights.
• Pre-training, fine-tuning, and alignment are forms of training because each changes the model’s learned parameters.
• RAG supplies documents as temporary prompt context during inference. It adds current or private knowledge without retraining the LLM.

Next, you will examine a practical limitation that follows from this distinction: why a model’s trained knowledge can be outdated, incomplete, private-data blind, or too specialized — precisely the gap RAG is designed to address.