Supervised Fine-Tuning: Adapting Pre-trained Models

The Problem: From Pattern Mimicry to Instruction Following

A base model fresh from pre-training is a master of statistical pattern completion. Its sole objective is to predict the next token in a sequence based on the trillions of tokens it has seen from the internet. While this process builds a profound understanding of language, it does not inherently teach the model the concept of being a "helpful assistant."

This leads to what is often called the "parrot problem." If you give a base model a prompt like, "Explain the moon landing to a six-year-old," it might not provide a simple explanation. Instead, it might complete the prompt with a similar-looking question, such as, "Explain the theory of gravity to a six-year-old." Why? Because it has learned from web data (like FAQs or lists of questions) that one question is often followed by another. The model isn't being unhelpful; it's perfectly executing its pre-training objective of completing the most statistically likely text sequence.

To bridge this gap, we need to explicitly teach the model the desired conversational behavior. This is the role of Supervised Fine-Tuning.

The SFT Solution: Learning from Expert Demonstrations

SFT reframes the training objective. Instead of continuing to predict the next token on unstructured internet text, we train the model on a curated, high-quality dataset of prompt-response pairs. Think of these as expert-written "flashcards" that demonstrate the exact behavior we want the model to learn.

The dataset consists of examples like:

• Prompt: Can you write a short introduction about the relevance of the term monopsony?
• Response: Monopsony refers to a market structure in which there is only one buyer for a particular good or service...

By training on thousands of these high-quality examples, the model learns to recognize the pattern of a user's instruction followed by a helpful, relevant, and well-formatted assistant response.

The Core Mechanism: Loss Masking

Now we arrive at the central technical question: how do we train the model to generate the response given the prompt? A naive approach would be to concatenate the prompt and response into a single sequence and continue the next-token prediction training. However, this would be incorrect. It would teach the model to predict the prompt tokens just as much as the response tokens, which is not our goal. We want the model to learn to generate the response when it sees the prompt.

The solution is an elegant engineering trick called loss masking. The process is as follows:

1. Format the Data: The prompt and response are formatted into a single sequence, often using special tokens to delineate the roles. For example: <|user|> Explain gravity. <|assistant|> Gravity is the force by which a planet or other body draws objects toward its center.
2. Create Input and Label Tensors: This single sequence is tokenized to create the input_ids that the model will see. A parallel labels tensor is also created.
3. Apply the Mask: This is the key step. In the labels tensor, the token IDs corresponding to the user's prompt (and any special formatting tokens) are replaced with a special value, typically -100. The token IDs for the assistant's response are left untouched.
4. Calculate Loss: When this pair of input_ids and labels is fed to the model, the cross-entropy loss function (which we studied in Module 5) is configured to ignore any positions where the label is -100.

As a result, the model sees the full context (prompt + response) but the training signal (the gradient) is calculated only based on its ability to correctly predict the tokens of the assistant's response. It learns the crucial rule: "When I see this kind of prompt, my job is to generate this kind of response."

The SFT Objective, Formalized

Your background in mathematics and computer science will appreciate that this process has a precise formal objective. SFT is an instance of maximum likelihood estimation, equivalent to behavior cloning in imitation learning. We are training the model's policy, ${\pi }_{\theta }$, to mimic the behavior of an expert (the human labeler).

Given a dataset $D$ of prompt-response pairs $(x,y)$, the SFT objective is to find the parameters $\theta$ that maximize the conditional log-likelihood of the expert responses:

${\mathcal{L}}_{\text{SFT}}(\theta )={\mathbb{E}}_{(x,y)\sim D}[\log {\pi }_{\theta }(y|x)]$

Expanding the log probability of the sequence $y$ of length $T$, this is equivalent to minimizing the negative log-likelihood, which is our familiar cross-entropy loss, summed over only the tokens in the target response:

${\text{Loss}}_{\text{SFT}}(\theta )={\mathbb{E}}_{(x,y)\sim D}\left[-{\sum }_{t=1}^T\log p_{\theta }(y_t\mid x,y_{<t})\right]$

Here, $y_t$ is the target token at timestep $t$ in the response, and $y_{<t}$ are the preceding response tokens. The loss masking we just discussed is the practical implementation of ensuring this summation runs only over the response tokens $y_t$, not the prompt tokens $x$.

The Broader SFT Process: Data is Everything

While loss masking is the core technical trick, the success of SFT hinges almost entirely on the quality of the fine-tuning data.

As highlighted in the reading, SFT primarily teaches the model behavioral adaptation and task specialization. It doesn't inject new factual knowledge; rather, it teaches the model how to access, format, and present the knowledge it already gained during pre-training. This is why SFT can be effective with surprisingly small datasets (thousands to tens of thousands of examples), provided they are of high quality.

As noted in the Stanford lecture from our previous lesson, this process is about teaching the model which "persona" from its vast pre-training data it should adopt—specifically, the persona of a helpful assistant that answers questions directly, rather than the persona of a quiz generator or a forum user.

Conclusion

In this lesson, we dissected the mechanics of Supervised Fine-Tuning, the foundational technique for turning a pre-trained LLM into an instruction-following assistant. We went from the high-level concept to the low-level implementation details that make it work.

Key takeaways include:

• SFT addresses the "parrot problem": It teaches a base model, which is a pure pattern completer, how to follow instructions and act as a helpful assistant.
• It works by imitation: SFT trains the model on a high-quality dataset of (prompt, response) pairs, demonstrating the desired behavior.
• Loss masking is the key mechanism: By setting the labels for prompt tokens to an "ignore" value (-100), we ensure that the training loss is calculated only on the model's ability to generate the correct response.
• The objective is behavior cloning: Mathematically, SFT maximizes the likelihood of the expert's responses given the prompts, using the standard cross-entropy loss function applied selectively.
• Data quality over quantity: SFT reshapes existing knowledge rather than teaching new facts. Therefore, the quality, style, and diversity of the fine-tuning dataset are the most critical factors for success.

You now understand how to perform a "full" fine-tuning, where all of the model's parameters are updated to learn a new behavior. However, this process can be computationally expensive and resource-intensive. What if there were a more efficient way?

In our next lesson, we will explore Parameter-Efficient Fine-Tuning (PEFT), focusing on the popular Low-Rank Adaptation (LoRA) method, which offers a powerful way to adapt LLMs with a fraction of the computational cost.