Supervised Fine-Tuning (SFT) for Instruction Following

1. The Parrot Problem: Why Base Models Need SFT

A base Large Language Model (LLM) is pre-trained on a massive, unstructured corpus of text from the internet. Its sole objective is next-token prediction. It becomes a master of mimicking linguistic patterns but has no concept of a user's intent or the goal of being helpful.

This leads to what is often called the "parrot problem." The model simply completes a text sequence based on the most statistically likely pattern it has seen during pre-training, which often isn't what a user wants.

The following video provides an excellent explanation of this disconnect and why a pre-trained model is not a helpful assistant.

As the video explains, if you ask a base model What is the primary cause of Earth's seasons?, it's more likely to complete the pattern as a multiple-choice question than to provide a direct answer. This happens because it learned from countless online quizzes that this question format often precedes a list of options. The model isn't being unhelpful; it's just perfectly executing its pre-training objective. SFT is the process we use to change this objective.

2. The SFT Solution: Imitating Expert Examples

Supervised Fine-Tuning (SFT), sometimes called Instruction Fine-Tuning (IFT), solves the parrot problem by changing the model's training diet. Instead of unstructured web text, we feed it a smaller, high-quality, and highly structured dataset.

This dataset consists of thousands of prompt-completion pairs. Each example is like a flashcard that shows the model how an expert assistant should behave.

• Prompt: An instruction given by a user.
• Completion (or Response): The ideal, helpful response an assistant should generate.

The following reading explains the core idea and provides the formal mathematical objective for SFT.

The key takeaway is that we are adapting the pre-trained model by further training it on this new, labeled dataset. The optimization objective is to minimize the negative log-likelihood of the expert completion, conditioned on the prompt. Formally, this is expressed as:

Don't be intimidated by the math. This equation simply states that we are only calculating the loss on the tokens of the desired completion $c_k$, not the prompt. This is the central trick of SFT, which we will now explore in detail.

3. The Mechanics of SFT

To implement SFT, we need to solve a practical problem: how do we format our prompt-completion pairs and tell the model to only learn from the completion part? This is done through two key techniques: chat templates and loss masking.

3.1. Formatting Data with Chat Templates

An LLM processes a single, continuous sequence of tokens. Therefore, we must combine our prompt and completion into one string, while also teaching the model the turn-taking structure of a conversation. We achieve this using a chat template.

A chat template is a standardized format that uses special tokens to delineate roles. A very popular one is the "Alpaca" format.

Below is an instruction that describes a task. Write a response that appropriately completes the request.




### Instruction:
{Your Prompt Here}




### Response:
{The Expert Completion Here}

Different models use different templates. The following video and image illustrate how raw data is formatted using these templates.

3.2. Loss Masking: The Core Trick

Now we face the critical issue. If we train the model on this entire formatted string, it will be learning to predict the user's prompt just as much as the assistant's response. This is wrong. We only want to compute the loss on the tokens corresponding to the ### Response: part.

The solution is loss masking. We create two token sequences:

1. input_ids: The full tokenized conversation (prompt + completion). This is what the model sees for context.
2. labels: A copy of the input_ids, but with the tokens corresponding to the prompt replaced by a special value, typically -100.

PyTorch's cross-entropy loss function is specifically designed to ignore any token with a label of -100. This elegantly forces the model to calculate loss only on its ability to generate the expert completion.

This next video segment provides a brilliant walkthrough of this exact mechanism.

The implementation of the prepare_SFT_batch function you just saw is the heart of the SFT process. By setting the prompt labels to -100, we perfectly align our practical implementation with the formal objective ${\mathcal{L}}_{\text{IFT}}$ we saw earlier. The model sees the full context but is only penalized for mistakes it makes when generating the desired response. This is what turns the "parrot" into an "apprentice assistant."

4. SFT's Role and Its Limitations

With SFT, we have successfully taught the model to:

• Understand structure: It recognizes user and assistant roles.
• Adopt a persona: It learns to imitate the helpful style of the expert data.
• Follow instructions: It can now perform tasks shown in its training data.

SFT is the foundational step that makes a base model controllable and useful. However, it has a profound limitation: it is based on imitation, not judgment.

SFT treats all correct answers in the dataset as equally good. It has no mechanism for understanding that one good answer might be more helpful, safer, or more insightful than another.

The following resources elaborate on this crucial weakness, which motivates the need for the RL-based alignment methods we will study next.

Because SFT cannot learn these relative preferences, we need a different paradigm. We need to move from simple imitation to optimizing a policy based on a reward signal that reflects human preferences. And that is exactly what RLHF does.

Conclusion

In this lesson, we deconstructed the first and most fundamental step of LLM alignment: Supervised Fine-Tuning.

Key Takeaways:

• Pre-trained base models are next-token predictors and cannot follow instructions, a phenomenon known as the "parrot problem."
• SFT adapts a base model by training it on a curated dataset of high-quality prompt-completion pairs.
• Chat templates are used to format the prompt and completion into a single text sequence with special tokens indicating conversational roles.
• The core mechanism of SFT is loss masking, where prompt tokens in the labels tensor are replaced with -100 so that the loss is only computed on the model's ability to generate the expert completion.
• The main limitation of SFT is that it teaches imitation, not judgment. It cannot learn relative preferences between different good responses.

Preview of the Next Lesson:
We have successfully turned our base model into an "apprentice assistant." It knows how to listen and respond. Now, we must teach it what we prefer. In the next lesson, we will explore Step 2 of the alignment pipeline: training a reward model. We will see how pairwise human preference data (choosing between two responses) is used to train a model that can score the quality of any given completion, providing the crucial reward signal for our subsequent RLHF training.