Recommended resources
The freely available resources this lesson will direct you to.
- Reinforcement Learning with Human Feedback (RLHF), Clearly Explained!!!www.youtube.com
- LLM Training & Reinforcement Learning from Google Engineer | SFT + RLHF | PPO vs GRPO vs DPOwww.youtube.com
- Understanding & Using Supervised Fine-Tuning (SFT)cameronrwolfe.substack.com
- Supervised Fine-Tuning (SFT) for LLMswww.geeksforgeeks.org
Hello! Welcome to your next lesson in our journey through LLM training concepts.
In our last lesson, we explored the massive, foundational process of pre-training. We saw how a model learns general knowledge about language, facts, and reasoning by predicting the next token across trillions of words from the internet. This results in a powerful, but raw, "base model."
Today, we'll focus on the next crucial step. A base model knows a lot, but it doesn't inherently know how to be a helpful, instruction-following assistant. This lesson addresses the learning outcome: Explain what supervised fine-tuning (SFT) achieves and how it differs from pre-training. We will transform our raw base model into something that can understand and respond to your specific requests.
1. From Raw Knowledge to Aligned Assistant
A pre-trained model is like an incredibly well-read person who has memorized the entire library but has never had a conversation. If you ask it a question, it might just continue your sentence with the most statistically likely text from a book, rather than actually answering you. In LLM terminology, this model is unaligned with user intent.
The first step to fix this is alignment. To get a clear idea of what we mean by "unaligned" and the overall training roadmap, let's start with a short, clear explanation.
Reinforcement Learning with Human Feedback (RLHF), Clearly Explained!!!
The following clip from StatQuest's 'Reinforcement Learning with Human Feedback (RLHF), Clearly Explained!!!' does an excellent job of illustrating why a pre-trained model isn't useful out-of-the-box and sets the stage for the two main alignment steps: SFT and RLHF.
Starting a little over two minutes in, please watch the pre-training explanation. Focus on the concept of an 'unaligned' model and how its pre-training objective (predicting the next token in Wikipedia) differs from how we want to use it (as a helpful chatbot).
As the video explains, we need to bridge the gap between what the model was trained to do (next-token prediction on web data) and what we want it to do (be a helpful assistant). This brings us to the first major alignment technique: Supervised Fine-Tuning.
2. What is Supervised Fine-Tuning (SFT)?
Supervised Fine-Tuning (SFT) is the process of taking a pre-trained LLM and further training it on a smaller, high-quality dataset of instruction-response pairs. The "supervised" part comes from the fact that we provide the model with explicit, "correct" examples to learn from.
Think of it as giving the model a stack of flashcards. Each card has a prompt (e.g., "Explain gravity like I'm five") on one side and an ideal, human-written answer on the other. The model's goal is to learn to produce the answer when it sees the prompt.
This diagram provides a great visual summary of the process, contrasting it with the pre-training we learned about previously.
Now, let's watch a brief segment that walks through this process.
Reinforcement Learning with Human Feedback (RLHF), Clearly Explained!!!
Let's return to the StatQuest video, which now explains the mechanics of SFT.
At about 4 minutes and 49 seconds in, watch supervised fine-tuning. Pay attention to how the dataset of human-created prompt/response pairs is used to train the model.
3. What SFT Achieves: Style and Specialization
The primary goal of SFT is alignment. It teaches the model the style and behavior of a helpful assistant. By training on thousands of examples where the response is helpful, harmless, and follows instructions, the model learns to generalize this behavior to new, unseen prompts.
It's important to distinguish SFT from other forms of fine-tuning that aim to create a "narrow expert." For example, you could fine-tune a model exclusively on medical textbooks. It might become a medical expert but lose its ability to write a poem. SFT, when done for general alignment, is different. It aims to teach a behavior without sacrificing the model's broad, general-purpose capabilities.
To deepen this point, let's read a short article that clarifies this distinction.
Understanding & Using Supervised Fine-Tuning (SFT)
The article 'Understanding and Using Supervised Fine-Tuning (SFT) for Language Models' provides excellent insights into the nuances of SFT.
Please read two short sections from this article: Just below the heading 'What is SFT?', read the SFT definition to get a crisp explanation of the mechanism. Then, find the section under the heading 'Fine-tuning before SFT'. Focus on the final paragraph and read the alignment contrast that differentiates generic fine-tuning (creating a 'narrow expert') from SFT for alignment. This will clarify what SFT achieves beyond simply learning a new task.
4. SFT vs. Pre-training: The Key Differences
You now have a good sense of both processes. Let's formalize the distinction. While both pre-training and SFT often use the same underlying mechanism (next-token prediction), they are fundamentally different in their goals, data, and scale.
Here's a breakdown of the core differences:
| Aspect | Pre-training | Supervised Fine-Tuning (SFT) |
|---|---|---|
| Goal | Learn general language patterns, facts, and reasoning from vast text. | Align the model to follow instructions and adopt a helpful, conversational style. |
| Data Source | Massive, mostly unlabeled text and code from the internet (e.g., Common Crawl, Wikipedia). | A smaller, curated dataset of high-quality, labeled prompt-response pairs. |
| Data Scale | Trillions of tokens. | Thousands to hundreds of thousands of examples. |
| Data Quality | Quantity over quality (though extensive cleaning is done). | Quality over quantity. The examples must be excellent. |
| Starting Point | Starts from scratch with random model weights. | Starts from the weights of a pre-trained model. |
| Compute Cost | Extremely high (months, thousands of GPUs, millions of dollars). | Relatively low (hours/days, a few GPUs, hundreds/thousands of dollars). |
To solidify this, let's hear a Google Engineer describe these differences from a practitioner's perspective.
LLM Training & Reinforcement Learning from Google Engineer | SFT + RLHF | PPO vs GRPO vs DPO
This video from Martin Is A Dad provides a very clear breakdown of the SFT process and its differences from pre-training.
About four and a half minutes in, watch the SFT explanation. The speaker explicitly lists the key differences between SFT and pre-training, focusing on the data and the objective.
5. The Limits of SFT
SFT is a powerful and essential step, but it has limitations.
- Dataset Dependency: The model's quality is heavily dependent on the quality and diversity of the SFT dataset. It's difficult and expensive for humans to write perfect answers for every imaginable type of prompt.
- Scalability: As the model gets better, it can generate responses that are as good as human-written ones, making it hard to create new, even better SFT data.
- Ambiguity: For many prompts, there isn't one single "correct" answer. SFT forces the model to learn from one specific response, when many others could also be good.
Because of these challenges, SFT is usually just the first step in the alignment process. It gets the model into the right ballpark of behavior, but another technique is needed to truly refine it.
Understanding & Using Supervised Fine-Tuning (SFT)
Let's return to the Cameron Wolfe article, which discusses the pros and cons of SFT and makes a clear case for why it isn't the final step.
Please read the section under the heading 'Pros and Cons of SFT'. Just below the heading, read the pros and cons. Focus on the challenges discussed, particularly around dataset creation and the conclusion that adding RLHF is beneficial.
Conclusion
In this lesson, we've seen how Supervised Fine-Tuning takes a raw, pre-trained model and begins the process of aligning it to be a useful AI assistant.
Key Takeaways:
- SFT Achieves Alignment: Its main purpose is to teach a pre-trained model how to follow instructions and respond in a helpful, conversational manner, without losing its general capabilities.
- It's "Supervised": SFT works by training the model on a curated dataset of high-quality prompt-response pairs, providing explicit examples of desired behavior.
- SFT vs. Pre-training: SFT is fundamentally different from pre-training in its goal (alignment vs. general knowledge), data (small, labeled vs. massive, unlabeled), and scale (cheap/fast vs. expensive/slow).
- SFT is Step One: While crucial, SFT is limited by the quality and scale of its human-generated data. It's the first step in a multi-stage alignment process.
Preview of Next Lesson:
The limitations of SFT—especially the difficulty of defining a single "best" answer—lead directly to the next phase of alignment. In our next lesson, we will dive into Reinforcement Learning from Human Feedback (RLHF). We'll see how models can learn from human preferences (i.e., rating which of two responses is better) to become even more helpful and nuanced.