Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 68

SFT vs. Alignment: Unpacking the Differences

Recommended resources

Introduction

In our previous lessons, we've focused on Supervised Fine-Tuning (SFT), mastering the techniques and hyperparameters needed to adapt a pre-trained model to a specific style or domain. You now know how to take a base model and, through curated examples, teach it to follow instructions and generate outputs in a desired format. This process makes a model capable.

However, a capable model is not necessarily a helpful, harmless, or reliable one. SFT teaches a model to imitate correct answers, but it doesn't explicitly teach it what makes one answer preferable to another from a human perspective. This is where the concept of alignment comes in.

Today, we'll explore this crucial next step in refining LLMs. Our goal is to explain the conceptual difference between supervised fine-tuning (SFT) and alignment techniques like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO). Understanding this distinction is fundamental to building models that are not just technically proficient but also genuinely useful and aligned with human values.

1. From Instruction-Following to Preference-Alignment

A base model, trained only on next-token prediction over a vast corpus, has immense knowledge but little sense of how to apply it in a conversational or task-oriented way. Supervised Fine-Tuning is the first step in correcting this.

Let's begin by watching a short video that clearly demonstrates the gap between a base model, which is unhelpful, and an instruction-tuned model, which has undergone alignment. This will highlight why we need to go beyond simple SFT.

Fine-tuning LLMs on Human Feedback (RLHF + DPO)

The video 'Fine-tuning LLMs on Human Feedback' by Shaw Talebi starts with a perfect example of why alignment is necessary, comparing a base Llama model's response to an aligned model's response for the same prompt.

Watch from 00:27 to 03:33. Notice how the powerful base model fails at a simple request, while the aligned model (InstructGPT) provides a helpful, structured answer. This illustrates the behavioral change that alignment techniques aim to achieve.

As you saw, the base model completes the text in a way that is statistically plausible but completely useless to the user. The aligned model, however, understands the user's intent and provides a helpful, formatted response. This difference is the result of moving from pure imitation (SFT) to preference optimization (alignment).

2. The Three Stages of Fine-Tuning

To build a truly helpful AI assistant, the process typically involves three phases, building upon each other:

  1. Pre-training: The model learns grammar, facts, and reasoning by predicting the next token on web-scale text. This creates the knowledgeable but unhelpful base model.
  2. Supervised Fine-Tuning (SFT): The model learns to follow instructions and respond in a specific format (e.g., as a chatbot). It is trained on high-quality (instruction, response) pairs.
  3. Alignment (RLHF/DPO): The model's behavior is refined to better match nuanced human preferences, making it more helpful, harmless, and honest. It is trained on preference data, such as (prompt, chosen_response, rejected_response).

Let's quickly review the SFT stage and its limitations before diving into alignment.

Reinforcement Learning with Human Feedback (RLHF), Clearly Explained!!!

The 'StatQuest with Josh Starmer' video on RLHF provides an exceptionally clear explanation of SFT and why it's not the final step.

Watch from 02:18 to 06:52. Focus on how SFT uses prompt-response pairs to align a pre-trained model and, crucially, understand the problem it introduces: potential overfitting and poor generalization due to the limited size of high-quality SFT datasets.

The key takeaway is that while SFT teaches the model to be a good instruction-follower on seen examples, it struggles to generalize the principles of being a good assistant to new, unseen prompts. Furthermore, creating a massive, diverse, and perfectly-written SFT dataset is prohibitively expensive.

This is the problem that alignment techniques were designed to solve.

3. The Classic Approach: Reinforcement Learning from Human Feedback (RLHF)

RLHF was the breakthrough technique (popularized by OpenAI's InstructGPT) that enabled large-scale alignment. The core idea is simple: it's often easier for humans to compare two responses and say which is better than to write a perfect response from scratch.

RLHF formalizes this into a three-step process:

  1. Start with an SFT model. This is the baseline policy.
  2. Train a Reward Model (RM).
    • For a given prompt, generate several different responses from the SFT model.
    • Have human labelers rank these responses from best to worst.
    • This creates a preference dataset.
    • Train a separate model (the Reward Model) on this data. Its job is to take a (prompt, response) pair and output a single scalar score—a "reward"—that predicts the human preference. In essence, the RM becomes a scalable, automated proxy for a human judge.
  3. Fine-tune the SFT model using Reinforcement Learning.
    • The SFT model (now called the "policy") generates a response to a new prompt.
    • The Reward Model scores that response.
    • A reinforcement learning algorithm, typically Proximal Policy Optimization (PPO), updates the policy's weights to maximize the reward score.
    • A penalty term (KL divergence) is used to prevent the policy from deviating too much from the original SFT model, ensuring it doesn't "forget" its capabilities or find a bizarre way to "hack" the reward model.

The following resource provides a structured breakdown of this complex architecture.

The Complete Guide to LLM Fine-tuning: Traditional RLHF Approach

Let's read a section from 'The Complete Guide to LLM Fine-tuning' that details the components and challenges of the traditional RLHF approach.

Read the section titled 'Traditional RLHF Approach.' Focus on understanding the three distinct components (Reward Model, PPO, Policy Model) and the associated computational and implementation challenges. This will make it clear why simpler alternatives were sought.

In summary, the conceptual difference is:

  • SFT: Supervised learning. Goal is to imitate a known, "correct" output. The feedback is explicit and absolute: generate Y for X.
  • RLHF: Reinforcement learning. Goal is to optimize a policy to maximize a learned preference score. The feedback is relative and learned: Y1 is better than Y2 for X.

However, as you just read, RLHF is notoriously complex, computationally expensive, and unstable to train. This led researchers to ask: can we achieve the same outcome more directly?

4. The Modern Approach: Direct Preference Optimization (DPO)

Direct Preference Optimization (DPO) was a groundbreaking development that achieves the goal of RLHF without the reinforcement learning or the separate reward model.

The key insight of DPO is that the reward modeling and policy optimization steps can be combined into a single, elegant supervised learning problem. It reformulates the RLHF objective so it can be optimized directly using preference data.

Here is the DPO process:

  1. Start with an SFT model. As before, this serves as the "reference policy," but now it's kept frozen.
  2. Use a preference dataset of (prompt, chosen_response, rejected_response).
  3. Directly fine-tune the SFT model with a new loss function. This DPO loss function effectively trains the model to do two things simultaneously:
    • Increase the log-probability of generating the chosen_response.
    • Decrease the log-probability of generating the rejected_response.

Essentially, the language model being trained learns to act as its own reward model. The training signal directly steers the model towards preferred outputs and away from dispreferred ones, all within a standard supervised fine-tuning framework.

This next resource elegantly explains the innovation of DPO and its advantages over RLHF.

The Complete Guide to LLM Fine-tuning: The DPO Revolution

Continuing with 'The Complete Guide to LLM Fine-tuning,' let's read the section on DPO to see how it simplifies the alignment process.

Read the section titled 'The DPO Revolution.' Pay close attention to the 'Key Advantages Over Traditional RLHF.' This directly contrasts the two methods and highlights why DPO has become so popular.

The image below provides a great high-level conceptual summary of the difference between the two alignment pipelines.

Conceptual Comparison of RLHF vs. DPO
This diagram contrasts the complex, multi-stage RLHF process (left) with the streamlined, direct optimization process of DPO (right). RLHF involves training a separate reward model and using an RL algorithm, while DPO directly optimizes the policy on preference pairs.
Test your understanding!

You are tasked with aligning a 7B parameter SFT model to be a better creative writer. You have a limited GPU budget and a team with deep experience in supervised learning but no experience with reinforcement learning.

Which alignment technique, RLHF or DPO, would be more appropriate and why?

Show answer

DPO would be the more appropriate choice. Here's why:

  1. Limited GPU Budget: DPO is more computationally efficient. It avoids training a separate reward model and doesn't require storing multiple model copies in memory during the PPO phase, making it less resource-intensive than RLHF.
  2. Team Experience: DPO reframes the alignment problem as a supervised learning task. A team experienced in standard fine-tuning can easily adapt to the DPO framework, as it uses a familiar training loop with a different loss function. RLHF, with its PPO algorithm and complex training dynamics, would require the team to learn a completely new and notoriously difficult paradigm.

Summary: SFT vs. RLHF vs. DPO at a Glance

Let's consolidate the key conceptual differences in a table.

Feature Supervised Fine-Tuning (SFT) Reinforcement Learning from Human Feedback (RLHF) Direct Preference Optimization (DPO)
Primary Goal Imitate a specific style or format. Optimize behavior based on a learned model of human preferences. Directly Optimize behavior based on human preferences.
Data Format (prompt, ideal_response) (prompt, chosen_response, rejected_response) (prompt, chosen_response, rejected_response)
Training Paradigm Supervised Learning Reinforcement Learning (PPO) Supervised Learning (direct optimization with a special loss)
Key Components 1. Policy Model 1. Policy Model
2. Reward Model
3. Reference Model (for KL penalty)
4. (Often) Value Model for PPO		 1. Policy Model
2. Frozen Reference Model
Complexity Low High (unstable, hard to debug, computationally expensive) Medium (stable, easier to implement, more efficient)
Core Question "How can I replicate this good example?" "How can I generate responses that a judge would score highly?" "How can I make preferred responses more likely and unpreferred ones less likely than they were before?"

Conclusion

You now understand the critical distinction between making a model capable (SFT) and making it aligned (RLHF/DPO). While SFT is the essential first step to teach a model how to follow instructions, alignment is what refines its judgment to better match nuanced human preferences.

Key Takeaways:

  • SFT teaches imitation: It uses high-quality examples to teach a model how to answer.
  • Alignment teaches preference: It uses human comparisons to teach a model what kind of answer is better.
  • RLHF is the classic, complex approach: It involves training a separate reward model and using reinforcement learning (PPO) to optimize the LLM policy.
  • DPO is the modern, direct approach: It achieves the same alignment goal as RLHF but with a simpler, more stable supervised learning objective, making it more efficient and accessible.

As an elite LLM engineer, knowing when and why to progress from SFT to an alignment phase is a crucial architectural decision. You're now equipped with the conceptual framework to make that choice.

What's Next?

We've covered the full fine-tuning lifecycle: preparing data, performing SFT, and refining with alignment. But how do we know if our efforts were successful? In the next lesson, we will focus on the final, critical step: evaluating a fine-tuned model against its base model using both automated benchmarks and custom evaluation sets.