Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 46

The RLHF Pipeline: SFT, Reward Models, and PPO

Recommended resources

Introduction

Welcome back! In our previous lessons, we built the first two fundamental components for aligning a large language model. First, we used Supervised Fine-Tuning (SFT) to teach a base model how to follow instructions and respond in a specific format, creating our "student" model. Then, we trained a Reward Model (RM) on human preference data, creating an automated "judge" that can score how "good" a response is.

Today, we will put all the pieces together. Our goal is to understand the complete, standard pipeline for Reinforcement Learning from Human Feedback (RLHF). We will see how the SFT model and the RM are used in a final optimization loop to produce a highly capable and aligned model like ChatGPT or Claude.

This lesson directly addresses the learning outcome: Outline the standard RLHF pipeline: SFT, reward model training, and policy optimization.

We will cover:

  • A high-level overview of the three main stages of the RLHF process.
  • An introduction to the final stage: policy optimization using an RL algorithm called Proximal Policy Optimization (PPO).
  • The roles of the different models involved in this final stage.

Let's begin by visualizing the entire process.

The Three Steps of the RLHF Pipeline
A high-level illustration of the three steps in the RLHF pipeline. We have already covered Step 1 (SFT) and Step 2 (RM Training). Today, we focus on how they combine in Step 3 (Policy Optimization).

The Three-Step RLHF Pipeline

The modern RLHF process can be broken down into a clear, three-step sequence. We'll briefly recap the first two steps before diving into the third.

  • Step 1: Supervised Fine-Tuning (SFT). We start with a general-purpose, pre-trained LLM. We then fine-tune it on a smaller, high-quality dataset of (prompt, response) pairs created by human demonstrators. The goal is to teach the model the desired interaction format (e.g., how to answer questions, follow instructions) through imitation. The output of this stage is our SFT model.

  • Step 2: Reward Model (RM) Training. We use our SFT model to generate several different responses for a given set of prompts. Human labelers then rank these responses, creating a dataset of preferences (e.g., (prompt, chosen_response, rejected_response)). We use this preference data to train a reward model, which learns to assign a scalar "goodness" score to any given (prompt, response) pair, effectively acting as a proxy for human judgment.

  • Step 3: Policy Optimization via RL. In this final, crucial stage, we use the reward signal from the RM to further fine-tune our SFT model. Here, the SFT model is treated as a policy in a reinforcement learning framework. The policy's goal is to generate responses that maximize the score given by the reward model. This is typically done using an algorithm like Proximal Policy Optimization (PPO).

Let's watch a short, clear overview of these stages to solidify our understanding.

LLM Training & Reinforcement Learning from Google Engineer | SFT + RLHF | PPO vs GRPO vs DPO

The video 'LLM Training & Reinforcement Learning from Google Engineer' by Martin Is A Dad provides a concise walkthrough of this three-step framework.

Please watch from 02:43 to 04:51. Focus on how the narrator breaks down the process into SFT, training a reward model from comparison data, and finally, optimizing a policy against that reward model.

Deep Dive into Step 3: Policy Optimization

Now that we have the full picture, let's focus on the final and most complex stage: policy optimization. This is where the "reinforcement learning" in RLHF comes into play.

In this stage, we have four key components working together. Understanding their roles is essential.

  1. The Policy Model (Actor): This is the model we are actively training. It is initialized with the weights of our SFT model from Step 1. In RL terms, this is the "actor" that takes an action (generates a token) based on the current state (the prompt and previous tokens).
  2. The Reward Model (The Judge): This is the RM we trained in Step 2. Its weights are frozen during this stage. Its only job is to provide a scalar reward for the policy's output.
  3. The Reference Model (The Anchor): This is a frozen, untouched copy of the SFT model from Step 1. Its purpose is to act as a baseline. The training process penalizes the active policy if its outputs diverge too much from this reference. This prevents the model from "forgetting" its language capabilities or finding a bizarre, nonsensical output that happens to get a high reward (a phenomenon known as "reward hacking"). This penalty is measured using the Kullback-Leibler (KL) divergence.
  4. The Value Model (Critic): In PPO, we also train another small model, called a "critic" or "value model". Its job is to predict the expected future reward from a given state. While the reward model tells us the immediate reward of a single response, the value model helps stabilize the training process by providing a more stable estimate of the overall "goodness" of a situation. You can think of it as helping the actor make smarter, more forward-looking updates.

The following video explains this PPO workflow in the context of RLHF, highlighting these four models.

LLM Training & Reinforcement Learning from Google Engineer | SFT + RLHF | PPO vs GRPO vs DPO

Let's return to the 'LLM Training & Reinforcement Learning' video to see how these components interact.

Watch from 12:13 to 15:28. Pay close attention to the four models identified: the policy model, the value model (critic), the reward model, and the reference model. Try to map each one to the roles described above.

This interaction is also shown in the diagram below, which is a more detailed version of the one we saw earlier.

RLHF Finetuning Pipeline Overview
A detailed diagram of the policy optimization step using PPO. It shows the Policy (Actor), the Reference Model, the Reward Model (RW), and the Value Model (Critic) all working together to update the policy.

Tying It All Together: The Full Process

We now have all the conceptual pieces. Let's formalize them with a textual resource and a final end-to-end example.

The Hugging Face blog provides a clear, written explanation of the RL fine-tuning stage, defining the key terms within the RL framework.

Illustrating Reinforcement Learning from Human Feedback (RLHF)

The blog post 'Illustrating Reinforcement Learning from Human Feedback (RLHF)' gives a solid textual description of the final RL fine-tuning process.

Please read the section titled 'Fine-tuning with RL'. Focus on how it defines the policy, action space, observation space, and the reward function as a combination of the preference model's score and a KL divergence penalty.

To see how these abstract steps translate into a practical workflow, let's conclude with a concrete example that walks through the entire post-training process from start to finish.

LLM Training & Reinforcement Learning from Google Engineer | SFT + RLHF | PPO vs GRPO vs DPO

This final clip from 'LLM Training & Reinforcement Learning' provides an end-to-end example, integrating SFT, reward model training, and PPO-based policy optimization.

Watch from 19:47 to 22:02. This segment will help you consolidate your understanding by showing the full sequence of operations on a specific example.

By completing this three-step process, the model learns not just to generate plausible text, but to generate text that is helpful, harmless, and aligned with nuanced human preferences—the very qualities that make modern chatbots so effective.

Conclusion

In this lesson, we have assembled the full picture of the standard Reinforcement Learning from Human Feedback (RLHF) pipeline. You now understand how the SFT and Reward Model components you learned about previously are integrated into a powerful optimization loop.

Key Takeaways:

  • The RLHF pipeline consists of three main stages:
    1. Supervised Fine-Tuning (SFT): To teach the model instruction-following and desired response formats.
    2. Reward Model (RM) Training: To learn a scoring function that reflects human preferences, based on pairwise comparisons.
    3. Policy Optimization: To fine-tune the SFT model (the policy) using reinforcement learning (PPO) to maximize the score from the RM.
  • The final PPO optimization stage involves four models:
    • The Policy (Actor), which is being actively trained.
    • The frozen Reward Model, which provides the reward signal.
    • The frozen Reference Model, which provides a baseline to prevent the policy from diverging too far (via a KL penalty).
    • The Value Model (Critic), which estimates future rewards to stabilize training.

Preview of the Next Lesson:

We've now outlined the "what" and "why" of the RLHF pipeline. In our next lesson, we will zoom in on the heart of the PPO algorithm itself. We will dissect the PPO clipped surrogate objective, the mathematical formula that makes this process work, and understand why the "clipping" mechanism is so critical for achieving stable and effective policy updates.