PPO-Based RLHF: The Four Essential Models

The Four-Model Ensemble

The standard PPO algorithm is an "actor-critic" method, meaning it involves a policy (the actor) and a value function (the critic) that learn in tandem. However, when adapted for RLHF to fine-tune large language models, the setup expands into a four-part ensemble. Two of these models are actively learning, while the other two are frozen to provide stable targets.

Let's begin with a clear, high-level introduction to each model's role and status.

As the video highlights, the four models are:

1. The Actor (The Student): The policy we are actively training.
2. The Critic (The Forecaster): A value function that is also learning.
3. The Reference Model (The Anchor): A frozen copy of the initial model.
4. The Reward Model (The Judge): A frozen model that scores outputs.

A Deeper Look at Each Model's Role

Now that you have a high-level overview, let's solidify our understanding with more formal definitions. The following reading clearly breaks down how standard RL concepts are adapted for the specific task of fine-tuning LLMs.

Let's synthesize this information to ensure the role of each model is perfectly clear.

1. The Policy Model (Actor)

• What it is: This is the language model we are actively fine-tuning (denoted ${\pi }_{\theta }$). It is the "student" in the process.
• Initialization: It begins as a copy of the Supervised Fine-Tuned (SFT) model.
• Role: Its job is to generate text responses to given prompts. Its parameters ($\theta$) are updated by the PPO algorithm to produce outputs that maximize the score from the reward model, while staying close to the reference model.

2. The Reference Model

• What it is: A frozen, unchanging copy of the initial SFT model (denoted ${\pi }_{\text{SFT}}$ or ${\pi }_{\text{ref}}$). It is the "anchor."
• Initialization: It is also a copy of the SFT model, but its weights are not updated during PPO training.
• Role: Its sole purpose is to serve as a stable, trusted distribution for calculating the KL divergence penalty. As we saw in the last lesson, this penalty prevents the Actor from "reward hacking" and forgetting its core language capabilities.

3. The Reward Model

• What it is: The model trained on human preference data (denoted $r_{\varphi }$). It is the "judge."
• Initialization: It is trained separately before the PPO phase begins, by showing it pairs of model responses and having it learn to predict which one a human would prefer. It is also frozen during PPO.
• Role: It takes a full prompt-response pair and outputs a single scalar score representing "goodness" or "helpfulness." This score is the primary signal that the Actor is trying to maximize.

4. The Value Model (Critic)

• What it is: A model that estimates the expected future reward (denoted $V_{\psi }$). It is the "forecaster."
• Initialization: It is often initialized from the SFT model's weights, but with a different output head that produces a scalar value instead of a token distribution. Like the Actor, it is an active learner.
• Role: Its job is subtle but critical. It takes a prompt (a state) and predicts the total reward the Actor is expected to get from that point onward. This prediction serves as a baseline. The PPO algorithm then uses the difference between the actual reward (from the Reward Model) and this expected reward to create a better, lower-variance training signal called the "advantage." This helps stabilize the training of the Actor.

Visualizing the Full Pipeline

Knowing the individual roles is one thing; seeing how they interact is another. The following diagrams and video will help you visualize the flow of information between these four models during the PPO training loop.

First, let's place the PPO step in the context of the entire RLHF process.

Now, let's zoom in on that PPO loop and trace the data flow.

Finally, let's watch a dynamic walkthrough of this process.

Conclusion

In this lesson, we have systematically identified and defined the four essential models that constitute the PPO-based RLHF pipeline. This complex architecture, while effective, underscores the significant computational and memory resources required for modern LLM alignment.

Key Takeaways:

• PPO-based RLHF is not a single-model process but a four-model ensemble.
• Actor (Policy Model): The learning model that generates text. Its parameters are what we are trying to optimize.
• Critic (Value Model): A learning model that predicts expected rewards to stabilize the Actor's training.
• Reward Model: A frozen model that scores the Actor's completed outputs based on learned human preferences.
• Reference Model: A frozen copy of the initial SFT model used to compute a KL penalty, anchoring the Actor to a known-good distribution.

Preview of the Next Lesson:

The complexity of managing this four-model system is a primary motivation for finding simpler alternatives. In our next lesson, we will explore Direct Preference Optimization (DPO), an elegant method that achieves a similar goal by baking the preference-learning and KL-constrained optimization into a single loss function, thereby eliminating the need for an explicit reward model and the entire actor-critic training loop.