Introduction to RLHF and Model Alignment

Hello! Let's dive into the next stage of our exploration of LLM training.

In the previous lesson, we saw how Supervised Fine-Tuning (SFT) takes a raw, pre-trained base model and teaches it the style of a helpful assistant using a dataset of high-quality prompt-response examples. However, we also touched on its limitations: creating these "perfect" answers is expensive, and for many prompts, a single correct answer doesn't exist. This approach doesn't scale well and struggles with nuance.

Today, we'll tackle the next, more sophisticated step in model alignment. This lesson addresses the learning outcome: Describe Reinforcement Learning from Human Feedback (RLHF) and its role in model alignment. We'll discover how models learn not from being told the right answer, but by learning what humans prefer.

1. Beyond a Single "Correct" Answer

The core problem with SFT is that "goodness" is often subjective. If you ask an LLM to "write a short, encouraging email to a team that just finished a difficult project," there are countless good answers. SFT trains the model on just one of them.

RLHF takes a different approach. It’s built on a simple psychological principle: it is often easier for people to compare two options and choose the better one than it is to create a perfect option from scratch.

This small shift in methodology has a massive impact on our ability to align models with complex human values like helpfulness, harmlessness, and honesty. Let's see a concrete example of why this alignment is so critical.

This is the goal: steering the model away from potentially harmful or unhelpful outputs towards responses that align with human values. SFT gets us part of the way there, but RLHF is what truly refines this behavior.

2. The Three Steps of RLHF

The RLHF process can be broken down into three main stages. This diagram provides a great high-level map of the entire workflow, which we will explore step-by-step.

Let's walk through these steps.

Step 1: Collect Human Preference Data

First, we take the model that has already undergone Supervised Fine-Tuning. For a given prompt, we have this model generate several different responses. Then, we present these responses to human labelers and ask them to rank them from best to worst, or simply choose the best one.

This process of collecting comparisons is much more scalable than writing perfect answers. The result is a large dataset of prompts, paired with responses that have been ranked by human preference.

Step 2: Train a Reward Model (RM)

The preference data is valuable, but we can't have humans in the loop for every single generation. We need to automate this judgment. The solution is to train another, separate model to act as a proxy for the human labelers. This is the Reward Model (RM).

The RM's job is to take a prompt and a response and output a single number—a score, or "reward"—that predicts how highly a human would have rated that response.

Training this model involves showing it the pairs of responses from our dataset (e.g., Response A was preferred over Response B) and teaching it to give a higher score to the preferred response.

Once trained, we have a Reward Model that can automatically score any response, effectively acting as our scalable, automated human preference judge.

Step 3: Fine-Tune the LLM with Reinforcement Learning

Now we have all the pieces:

1. The SFT model (our "policy"), which we want to improve.
2. The Reward Model (our "judge"), which knows what "good" looks like.

In this final step, we use a reinforcement learning algorithm—most famously Proximal Policy Optimization (PPO)—to fine-tune our SFT model. The process works like this:

1. A new prompt (one the model hasn't seen) is given to the SFT model.
2. The model generates a response.
3. The Reward Model scores that response.
4. This reward signal is used to update the weights of the SFT model. If the reward was high, the update encourages the model to produce similar responses in the future. If it was low, the update discourages it.

This loop is repeated millions of times, pushing the model to consistently generate outputs that our Reward Model—and by extension, our human labelers—would prefer.

The KL Loss component in the diagram is a crucial detail. It's a penalty term that measures how much the updated model has strayed from the original SFT model. In software terms, think of this as a constraint to prevent "over-optimization." It stops the model from finding some bizarre, gibberish output that happens to trick the reward model into giving a high score (a phenomenon known as "reward hacking"). This keeps the responses coherent and on-topic.

3. The Role and Limitations of RLHF

RLHF has been a breakthrough in making LLMs like GPT-4, LLaMA, and Claude more helpful and safer to use. Its main role is to imbue models with nuanced human preferences that are difficult to specify in a supervised dataset.

However, the process is not without its challenges. For a more detailed look at the mechanics and limitations, the following article provides a great summary.

The main limitations highlighted in the article are:

• Cost and Scalability: While more scalable than SFT, collecting high-quality human preference data is still expensive and time-consuming.
• Human Subjectivity and Bias: The final model will reflect the preferences and biases of the human labelers. If the labeler group is not diverse, the model may be poorly aligned for a global audience.
• Reward Hacking: The model might find clever ways to maximize the reward score that don't actually align with the true underlying human intent.

Due to these challenges, there is active research into alternatives, such as Reinforcement Learning from AI Feedback (RLAIF), where another capable AI provides the preference labels, potentially reducing cost and some biases.

Conclusion

In this lesson, we've unpacked RLHF, the critical process that refines an SFT model into the highly capable and aligned assistants we interact with today.

Key Takeaways:

• RLHF learns from preferences, not examples. This makes it more scalable and better suited for nuanced, subjective tasks than SFT.
• The process involves three key steps:
  1. Collecting a dataset of human preferences between different model-generated responses.
  2. Training a Reward Model (RM) to act as an automated proxy for human judgment.
  3. Using reinforcement learning (e.g., PPO) to fine-tune the LLM to maximize the score from the RM.
• RLHF is essential for aligning models to be helpful, harmless, and honest.
• The process has limitations, including the cost of data collection, the potential for bias inherited from human labelers, and the risk of "reward hacking."

Preview of Next Lesson:
So far, we've focused on the process of training and aligning models. In our next lesson, we'll zoom out and explore some of the fascinating phenomena that arise from this process, specifically when we increase the scale of the models and data. We will cover the concepts of emergent abilities and scaling laws, which help explain why bigger models aren't just better—they are often qualitatively different.