Training Reward Models from Human Preferences

1. Capturing Human Preferences

The core idea behind reward modeling is that while it's difficult for humans to assign a consistent, absolute "quality score" to a response, it's much easier and faster for them to compare two responses and choose the better one. This process forms the basis of the preference dataset.

Here’s how the data is collected:

1. A prompt is given to the SFT-tuned model from our last lesson.
2. The model generates multiple different responses (completions) to this prompt.
3. These responses are presented to human labelers in pairs.
4. For each pair, the labeler chooses the one they prefer. The one they pick is called the chosen response ($y_c$), and the other is the rejected response ($y_r$).

This video from StatQuest provides a clear, animated explanation of this data collection process.

This dataset of (prompt, chosen_response, rejected_response) tuples is the fuel we'll use to train our reward model. The model's task is to learn a function that automatically replicates these human judgments.

2. Reward Model Architecture

Now that we have our preference data, what does the model we're training actually look like?

Conveniently, we don't need to start from scratch. The standard practice is to take the language model we already fine-tuned with SFT and adapt it. Since this model already understands language and the instruction-following format, we just need to change its "job."

The process is as follows:

1. Start with the SFT model.
2. Replace the final output layer. The original model had a large output layer that produced logits over the entire vocabulary (for next-token prediction). We remove this.
3. Add a new regression "value" head. This is typically a single linear layer that takes the final hidden state of the model and maps it to a single scalar number. This number is our reward.

The input to this model is a full sequence, like [prompt] [response] [EOS]. The model processes this entire sequence, and the hidden state corresponding to the final token (e.g., [EOS]) is fed into the value head to produce the reward score. This final hidden state acts as a summary representation of the entire prompt-response pair.

The following resource provides a more formal description of this architecture.

By making this small architectural change, we transform a generative model into an evaluative one. It no longer asks, "What token comes next?" but rather, "How good is this entire sequence?"

3. Training the Reward Model: From Preferences to Loss

This is the core of our lesson. How do we use (chosen, rejected) pairs to train a model to output a scalar reward? The process is grounded in a statistical method called the Bradley-Terry model, which relates pairwise comparisons to underlying "strength" scores.

The Bradley-Terry Model

The Bradley-Terry model proposes that the probability of preferring one item ($y_c$) over another ($y_r$) is a function of their latent scores, which we'll call rewards ($r$). Specifically, this probability can be expressed using the sigmoid function, $\sigma$:

Where:

• $r(x,y)$ is the scalar reward for a response $y$ to a prompt $x$.
• The model's preference depends only on the difference in rewards.
• The sigmoid function, $\sigma (z)=\frac{1}{1+e^{-z}}$, maps this difference to a probability between 0 and 1.

If the reward for the chosen response is much higher than for the rejected one, the difference is large and positive, and the probability approaches 1. If the rewards are equal, the probability is 0.5.

The Pairwise Preference Loss

Our goal is to train the reward model's parameters $\theta$ to produce rewards that maximize this probability for all the preference pairs in our dataset. In machine learning, maximizing this log-probability is equivalent to minimizing the negative log-likelihood loss. This gives us our loss function:

This loss function effectively pushes the model to increase the reward for the chosen response ($r(y_c)$) and decrease the reward for the rejected response ($r(y_r)$), thereby widening the gap between them.

The following video and text provide both an intuitive walkthrough and a formal derivation of this process.

Now, for a more rigorous look at the mathematics, we'll turn to the RLHF Book. This will connect the intuitive explanation you just saw to the formal derivation from the Bradley-Terry model.

By training a model with this loss function, we create a powerful proxy for human judgment. The model learns to internalize the patterns of preference from the data and can then assign a plausible "goodness" score to any new response it sees.

4. The Result: A Judge for the Machine

Once training is complete, we have a reward model that can take any (prompt, response) pair and output a scalar reward. This is the crucial signal we were missing. We've moved beyond simple imitation.

The final segment of the StatQuest video demonstrates how this trained reward model is then used to evaluate new responses, providing the feedback needed for the final stage of RLHF.

The reward model's output provides a dense, continuous signal that we can optimize against. Instead of just "right" or "wrong" (like in SFT), we now have a spectrum of "how good" a response is. This opens the door to using powerful reinforcement learning algorithms to further refine our policy.

Conclusion

In this lesson, we have constructed the second crucial component of the RLHF pipeline: the reward model. This model serves as our automated human preference judge.

Key Takeaways:

• Reward models are trained because it's easier for humans to compare two responses than to assign an absolute score to one.
• The training data consists of tuples: (prompt, chosen_response, rejected_response).
• The reward model is typically an SFT-tuned language model with its final layer replaced by a linear head that outputs a single scalar value (the reward).
• The training objective is derived from the Bradley-Terry model and uses a pairwise preference loss. This loss function trains the model to maximize the reward difference between chosen and rejected responses.
• The final trained reward model acts as a proxy for human judgment, providing the essential reward signal for the final stage of RLHF.

Preview of the Next Lesson:
We now have our SFT-tuned policy (the "student") and our reward model (the "judge"). In the next lesson, "Outline the standard RLHF pipeline: SFT, reward model training, and policy optimization," we will finally bring everything together. We'll introduce the Proximal Policy Optimization (PPO) algorithm and see how it uses the reward signal from our new model to iteratively improve the language model's ability to generate responses that are not just correct, but truly aligned with human preferences.