Training Reward Models with Pairwise Preferences

Welcome back! In our previous lesson, we outlined the complete three-stage architecture of Reinforcement Learning from Human Feedback (RLHF), giving you a "map" of the entire alignment process. We saw how a base model is first fine-tuned on demonstration data (SFT), then used to train a Reward Model (RM), and finally optimized using the RM's feedback in an RL loop.

Today, we will zoom in on the most crucial and conceptually novel part of this pipeline: Stage 2, Reward Model Training. Our goal is to understand precisely how we can transform subjective, nuanced human preferences—statements like "I like this answer better than that one"—into a concrete, scalable, and differentiable reward signal, $r (x, y)$ , that a machine can learn from. You'll see how a classic statistical model from the 1950s provides the elegant mathematical foundation for this very modern problem.

By the end of this lesson, you will be able to explain how a reward model is trained from pairwise human preference data to learn a scalar reward signal, demystifying the "magic" that allows an LLM to learn what humans find helpful and appropriate.

From Absolute Scores to Relative Preferences

The first challenge in teaching a model about human preferences is framing the problem correctly. If we want a model to output a scalar "goodness" score, we need training data. But how do we get it? Asking human labelers to assign a numerical score (e.g., 7.3 out of 10) to a model's response is notoriously difficult and inconsistent. What one person considers a "7," another might call a "9."

A much more robust and consistent approach is to rely on relative judgments. Humans are far better at comparing two options and picking the better one. The RLHF process leverages this by collecting preference rankings.

The data collection for the reward model works as follows:

Sample a prompt: A prompt, $x$ , is taken from a predefined dataset.
Generate multiple responses: The SFT model (from Stage 1) is used to generate several different completions for that prompt, say A, B, C, and D.
Rank the responses: A human labeler is shown the prompt and the four responses and asked to rank them from best to worst. For example, they might decide the ranking is D > A > B > C.
Create pairwise data: This single ranking is then decomposed into multiple pairwise comparison data points. Each data point consists of the prompt, a "chosen" (winner) response $y_{c}$ , and a "rejected" (loser) response $y_{r}$ . The ranking D > A > B > C would yield pairs like (D, A), (D, B), (D, C), (A, B), (A, C), and (B, C).

This process results in a large dataset of (prompt, chosen_response, rejected_response) tuples.

This diagram shows the core workflow for training the reward model (highlighted as Step 2). A prompt leads to multiple outputs, which are ranked by a human. This preference data is then used to train the Reward Model (RM).

RLHF in 90 min

The video "RLHF in 90 min" provides a clear, practical walkthrough of this data collection pipeline and motivates the need for a reward model.

First, watch the segment on building the preference dataset. This will show you exactly how rankings are broken down into pairwise comparisons. Then, watch the next part explaining why we need a reward model as a scalable proxy for direct human feedback.

The Bradley-Terry Model: A Probabilistic View of Preference

Now we have a dataset of preferences, (x, y_c, y_r). The goal is to train a model, our Reward Model $r_{θ} (x, y)$ , that learns to assign a scalar score to any given prompt-completion pair, such that the score for a chosen completion is higher than for a rejected one: $r_{θ} (x, y_{c}) > r_{θ} (x, y_{r})$ .

How do we formalize this? We turn to the Bradley-Terry model, a statistical framework developed in 1952 for converting pairwise comparisons into rankings. It assumes that each item being compared has a latent, unobserved "strength" or "quality" score. The probability that item $i$ is preferred over item $j$ is then a function of their respective strengths.

Bradley-Terry Model: Converting Preferences to Rankings ...

This article provides an excellent introduction to the Bradley-Terry model and its mathematical formulation, which is the cornerstone of reward modeling.

Please read the sections from the introduction up to the end of the section on the log-linear form. Focus on understanding how the model connects the probability of a win to the latent "strength" parameters and why the log-linear formulation using the sigmoid function is so convenient.

As the reading explains, the Bradley-Terry model states that the probability of preferring item $i$ over item $j$ can be modeled as:

P (i > j) = \frac{p _{i}}{p _{i} + p _{j}}

where $p_{i}$ and $p_{j}$ are the positive "strength" scores of the items. By reparameterizing with $p_{i} = e^{r_{i}}$ , where $r_{i}$ is an unbounded score, this equation simplifies beautifully to:

P (i > j) = \frac{e ^{r_{i}}}{e ^{r_{i}} + e ^{r_{j}}} = \frac{1}{1 + e ^{- (r_{i} - r_{j})}} = σ (r_{i} - r_{j})

Here, $σ$ is the sigmoid function. This elegant result is the key to training our reward model. We can model the probability that a human prefers $y_{c}$ over $y_{r}$ as the sigmoid of the difference in the scores assigned by our reward model:

P (y_{c} > y_{r} ∣ x) = σ (r_{θ} (x, y_{c}) - r_{θ} (x, y_{r}))

This equation has a very intuitive behavior:

If $r_{θ} (x, y_{c})$ is much larger than $r_{θ} (x, y_{r})$ , the difference is a large positive number, and $σ (large positive) \approx 1$ . The model is confident in this preference.
If the scores are very similar, the difference is close to zero, and $σ (0) = 0.5$ . The model is uncertain, assigning a 50/50 probability.
If the model incorrectly assigns $r_{θ} (x, y_{c}) ≪ r_{θ} (x, y_{r})$ , the difference is a large negative number, and $σ (large negative) \approx 0$ . The model is confident that its prediction is wrong, which will lead to a large training loss.

The Reward Model Loss Function

With our probabilistic model in place, training becomes a standard machine learning task: we want to find the parameters $θ$ of our reward model that maximize the probability of the human preference data we've collected. In deep learning, maximizing a probability is equivalent to maximizing its logarithm (log-likelihood), which in turn is equivalent to minimizing the negative log-likelihood.

This leads us directly to the loss function for the reward model. For a single preference pair $(x, y_{c}, y_{r})$ , the loss is:

L (θ) = - lo g (σ (r_{θ} (x, y_{c}) - r_{θ} (x, y_{r})))

The total loss is the average of this value over the entire preference dataset. This is a form of binary cross-entropy loss, where the goal is to correctly classify which response is the "winner." By minimizing this loss, the model is forced to learn a scoring function $r_{θ}$ that assigns higher scores to completions that humans prefer.

Reward Models | RLHF Book by Nathan Lambert

The RLHF Book by Nathan Lambert provides a formal derivation of this process. It connects the Bradley-Terry model directly to the loss function used in seminal papers like InstructGPT.

Please read the section titled Training Reward Models. Pay close attention to the mathematical steps in equation (4), which show how maximizing the Bradley-Terry probability leads directly to the negative log-sigmoid loss function. Note the equivalence to the alternative log(1 + exp(...)) form, which you may also encounter.

Reward Model Architecture and Training

What does the reward model itself look like? Given your background, it's helpful to think of it as a form of "brain surgery" on the SFT model.

Initialization: We start with the SFT model. We do this to retain all the rich linguistic knowledge the model has already learned. A model that can judge text quality must first understand the text deeply.
Architecture Modification: The SFT model's final layer is a "language model head," a linear layer that projects the hidden states to logits over the entire vocabulary (its job is to predict the next word). For the reward model, we don't need to predict words; we need a single score. So, this head is removed and replaced with a new, much smaller linear head that projects the final hidden state to a single scalar value.
Forward Pass: During training, two forward passes are done for each data point: one for the prompt and chosen completion (x, y_c) and another for the prompt and rejected completion (x, y_r). The model processes the full sequence and uses the hidden state of the final token to produce the scalar scores $r_{θ} (x, y_{c})$ and $r_{θ} (x, y_{r})$ . These two scores are then plugged into the loss function.

This diagram illustrates the two ways a preference can be modeled. On top is the Bradley-Terry approach we've discussed: two separate passes through a reward model produce two scores, $r^1_\theta$ and $r^2_\theta$, and the preference probability is a function of their difference. The bottom approach, used in methods like DPO, models the preference directly.

The following video provides an exceptionally clear visualization and code-level explanation of this architectural change and the training loop.

RLHF in 90 min

The "RLHF in 90 min" video connects the theory directly to implementation, which will resonate with your CS background.

First, watch the segment on the reward model loss function for a concise summary. Then, watch the excellent "brain surgery" analogy explaining the architectural changes from an SFT model to a reward model, complete with code snippets. Finally, see how the math is translated directly into PyTorch code.

Conclusion

In this lesson, we have dissected the critical process of training a reward model. We've moved from the abstract idea of "learning from human feedback" to a concrete, mathematical, and architectural understanding of how it's done.

Here are the key takeaways:

Pairwise Data: Reward modeling sidesteps the difficulty of absolute scoring by using more reliable pairwise comparison data, collected by having humans rank multiple model-generated responses.
Bradley-Terry Model: This statistical model provides the theoretical link between latent quality scores and the probability of one item being preferred over another. The relationship is captured by applying the sigmoid function to the difference of the scores: $P (y_{c} > y_{r}) = σ (r_{c} - r_{r})$ .
Negative Log-Likelihood Loss: The training objective is to maximize the probability of the observed human preferences, which translates to minimizing a negative log-likelihood loss function based on the Bradley-Terry formula.
Architecture: The reward model is typically an SFT model with its language modeling head replaced by a scalar head that outputs a single "goodness" score for a given prompt-completion pair.

You now understand how Stage 2 of the RLHF pipeline produces a crucial artifact: a trained reward model that acts as a differentiable and scalable proxy for human judgment.

In our next lesson, we will move to Stage 3. We will explore how this newly trained reward model is used as the core of a reinforcement learning objective to fine-tune our policy (the SFT model) to produce outputs that are not just coherent, but also aligned with what humans actually want.

Recommended resources

From Absolute Scores to Relative Preferences

The Bradley-Terry Model: A Probabilistic View of Preference

The Reward Model Loss Function

Reward Model Architecture and Training

Conclusion