Understanding the RLHF Pipeline

Introduction

In our last lesson, we focused on Supervised Fine-Tuning (SFT), the process of adapting a pre-trained base model into a helpful, instruction-following assistant using a curated dataset of (prompt, completion) pairs. We established that SFT is the foundational step in aligning an LLM with user intent. However, SFT has a significant practical limitation: it relies on human labelers writing the "perfect" response for every prompt. This is not only labor-intensive but often impractical. It can be incredibly difficult to write a flawless answer, but it's much easier to look at two different answers and decide which one is better.

This observation is the cornerstone of our topic today: Reinforcement Learning from Human Feedback (RLHF). RLHF is a more advanced alignment technique that refines a model using human preferences rather than human-written demonstrations. In this lesson, we will dissect the stages of the RLHF process, focusing on how a preference-based reward model is trained and how that model is then used to optimize the LLM's policy using reinforcement learning.

Why RLHF? The Asymmetry of Judging vs. Generating

The core motivation for RLHF lies in a fundamental asymmetry in human capability: judging is often easier than creating. A person who isn't a great poet might still be able to recognize a beautiful poem when they see one. Andrej Karpathy eloquently explains this concept.

By shifting the human task from generating ideal responses to ranking existing ones, RLHF taps into a more scalable and reliable source of feedback. This allows us to train models to produce outputs that are better than what the human labelers themselves might have been able to write, effectively bootstrapping the model's capabilities.

The RLHF Pipeline: A Three-Step Process

RLHF is not a single training run but a multi-stage pipeline that builds upon the SFT model we discussed in the previous lesson. The entire process can be broken down into three main steps, which are well-illustrated in the following diagram.

Let's briefly outline these steps:

1. Train a Supervised Fine-Tuned (SFT) Model: This is our starting point. We take a pre-trained base LLM and fine-tune it on a high-quality dataset of prompt-completion pairs, as covered in our previous lesson. This initial model is already a capable assistant.

2. Train a Reward Model (RM): We use the SFT model to generate several different responses to a set of prompts. Human labelers then rank these responses from best to worst. This collection of comparison data is used to train a new model—the reward model—whose job is to predict a scalar "reward" score that reflects human preference.

3. Optimize the Policy with Reinforcement Learning (RL): The SFT model (now referred to as the "policy") is further fine-tuned. In this stage, we use the frozen reward model as an automated human preference simulator. The policy generates a response, the reward model scores it, and that score is used as a reward signal to update the policy's parameters using an RL algorithm, typically Proximal Policy Optimization (PPO).

The Hugging Face blog post, "Illustrating Reinforcement Learning from Human Feedback," provides a good written overview of this entire pipeline.

Now, let's dive deeper into steps 2 and 3.

Step 2: Training a Reward Model

The goal of this stage is to create an automated judge that can accurately score how much a human would prefer a given model response. This transforms the subjective, qualitative feedback from humans into a quantitative, scalar reward signal that an RL algorithm can optimize.

The process for gathering data and training the RM is as follows:

1. Sample Prompts: A batch of prompts is selected from a diverse dataset.
2. Generate Responses: The SFT model generates multiple, distinct responses for each prompt (e.g., by using a non-zero temperature).
3. Human Ranking: Human labelers are presented with the prompt and the generated responses. They rank the responses from best to worst. For example, given four responses (A, B, C, D), a labeler might provide a ranking like D > C > A = B.
4. Create Comparison Dataset: This ranking is broken down into a set of pairwise comparisons. From the ranking D > C > A, we get pairs like (D is better than C), (D is better than A), and (C is better than A).
5. Train the Reward Model: The reward model itself is typically initialized from the SFT model, but with its final layer replaced by a single output neuron (a regression head). It is then trained on the comparison dataset. For each pair, the model predicts a score for the "winning" response and the "losing" response. The training objective is to maximize the margin between these scores, ensuring preferred outputs receive a higher reward.

This training process results in a powerful tool: a reward model that can instantly score any (prompt, completion) pair, effectively serving as a proxy for human judgment at a massive scale.

Step 3: Policy Optimization with Reinforcement Learning

With a frozen reward model in hand, we can begin the final and most complex stage: fine-tuning our SFT model using reinforcement learning. Given your computer science background, framing this in formal RL terms will be helpful.

• Policy ($\pi$): The LLM we are training. It takes a state (the prompt and current generated text) and produces a probability distribution over the next action (the next token).
• Action Space: The entire vocabulary of the model (e.g., ~50,000 tokens). An "action" is the selection of the next token to generate.
• Observation/State Space: The sequence of tokens generated so far for a given prompt.
• Reward Function: This is where the magic happens. The reward is determined by our reward model. However, there's a critical addition.

If we only optimized for the reward model's score, the policy might discover strange, nonsensical strings of text that "trick" the reward model into giving a high score. This is a common failure mode in RL known as "reward hacking" or "policy collapse." To prevent this, we introduce a constraint: the policy should not deviate too far from the original SFT model it was initialized from.

This constraint is implemented as a penalty term using the Kullback-Leibler (KL) divergence. The KL divergence measures how different two probability distributions are. The final reward for a generated sequence y from a prompt x is calculated as:

Where:

• $R_{RM}(x,y)$ is the score from our reward model.
• $D_{KL}(...)$ is the KL divergence between the output distributions of our current policy (${\pi }_{\text{RLHF}}$) and the original SFT policy (${\pi }_{\text{SFT}}$).
• $\lambda$ is a hyperparameter that controls the strength of the KL penalty.

This combined reward signal encourages the model to generate outputs that humans would prefer, while ensuring it remains a coherent and capable language model. The algorithm used to perform this optimization is almost always Proximal Policy Optimization (PPO).

A Deeper Look at the RLHF Architecture

The PPO algorithm is an "actor-critic" method, which adds another layer of sophistication to the system. The full architecture involves not just the policy and reward models, but also a Value Model (the "critic").

Let's break down this more complex diagram:

• Policy LM (${\pi }_{\theta }$): This is the "actor" that we are actively training. It generates text.
• SFT Model (${\pi }^{\text{SFT}}$): This is a frozen copy of the original SFT model. It's used as a reference to calculate the KL divergence penalty, ensuring our policy doesn't stray too far.
• Reward Model ($r(x,y)$): This is our frozen, pre-trained judge. It provides the base reward signal.
• Value Model ($V_{\varphi }(s_t)$): This is the "critic". Its job is to predict the expected future reward from the current state (i.e., the text generated so far). In PPO, the critic's prediction is used to calculate the "Advantage" ($\hat{A}(s_t,a_t)$), which represents how much better a specific action was than the expected average. Using an advantage estimate rather than just the raw reward helps to reduce the variance of the updates and stabilize training significantly. The Value Model is trained concurrently with the Policy LM.

This intricate dance between four different models—policy, reward, SFT reference, and value—is what allows PPO to effectively and stably navigate the vast search space of language generation to find outputs that align with human preferences.

For a deeper dive into the mechanics of PPO, the following article is an excellent, technically-grounded resource.

Conclusion

In this lesson, we have moved beyond supervised fine-tuning to explore the more powerful and nuanced alignment technique of Reinforcement Learning from Human Feedback. You now have the vocabulary and conceptual framework to discuss this critical part of modern LLM training.

Here are the key takeaways:

• Core Principle: RLHF is motivated by the fact that it's easier for humans to compare outputs than to create a perfect one from scratch.
• Three-Step Pipeline: The process involves (1) starting with an SFT model, (2) training a Reward Model on human preference data, and (3) optimizing the SFT model (the policy) using PPO against the reward model.
• Reward Model (RM): The RM is trained on ranked comparisons of model outputs to produce a scalar score representing human preference.
• Policy Optimization: The policy is trained using PPO to maximize the reward from the RM. To ensure stability and prevent the model from generating gibberish, a KL divergence penalty is used to keep the policy from deviating too far from the original SFT model.
• Actor-Critic Architecture: The PPO algorithm uses a Value Model (critic) alongside the policy (actor) to estimate expected future rewards, which helps stabilize the training process.

While RLHF is incredibly effective at making models more helpful and harmless, it is not a silver bullet. The process is complex, computationally expensive, and the resulting model's behavior is heavily influenced by the biases of the human labelers. Even with RLHF, models can still "hallucinate" facts and exhibit other failure modes.

In our next and final lesson of this foundational module, we will identify and discuss the key capabilities and inherent limitations of current large language models, setting the stage for the more advanced application-focused modules to come.