Understanding the RLHF Pipeline

Welcome to the third lesson in our course on GRPO. In our previous sessions, we deconstructed the core mechanics of Proximal Policy Optimization (PPO), focusing on the clipped surrogate objective for stable policy updates and the crucial role of the Generalized Advantage Estimator (GAE) in reducing gradient variance.

Now, we'll zoom out from the specifics of the PPO algorithm to understand the broader context in which it operates when fine-tuning large language models (LLMs). This lesson will outline the standard Reinforcement Learning from Human Feedback (RLHF) pipeline. Understanding this multi-stage process is essential, as it provides the full picture of how an LLM is trained to be helpful and aligned, and it perfectly situates the RL fine-tuning stage where algorithms like PPO—and its alternative, GRPO—are applied.

The Full Picture: From Pre-training to RLHF

Modern instruction-following LLMs like ChatGPT are not created in a single step. Their development follows a sophisticated, multi-stage pipeline designed to first imbue them with broad world knowledge and then align their behavior with human preferences. This process is generally broken down into three or four key phases.

To get a high-level overview of this entire process, let's start with a segment from Andrej Karpathy's "State of GPT" talk.

As outlined, the journey from a raw language model to a polished assistant involves several distinct training phases. Let's examine each one.

Stage 1: Pre-training a Base Model

The foundation of any modern LLM is the pre-training phase.

• Goal: To create a "base model" that has a comprehensive statistical understanding of language and broad world knowledge.
• Data: An enormous, diverse corpus of text scraped from the internet (e.g., Common Crawl, Wikipedia, books). This can be on the order of trillions of tokens.
• Task: The model is trained on a simple self-supervised objective, typically next-token prediction. Given a sequence of text, it learns to predict the most likely next word.

The result is a powerful "completion engine." While it's not yet a helpful assistant, this pre-training phase is what gives the model its raw linguistic and reasoning capabilities.

For a more detailed explanation of this phase, including the mathematical formulation of the language modeling task, please read the following section from Chip Huyen's excellent blog post on RLHF.

Stage 2: Supervised Fine-Tuning (SFT)

The pre-trained base model knows about language, but it doesn't know how to converse or follow instructions in a helpful way. The SFT stage aims to fix this.

• Goal: To adapt the base model to follow instructions and respond in a dialogue-oriented format. This is sometimes called "behavior cloning."
• Data: A much smaller, but very high-quality dataset of prompt-response pairs. These are curated by human labelers who are given instructions to write ideal responses to a variety of prompts.
• Task: The model is fine-tuned on this dataset using the same language modeling objective as in pre-training. However, the loss is typically calculated only on the tokens of the desired response.

This process yields an "SFT model" which is already a capable instruction-following assistant. In fact, many open-source models are only SFT-tuned.

The following video segment explains this process with a concrete example.

Stage 3: Training a Reward Model (RM)

While the SFT model is good, it's limited by the quality and diversity of the demonstration data. It's often easier for humans to judge which of two responses is better than to write a perfect response from scratch. The next stage leverages this insight to create a more nuanced signal of human preference.

• Goal: To train a separate model, the Reward Model (RM), that can take a prompt and a response and output a scalar score representing how much a human would prefer that response.
• Data: To train the RM, we collect comparison data.
  1. A prompt is selected.
  2. The SFT model generates several different responses ($y_1,y_2,...,y_k$).
  3. A human labeler ranks these responses from best to worst (e.g., $y_2\succ y_4\succ y_1\succ y_3$).
  4. This ranking is decomposed into pairs of (winning response, losing response) for a given prompt.
• Task: The RM (often initialized from the SFT model) is trained to predict these human preferences. The objective is to maximize the score difference between preferred and non-preferred responses. A common loss function, which should be familiar from your statistics background (it's related to the Bradley-Terry model for pairwise comparisons), is: $$\mathcal{L}(\theta )=-{\mathbb{E}}_{(x,y_w,y_l)\sim D}\left[\log (\sigma (r_{\theta }(x,y_w)-r_{\theta }(x,y_l)))\right]$$ where $r_{\theta }$ is the reward model, $x$ is the prompt, $y_w$ is the winning response, and $y_l$ is the losing response. This loss pushes the RM to assign a higher score to $y_w$ than to $y_l$.

Let's watch this process being explained.

Stage 4: RL Fine-Tuning with PPO

We now have an SFT model that knows how to follow instructions and a Reward Model that can score any response for its quality. The final stage combines these to further refine the SFT model using reinforcement learning. This is where PPO, and later GRPO, come into play.

• Goal: To optimize the LLM policy to generate responses that maximize the score from the reward model.

• Setup (The RL Problem):

  • Policy (${\pi }_{\theta }$): The LLM being fine-tuned, initialized with the SFT model's weights.
  • Action Space: The model's entire vocabulary of tokens. An "action" is sampling the next token.
  • Observation Space: The sequence of tokens in the prompt.
  • Reward Function: The reward model, $r_{\theta }(x,y)$.

• Process: The training loop proceeds as follows:

  1. A prompt $x$ is sampled from a dataset.
  2. The policy ${\pi }_{\theta }$ generates a response $y$.
  3. The reward model calculates a scalar reward $r=r_{\theta }(x,y)$.
  4. This reward is used to update the policy ${\pi }_{\theta }$ using an RL algorithm like PPO. The update encourages the policy to generate responses that receive high rewards.

This is a direct application of the PPO algorithm we've studied. The "advantage" is calculated based on the scalar reward from the RM, and the policy is updated using the clipped surrogate objective.

To see this final, crucial step in action, please watch the next video segment and study the accompanying diagram.

The RLHF pipeline effectively "unlocks" the capabilities of the base model and aligns them with complex, nuanced human values that are difficult to specify in a simple loss function.

For a final, comprehensive walkthrough that consolidates all these steps with mathematical details, please read the following sections from Chip Huyen's article.

Conclusion

In this lesson, we mapped out the entire RLHF pipeline, placing the reinforcement learning stage within its proper context.

Here are the key takeaways:

• Multi-Stage Process: Aligning LLMs is a multi-stage process involving pre-training, supervised fine-tuning (SFT), reward modeling (RM), and RL fine-tuning.
• SFT Sets the Stage: Supervised fine-tuning on human demonstrations teaches the model to be a helpful, instruction-following assistant.
• The Reward Model as a Judge: A reward model, trained on human preference rankings, learns to act as a proxy for human judgment, providing a scalable reward signal.
• RL as the Final Polish: The RL fine-tuning stage uses the reward model's score as a reward signal to optimize the SFT model's policy, typically with PPO. This steers the model towards generating outputs that are more aligned with complex human preferences.

You now understand the standard framework for which GRPO is designed to be a more efficient alternative. The RL fine-tuning stage, with its reliance on a separate critic (the value function in PPO) and a reward model, is precisely what GRPO aims to simplify.

In our next lesson, we will take a closer look at a crucial detail of the RL fine-tuning objective: the KL-divergence penalty. We'll explore why it's necessary for regularizing the policy and preventing it from deviating too far from the original SFT model, a principle that is fundamental to both PPO and GRPO.