GRPO vs. PPO: The Motivation for Critic-Free Learning

Welcome to our second module, where we will transition from the foundational concepts of policy optimization to the specific theory behind Group Relative Policy Optimization (GRPO).

In the previous module, we established a complete picture of the standard RLHF pipeline, culminating in the use of Proximal Policy Optimization (PPO) to fine-tune a language model. A central feature of PPO is its actor-critic architecture, where the 'actor' (the LLM policy) is guided by a 'critic' (a learned value function).

Today's lesson addresses a fundamental question: Why did GRPO move away from this established actor-critic paradigm? We will explore the motivation for GRPO's critic-free design by directly contrasting it with PPO, setting the stage for understanding its unique mechanics.

The PPO Approach: An Actor-Critic Framework

Let's begin by formalizing the actor-critic structure within PPO. As we've discussed, policy gradient methods can suffer from high variance in their gradient estimates, leading to unstable training. The critic is PPO's primary tool for variance reduction.

• The Actor (${\pi }_{\theta }$): This is the policy itself—the language model with parameters $\theta$ that we are training to generate better responses.
• The Critic ($V_{\varphi }$): This is a separate model, with parameters $\varphi$, trained to estimate the value function, $V_{\varphi }(s)$. The value function predicts the expected future reward from a given state $s$. In the context of LLMs, the "state" is the prompt and the sequence of tokens generated so far.
• The Advantage Function ($A(s,a)$): The critic's value estimates are used to compute the advantage, which measures how much better a specific action $a$ is compared to the average action from state $s$. A common estimator for this is $A(s,a)\approx r+\gamma V_{\varphi }(s^{\prime })-V_{\varphi }(s)$, where $r$ is the immediate reward and $s^{\prime }$ is the next state. The value function $V_{\varphi }(s)$ acts as a baseline, centering the advantages and reducing the variance of the policy updates.

The key takeaway is that in PPO, we are training two neural networks in tandem: the policy (actor) and the value function (critic). This brings us to the core motivations for seeking an alternative.

The Problem with Critics for LLM Fine-Tuning

While elegant in theory, the actor-critic approach introduces significant practical challenges, especially in the resource-intensive domain of large language models. The following article outlines the complexity of PPO and the core arguments for moving towards a simpler, critic-free approach.

As the article highlights, the use of a critic introduces three main issues:

1. Resource Overhead: The critic is typically another large model, often a copy of the policy LLM with a regression head. Training it alongside the policy effectively doubles the number of trainable parameters, leading to a massive increase in GPU memory consumption (for weights, gradients, and optimizer states) and computational cost.

2. Training Complexity: The actor-critic framework is a delicate balancing act. You are now optimizing a joint loss function with more hyperparameters, and the instability of one component can negatively affect the other. This adds what the video below calls "cognitive load" for the practitioner.

3. Conceptual Mismatch with Sparse Rewards: Many important LLM alignment tasks, such as mathematical reasoning, provide a single, sparse reward at the very end of a generation (e.g., a binary reward for a correct or incorrect final answer). The critic, however, is tasked with estimating a per-token value. Learning meaningful, dense, per-token value estimates from a single, sparse, sequence-level reward is a notoriously difficult credit assignment problem. The resulting value estimates may be noisy or inaccurate, undermining their purpose for variance reduction.

This video segment provides a concise visual explanation of these drawbacks.

GRPO’s Solution: A Critic-Free Architecture

Given the challenges of the actor-critic model, GRPO proposes a simple yet effective alternative: eliminate the critic entirely.

Instead of learning a parametric value function to act as a baseline, GRPO computes a non-parametric, empirical baseline directly from sampled data. The following image provides a high-level visual contrast between the two architectures.

The core mechanism is as follows:

1. For a single prompt, generate a "group" of multiple responses (e.g., $G=8$ responses).
2. Calculate the scalar reward for each of the $G$ responses.
3. The baseline for any single response is simply the mean reward of the group.
4. The advantage for a response is its reward relative to this group-based baseline, typically normalized by the group's standard deviation.

This "group-relative" advantage estimation replaces the complex, learned critic with a simple, statistical calculation performed on a batch of data.

The following reading from a comparative analysis paper formalizes this distinction. Given your background, you will appreciate its precision.

This critic-free design is GRPO's main selling point. It directly addresses the shortcomings of the PPO actor-critic framework in the LLM context by simplifying the training pipeline, reducing resource requirements, and adopting a baseline that is arguably more natural for sequence-level reward signals.

To conclude, let's watch one more short clip that summarizes GRPO's key contribution.

Conclusion

In this lesson, we have explored the fundamental motivation behind GRPO's design. By moving away from PPO's actor-critic framework, GRPO offers a more streamlined and resource-efficient approach to RL fine-tuning for LLMs.

Here are the key takeaways:

• PPO's Architecture: PPO is an actor-critic method. It trains a separate value function model (the critic) to provide a baseline for advantage estimation, which helps stabilize training by reducing variance.
• Drawbacks of the Critic: In the context of LLMs, the critic introduces significant resource overhead (memory and compute), increases training complexity, and is conceptually awkward to train with the sparse, sequence-level rewards common in alignment tasks.
• GRPO's Critic-Free Solution: GRPO eliminates the critic. It replaces the learned value function with a simple, empirical baseline: the average reward of a group of responses sampled for the same prompt.
• Motivation Summary: GRPO's critic-free approach is motivated by a desire for a simpler, more efficient, and more practical algorithm tailored to the realities of fine-tuning large language models.

Now that you understand why GRPO is designed the way it is, our next lesson will focus on the how. We will take a closer look at the group-relative advantage and formally derive the estimator that lies at the heart of the GRPO algorithm.