GRPO: Group-Relative Advantage Estimation

The Problem with the Critic in PPO

To understand why eliminating the critic is a significant improvement, we must first recall its role in PPO. The critic is a learned value function, typically denoted as $V(s)$, whose job is to predict the expected future reward from a given state $s$. This prediction serves as a baseline for calculating the advantage, which measures how much better a specific action was compared to the average action from that state.

While this actor-critic architecture is a cornerstone of modern RL, it introduces several challenges, especially in the context of LLMs.

1. Computational Overhead: The critic is usually another large neural network, often a copy of the policy model itself. As the Understanding GRPO article notes, PPO requires four models in memory: the policy (actor), the critic, the reference model, and the reward model. This dramatically increases memory requirements and computational cost.
2. Training Complexity: Training two large networks (actor and critic) simultaneously is complex and can be unstable. Practitioners face what one of our resources calls a "cognitive load," making the algorithm harder to understand and tune.
3. Reward Signal Mismatch: For many LLM reasoning tasks (like math or code generation), we only receive a single, final reward indicating whether the entire answer was correct. The critic, however, is trained to estimate a value for every intermediate token's state. Training an accurate per-token value function with only a final reward signal is fundamentally difficult and often inaccurate.

The following video section provides an excellent intuitive breakdown of these challenges.

These problems motivate the search for a simpler way to estimate the advantage baseline—a way that doesn't require training another massive neural network.

GRPO's Solution: Replacing the Critic with Group Sampling

The core insight of GRPO is that the critic's value function is just one way to establish a baseline for advantage calculation. GRPO replaces this learned, predictive baseline with a direct, empirical one derived from sampling.

This process is straightforward and can be broken down into a few simple steps:

1. Group Sampling: For a single prompt, instead of generating just one response, the policy generates a "group" of $G$ different responses (e.g., $G=64$).
2. Reward Computation: A reward function (e.g., a rule-based checker for a math problem) assigns a scalar reward $r_i$ to each of the $G$ responses.
3. Monte Carlo Baseline: The baseline is then calculated as the average reward of the group.
   $b(q)=\frac{1}{G}{\sum }_{i=1}^G{r}_i$
   This is a Monte Carlo estimate of the expected reward for that prompt. With a large enough group, this average converges to what a perfectly trained critic would predict, but without needing to train a model at all.
4. Group-Relative Advantage Calculation: The advantage for a specific response $i$ is its reward compared to this group-average baseline. To improve stability across different prompts, GRPO normalizes this advantage by the standard deviation of the group's rewards.

The following image illustrates this critic-free advantage calculation perfectly.

This "group-relative" comparison is intuitive: it evaluates how good a response is, not in an absolute sense, but relative to other possible responses the model could have generated in the same situation.

This next reading provides a clear, step-by-step explanation of how PPO's critic is replaced by this group sampling method.

The diagram below provides a high-level architectural comparison, visually summarizing how GRPO simplifies PPO.

By replacing the complex, learned critic with a simple, empirical group average, GRPO achieves a significant reduction in complexity and resource usage while maintaining the stability that makes PPO effective.

Conclusion

In this lesson, we have focused on the central innovation of Group Relative Policy Optimization: the elimination of the critic network. We saw that the critic in PPO, while theoretically sound, introduces significant computational overhead and training complexity, particularly for LLM fine-tuning where reward signals are often sparse.

Key Takeaways:

• PPO's critic network is a learned value function $V(s)$ used as a baseline to calculate advantage estimates.
• Training this critic is expensive, complex, and ill-suited for tasks with only final-outcome rewards.
• GRPO eliminates the critic by replacing its baseline with a Monte Carlo estimate derived from group sampling.
• For each prompt, GRPO samples a group of responses and uses their mean reward as the baseline.
• The advantage for a given response is then its own reward relative to the group's average reward, a method called group-relative reward comparison. This critic-free approach significantly reduces the memory and computational costs of RL-based alignment.

Preview of the Next Lesson:

Now that we understand how GRPO eliminates the critic to calculate advantages, our next lesson will formalize the complete algorithm. We will describe the full GRPO training loop, including the group sampling, reward computation, advantage normalization, and finally, the clipped policy update that uses this advantage to train the model. This will give us a complete picture of the algorithm used to train models like DeepSeek-R1.