The Batch Relative Advantage Estimator

In our last lesson, we explored the motivation for GRPO's critic-free design, contrasting it with PPO's more complex actor-critic framework. We established that GRPO achieves its efficiency by replacing the learned value function (the critic) with a simpler, empirical baseline.

Today, we will derive and dissect this very baseline: the group relative advantage estimator. This component is the heart of GRPO's innovation, enabling it to evaluate policy updates without needing a separate critic network. By the end of this lesson, you will understand exactly how the advantage for a given model generation is calculated from a batch of its peers.

From Learned Baselines to Empirical Baselines

Let's quickly recall the role of the advantage function in policy gradient methods. The advantage, often denoted as $A(s,a)$, quantifies how much better a particular action $a$ is compared to the average action one might take in state $s$. A standard formulation is $A(s,a)=Q(s,a)-V(s)$, where the value function $V(s)$ serves as a variance-reducing baseline.

In PPO, $V(s)$ is learned by a critic network. GRPO's central idea is to construct a baseline directly from the data sampled during a rollout. This process involves three key steps:

1. Group Sampling: For a given input prompt $q$, the current policy ${\pi }_{\theta }$ is used to generate a group of $G$ distinct output sequences, $\{o_1,o_2,\dots ,o_G\}$.
2. Reward Evaluation: A reward function $R$ (e.g., a simple binary check for correctness, or a more complex reward model) assigns a scalar reward $r_i$ to each output $o_i$. This gives us a set of rewards $\{r_1,r_2,\dots ,r_G\}$.
3. Advantage Calculation: The advantage for each output is calculated based on its performance relative to the group.

Deriving the Group Relative Advantage

The "relative" aspect is key. Instead of comparing an action's outcome to a learned expectation $V(s)$, we compare it to the empirical average performance on the same task.

The baseline in GRPO is simply the mean reward of the group:

An initial, unnormalized advantage for output $o_i$ is then $r_i-\bar{r}$. A positive value means the output was better than the group average, while a negative value means it was worse.

However, policy gradient updates are sensitive to the scale of the advantage values. To create a more stable and robust estimator, GRPO normalizes this raw advantage. Given your background in statistics, you'll recognize this normalization as the computation of a standard score (z-score). We divide the raw advantage by the standard deviation of the group's rewards.

This gives us the final group relative advantage estimator, $A_i$, for the $i$-th output:

The official DeepSeek paper presents this formula concisely. Take a moment to see how it's formally defined.

Building Intuition for the Estimator

Now that you've seen the formal definition, let's explore the intuition behind its components. The following video provides an excellent breakdown of what each part of the advantage formula accomplishes.

As the video explains, this normalization does more than just control the scale of the updates. By dividing by the standard deviation, the algorithm becomes more sensitive to outcomes on prompts where the policy's performance is consistent (low std. dev.) and more conservative on prompts where performance is mixed (high std. dev.).

For a quick visual recap of this entire process, watch the following clip.

A Worked Example

Let's make this concrete with a numerical example. The Hugging Face guide on GRPO provides a clear, step-by-step calculation.

As you saw in the example:

• A group of 8 responses with rewards $\{1,0,1,0,1,0,1,0\}$.
• Mean: $\bar{r}=(4\times 1+4\times 0)/8=0.5$.
• Standard Deviation: The example uses a value of 0.53. A more precise calculation (using population standard deviation, which is common in these implementations) is $\sqrt{0.25}=0.5$. Let's use 0.5 for simplicity.
• Advantage for a correct response ($r_i=1$): $A_i=(1-0.5)/0.5=1.0$.
• Advantage for an incorrect response ($r_i=0$): $A_i=(0-0.5)/0.5=-1.0$.

The resulting positive and negative advantages will be used to respectively increase or decrease the probability of generating those responses in the future.

Self-Check

Let's test your understanding. Suppose for a given prompt, you sample a group of $G=4$ responses and obtain the following binary rewards: $\{0,1,0,0\}$.

Calculate the group relative advantage $A_i$ for the correct response ($r_i=1$). You can use the population standard deviation formula: $\sigma =\sqrt{\frac{1}{G}{\sum }_{i=1}^G(r_i-\bar{r}{)}^2}$.

Conclusion

In this lesson, we have formally derived the group relative advantage estimator, the cornerstone of GRPO's critic-free architecture.

Here are the key takeaways:

• Critic Replacement: GRPO replaces PPO's learned critic with a non-parametric, empirical baseline computed on-the-fly.
• Derivation Steps: The advantage is calculated by (1) sampling a group of responses, (2) collecting their rewards, (3) subtracting the group's mean reward, and (4) normalizing by the group's reward standard deviation.
• Statistical Intuition: The resulting estimator is effectively the z-score of a response's reward within its sample group. This provides a scaled, relative measure of performance.
• Benefit: This simple statistical calculation provides a powerful-enough signal for policy updates while completely sidestepping the cost and complexity of training a separate critic network.

We now have the advantage term $A_i$. In our next lesson, we will see how this term is integrated into the full GRPO objective function, alongside the clipped policy ratio from PPO and a regularizing KL-divergence penalty.