Understanding VD-GRPO and Centering for Safe Reinforcement Learning

1. The Flaw in Standard GRPO for Safety-Critical Tasks

Let's begin by recalling the core mechanism of GRPO. Unlike PPO, which requires a trained critic network to estimate a baseline, GRPO computes the advantage for a given trajectory by comparing its reward to the average reward of a group of trajectories sampled for the same prompt.

To see the standard formula for the GRPO advantage, let's watch a brief clip.

As the video explains, the standard GRPO advantage for a trajectory $i$ within a group is:

where:

• $r_i$ is the reward for the $i$-th trajectory.
• ${\mu }_R$ is the mean reward of the group.
• ${\sigma }_R$ is the standard deviation of the rewards in the group.
• $ϵ$ is a small constant for numerical stability.

This normalization works well for tasks like math or coding, where it helps stabilize training by ensuring advantages fall within a consistent range. However, it has a perilous side effect in safety-critical applications.

The division by the standard deviation (${\sigma }_R$) erases the absolute magnitude of the rewards. All that matters is the relative ranking within the group.

Consider this scenario:
An autonomous vehicle is in a difficult situation where every possible short-term trajectory leads to a collision.

• Trajectory A: A severe head-on collision. Reward = -1000.
• Trajectory B: A minor side-swipe. Reward = -100.
• Trajectory C: Another minor collision. Reward = -120.

The mean reward ${\mu }_R$ is -406.7. The standard deviation ${\sigma }_R$ is large, around 495.
Using standard GRPO, the advantage for Trajectory B (the "least bad" option) would be:
$\text{Advantage}(B)=\frac{-100-(-406.7)}{495}\approx \frac{306.7}{495}\approx +0.62$
The model receives a positive advantage signal for a trajectory that results in a crash. It learns that this collision is a "good" action simply because it was better than the other catastrophic options. This is an unacceptable failure mode. We need the model to understand that all of these options are bad.

The Plan-R1 paper was the first to formally identify this critical limitation.

2. The Solution: Variance-Decoupled GRPO (VD-GRPO)

The authors of Plan-R1 propose a simple but profound change to the advantage calculation: decouple the advantage from the group's variance. This is the core of Variance-Decoupled GRPO (VD-GRPO).

The fix is to remove the division by standard deviation and replace it with division by a fixed, global scaling constant, $c$.

The VD-GRPO advantage calculation is:

This is essentially just centering the rewards around the group mean. Let's revisit our collision scenario with this new formula (assuming $c=1$ for simplicity):

• $r_A=-1000$, $r_B=-100$, $r_C=-120$. ${\mu }_R=-406.7$.
• $\text{Advantage}(A)=-1000-(-406.7)=-593.3$
• $\text{Advantage}(B)=-100-(-406.7)=+306.7$
• $\text{Advantage}(C)=-120-(-406.7)=+286.7$

Wait, this doesn't seem to have fixed the problem; the "least bad" crash still gets a positive advantage. This reveals a subtlety in how the rewards are designed. The safety gate ensures a collision results in a reward of 0. Soft costs are positive. So, a collision trajectory gets $R=0$, while safe trajectories get $R>0$.

Let's re-run the scenario with the correct reward structure:

• Trajectory A (Collision): Reward $r_A=0$.
• Trajectory B (Collision): Reward $r_B=0$.
• Trajectory C (Safe, but low progress): Reward $r_C=10$.

Standard GRPO:

• ${\mu }_R=(0+0+10)/3=3.33$. ${\sigma }_R\approx 4.71$.
• $\text{Advantage}(A)=(0-3.33)/4.71\approx -0.71$. (Negative, which is good).
• $\text{Advantage}(C)=(10-3.33)/4.71\approx +1.41$. (Positive, also good).

Now, consider a group of all unsafe trajectories:

• Trajectory A (Collision): $r_A=0$.
• Trajectory B (Collision): $r_B=0$.
• Trajectory C (Collision): $r_C=0$.

Standard GRPO:

• ${\mu }_R=0$, ${\sigma }_R=0$.
• $\text{Advantage}(A)=(0-0)/(0+ϵ)\approx 0$.
  The model receives zero advantage for a catastrophic action. It gets no learning signal that this was bad.

VD-GRPO:
Now let's imagine the model is in a scenario where it can choose between a safe but inefficient path ($r=10$) and a group of unsafe paths ($r=0$). VD-GRPO preserves the absolute scale difference. The advantages for the unsafe group will be calculated relative to their mean of 0, resulting in an advantage of 0. The advantage for the safe path is calculated relative to its own group's mean, which will likely be positive. The optimizer sees a clear gradient towards the safe actions and a zero gradient for the unsafe ones.

By preserving the absolute reward magnitudes, VD-GRPO ensures that a group of trajectories with low rewards (e.g., all collisions) will produce low or negative advantages, preventing the model from reinforcing unsafe behavior.

The paper provides compelling visual evidence of this effect.

As you can see in Figure 4, standard GRPO causes the advantage distributions for safe and unsafe groups to overlap significantly. In contrast, VD-GRPO clearly separates them, with unsafe groups naturally producing larger advantages (and thus larger gradients), forcing the optimizer to pay attention to them. Figure 5 confirms the result: VD-GRPO steadily reduces the rate of unsafe plans.

3. A Related Problem: The Length Bias

The issue with variance normalization in GRPO is part of a broader class of biases that can emerge from implementation details. Another significant bias, which VD-GRPO's design philosophy helps mitigate, is length bias.

A video from the authors of a paper titled "Dr. GRPO" (GRPO Done Right) explains this phenomenon very clearly.

As the video explains, many RL implementations normalize the loss by the length of the generated sequence. This has an unintended consequence:

• For a correct answer (positive advantage), the reward is concentrated over fewer tokens if the answer is short, leading to a stronger update. The model learns to be brief when it's right.
• For an incorrect answer (negative advantage), the penalty is diluted over more tokens if the answer is long. The model learns it can reduce its penalty for being wrong by generating a longer, more verbose incorrect response.

This is exactly the opposite of what we want in a safety-critical system. The solution proposed by Dr. GRPO is to replace the dynamic division by response length with a division by a fixed global constant (like max_token_length).

This is philosophically identical to VD-GRPO's solution. By replacing a dynamic, group-specific normalizer (${\sigma }_R$) with a fixed, global constant ($c$), VD-GRPO not only solves the variance-scaling problem but also avoids introducing downstream biases like length bias. It ensures that the learning signal is clean and reflects the true priorities encoded in the reward function.

Conclusion

In this lesson, we've uncovered a critical flaw in applying standard reinforcement learning algorithms to safety-critical domains and examined the elegant solution provided by VD-GRPO.

Key Takeaways:

• The Flaw in Standard GRPO: Group-wise normalization of rewards ($(r-\mu )/\sigma$) erases the absolute magnitude of rewards. This can cause the algorithm to assign positive advantage to the "least bad" option in a group of entirely unsafe trajectories, reinforcing dangerous behavior.
• The VD-GRPO Solution: By replacing normalization with simple centering and a fixed scaling constant ($(r-\mu )/c$), VD-GRPO preserves the absolute scale of rewards. A group of bad trajectories will consistently produce negative or zero advantages, providing a clear signal to avoid them.
• Preventing Implicit Biases: This design philosophy also helps prevent related issues like length bias, where the model could learn to generate longer incorrect responses to dilute penalties. Using fixed constants rather than dynamic normalizers provides a more stable and predictable optimization landscape.

Preview of the Next Lesson:

We have now completed our analysis of the Plan-R1 architecture, from its generative foundation and tokenization scheme to its sophisticated safety alignment via a rule-based reward function and VD-GRPO. But how do we know if it actually works? In our next and final lesson of this module, we will explore how autonomous driving planners are evaluated, distinguishing between open-loop and closed-loop testing and discussing industry-standard benchmarks like nuPlan.