GRPO vs. PPO: Verifiable Rewards for Reference-Free Safety Alignment

Introduction

Welcome to the lesson. In our previous session, we drilled down into a crucial detail of the Plan-R1 model: the shift from standard Group Relative Policy Optimization (GRPO) to Variance-Decoupled GRPO (VD-GRPO). We saw that for safety-critical tasks, normalizing rewards by the group's standard deviation is dangerous, as it can inadvertently reinforce unsafe actions. VD-GRPO's simple fix—replacing normalization with centering—preserves the absolute scale of rewards, ensuring that a group of bad trajectories is always recognized as bad.

Today, we will zoom out and place GRPO in the broader context of reinforcement learning alignment. Our goal is to compare GRPO with the industry-standard Proximal Policy Optimization (PPO), specifically for the task of trajectory planning. You will learn why combining GRPO with the rule-based rewards we've designed—a paradigm known as Reinforcement Learning with Verifiable Rewards (RLVR)—creates a powerful, efficient, and transparent framework for reference-free safety alignment.

1. The Baseline: Proximal Policy Optimization (PPO)

To appreciate what makes GRPO different, we must first understand the algorithm it was designed to improve upon: Proximal Policy Optimization (PPO). PPO was the workhorse behind the alignment of models like ChatGPT and is a cornerstone of modern Reinforcement Learning from Human Feedback (RLHF).

PPO is notoriously complex, often requiring four separate models to operate simultaneously during training.

A brief introduction to PPO: RL's workhorse algorithm

This article by Sebastian Raschka provides an excellent high-level breakdown of the PPO algorithm and the different models involved. It's a great starting point for our comparison.

Please read the section titled 'A brief introduction to PPO: RL's workhorse algorithm'. Focus on the final summary which lists the four models involved: the policy, the reward model, the critic, and the reference model. Try to understand the distinct role of each.

As the article explains, a typical PPO setup for LLM alignment involves:

The Policy (Actor): The main language model we are training. It generates outputs (e.g., trajectories).
The Reward Model (RM): A separate, large model trained on human preference data to predict a reward score for the policy's outputs. It acts as a proxy for human judgment.
The Critic (Value Model): Another trainable model that estimates the "value" or expected future reward of a given state. Its purpose is to provide a baseline to calculate the advantage (how much better or worse an action was than expected).
The Reference Model: A frozen, initial copy of the policy model. It's used to calculate a KL divergence penalty, which prevents the policy from changing too drastically and "forgetting" its core language capabilities.

The core of the PPO update relies on the advantage, which is roughly Advantage = Reward - Value. The policy is then updated to increase the probability of actions that yield a positive advantage.

The video below gives a great visual walkthrough of this process.

Group Relative Policy Optimization(GRPO) Visualized

This animation visualizes the PPO algorithm. It will help clarify how the policy, reward model, and value model interact to produce a policy update.

Watch from the beginning to 03:10. Pay close attention to how the 'advantage value' is calculated using both the reward model and the value model, and how this advantage is then used to update the policy.

The key takeaway is that PPO is resource-intensive. Training the critic network is particularly challenging, as it's trying to predict the highly variable and complex value of states in a dynamic environment.

2. An Efficient Alternative: Group Relative Policy Optimization (GRPO)

GRPO was introduced by DeepSeek AI as a more efficient variant of PPO. Its primary innovation is the elimination of the critic network.

Instead of training a separate value model to provide an expected reward baseline, GRPO computes the advantage by comparing a trajectory's reward to the average reward of a group of trajectories sampled for the same prompt.

PPO vs. GRPO: Architectural and Objective Comparison — This diagram from the DeepSeekMath paper clearly illustrates the main architectural difference: PPO (left) relies on an Actor-Critic framework, while GRPO (right) dispenses with the Critic entirely.

Let's watch a visualization of how this works.

Group Relative Policy Optimization(GRPO) Visualized

This video segment visually explains the core mechanism of GRPO. Focus on how it uses a group of sampled responses to calculate the advantage.

Please watch from 04:19 to 06:07. The video explains how GRPO samples a group of responses, calculates their rewards, and then normalizes these rewards against the group mean to determine the advantage for each response.

The GRPO advantage for a trajectory $i$ is calculated relative to its peers:

Advantage_{GRPO} (r_{i}) = \frac{r _{i} - μ _{group}}{σ _{group}}

This simple change has profound implications:

No Critic Training: It eliminates the need to train and maintain a separate, large value model, significantly reducing computational cost and complexity.
Adaptive Baseline: The baseline for the advantage ( $μ_{group}$ ) is dynamic and context-specific, calculated on-the-fly from the current policy's outputs for a given situation.

3. Verifiable Rewards and Reference-Free Alignment

The second key piece of our comparison is the source of the reward signal itself. PPO, in the context of RLHF, relies on a learned reward model trained on noisy and expensive human preference data.

However, for tasks with objective, verifiable correctness—like math, coding, or adhering to traffic laws—we can bypass the reward model entirely. This paradigm is called Reinforcement Learning with Verifiable Rewards (RLVR).

RL reward modeling: from RLHF to RLVR

This section of Sebastian Raschka's article clearly distinguishes RLHF from RLVR and explains why RLVR is a more efficient and reliable approach for tasks with objective correctness.

Please read the section 'RL reward modeling: from RLHF to RLVR'. Note how RLVR uses deterministic, rule-based tools instead of a trained reward model.

The rule-based reward functions we designed for Plan-R1 are a perfect example of a verifiable reward system.

Collision Check: is_collision(trajectory) is a deterministic, geometric check. It returns a definitive true/false.
Speed Limit Check: is_over_speed_limit(trajectory) is a simple numerical comparison.

When we combine the critic-free nature of GRPO with the reward-model-free nature of RLVR, we get a highly efficient and robust alignment framework.

Comparison of RLHF and RLVR Architectures — This diagram illustrates the core difference. RLHF (right) uses a learned Reward Model. RLVR (left) uses a direct, verifiable reward function (e.g., `reward = 1 if correct else 0`).

This combination is what enables reference-free safety alignment. The term "reference-free" can be interpreted in two ways here:

Critic-Free Advantage: The advantage calculation is self-referential to the sampled group, not to a baseline from an external, trained critic network.
Reward-Model-Free Alignment: The safety principles are not learned by proxy from human preferences. They are encoded directly in verifiable, rule-based code. The "reference" for what is safe is not a learned model, but the ground-truth rules of physics and traffic law.

4. Comparison Summary: PPO vs. GRPO for Trajectory Planning

Let's consolidate our comparison in the specific context of autonomous driving trajectory planning, as exemplified by Plan-R1.

Feature	PPO (with RLHF)	GRPO/VD-GRPO (with RLVR)	Rationale for Trajectory Planning
Components	Policy, Critic, Reward Model, Reference Model	Policy, Reference Model, Rule-Based Reward Fn	GRPO is far more computationally efficient, eliminating two large neural networks (Critic & RM).
Reward Source	Learned from human preferences	Hard-coded, verifiable rules (e.g., collision checks)	Verifiable rewards provide unambiguous, trustworthy safety signals, avoiding the risk of "reward hacking" a learned RM.
Advantage Calc.	`r - V(s)` (Reward - Value from Critic)	`(r - μ_group) / c` (Centered group-relative reward)	GRPO's approach is simpler and avoids the instability of training a high-variance value function for complex driving states.
Safety Alignment	Indirect, via learned reward model	Direct, via verifiable rules	GRPO+RLVR grounds the model directly in first principles of safety, providing a more robust and transparent alignment.

The authors of Plan-R1 explicitly chose this path for these reasons. They used a GRPO-based approach (specifically, their own VD-GRPO variant) precisely because it pairs so effectively with rule-based rewards for safety alignment.

Plan-R1: Safe and Feasible Trajectory Planning as Language ...

The Plan-R1 paper summary highlights the benefits of this approach.

Read the sections 'Stage 2: Reinforcement Learning Fine-tuning' and 'Technical Contributions and Significance'. Notice the emphasis on using rule-based rewards over human preference data for consistency and verifiability.

In essence, for a safety-critical task like driving, PPO asks a trained critic to estimate the value of a state and a trained reward model to score the outcome. GRPO, combined with RLVR, simply asks a piece of code: "Did it crash?" This direct, verifiable approach is inherently more reliable.

Conclusion

In this lesson, we compared the PPO and GRPO algorithms for trajectory planning, concluding that GRPO offers significant advantages in this domain.

Key Takeaways:

PPO is Resource-Intensive: It requires four models, including a trained critic (value model) and a trained reward model, making it computationally expensive and complex.
GRPO is Critic-Free: It simplifies the architecture by calculating advantage relative to a sampled group average, eliminating the need for a critic network.
Verifiable Rewards (RLVR) are Superior for Safety: For tasks with objective correctness like driving, using deterministic, rule-based reward functions (RLVR) is more reliable and transparent than using a learned reward model (RLHF).
GRPO + RLVR = Reference-Free Safety Alignment: This combination creates a powerful framework. The advantage calculation is self-referential to the group, and the safety criteria are grounded in verifiable code, not a learned proxy. This makes the system more efficient, stable, and trustworthy for safety-critical applications.

Preview of the Next Lesson:

We have now thoroughly analyzed the Plan-R1 architecture, its tokenization, its reward function, and its alignment algorithm. But a planner is only as good as its real-world performance. In our next lesson, we will shift our focus to evaluation, exploring how autonomous driving systems are tested and benchmarked, and distinguishing between open-loop and closed-loop evaluation methodologies.