Plan-R1: Two-Stage Training (Imitation + GRPO)

The Two-Stage Paradigm: A Common Pattern in Modern AI

The idea of first learning a general skill and then refining it for a specific goal is a recurring theme in state-of-the-art AI. Before we look at Plan-R1, let's consider the training pipeline for DeepSeek-R1, a model designed for complex reasoning tasks. This analogy will help situate Plan-R1's approach within a broader context.

As the video explains, DeepSeek-R1 first undergoes Supervised Fine-Tuning (SFT) on "Chain of Thought" examples. This initial stage teaches the model the format and structure of reasoning. Only after this foundation is laid does the model undergo RL to improve the correctness of its reasoning.

Plan-R1 applies the exact same philosophy to autonomous driving:

1. Stage 1: Learn the "grammar" of driving by imitating human experts.
2. Stage 2: Refine the learned behavior to ensure it adheres to safety principles.

Deconstructing the Plan-R1 Framework

Let's begin with a clear, high-level description of the Plan-R1 framework from a summary of the paper.

The core idea is to decouple behavior learning from principle alignment. The framework is visualized beautifully in the original paper.

This diagram is our roadmap for the lesson. Let's break down each stage.

Stage 1: Imitation Pre-training

The goal of the first stage is not to create a perfect driver, but to create a plausible one. The model needs to learn the fundamental patterns of vehicle movement, multi-agent interactions, and road navigation.

• Goal: Learn to generate diverse, human-like driving behaviors.
• Method: Autoregressive pre-training on a large dataset of expert (human) driving demonstrations. The model is trained with a next-motion-token prediction objective, which is directly analogous to the next-word prediction objective used to train models like GPT.
• Outcome: A generative model that can produce trajectories that look and feel human. As we learned in the last lesson, this also means it will replicate human flaws.

Let's look at the paper's description of this stage.

At the end of Stage 1, we have a competent but potentially unsafe planner. It's a student driver that has watched thousands of hours of driving footage and can mimic the actions, but doesn't yet understand the underlying rules of the road.

Stage 2: GRPO Fine-tuning for Alignment

The second stage is where the model is taught to be a safe and responsible driver. It takes the human-like prior from Stage 1 and aligns it with explicit, hard-coded principles.

• Goal: Correct the flaws from Stage 1 (e.g., speeding, tailgating) and align the model with principles of safety, comfort, and legality.
• Method: Reinforcement Learning fine-tuning using rule-based rewards and Group Relative Policy Optimization (GRPO).
• Outcome: A planner that retains human-like, naturalistic driving behavior but actively avoids unsafe actions.

Since RL is a new topic for you, let's get a quick, intuitive explanation of GRPO.

The video provides the key insight: GRPO works by generating a group of candidate outputs, using a reward function to score them, and then updating the model to make it more likely to produce the high-reward outputs. This is perfect for driving, where we can write simple, deterministic rules to score a trajectory:

• Did it cause a collision? Very bad reward.
• Did it stay in the lane? Good reward.
• Did it exceed the speed limit? Bad reward.

Now, let's see how the Plan-R1 paper describes this fine-tuning stage.

To summarize this crucial stage:

1. It starts with the pre-trained model from Stage 1.
2. In a given scenario, the model generates a group of possible trajectories.
3. Each trajectory is scored using deterministic, code-based rule-based rewards.
4. GRPO then updates the model to favor trajectories that received higher scores (i.e., were safer and more compliant).
5. A Dual-Model Rollout is used to make the process realistic: a trainable model plans the ego-vehicle's path, while a frozen (unchanging) copy of the pre-trained model predicts how other agents on the road will react.

Evaluating the Impact

This two-stage process is elegant, but does it work? The Plan-R1 paper provides a clear ablation study that compares the performance of the model after Stage 1 (pre-training only) with the final model after Stage 2 (fine-tuning).

Let's analyze this result:

• Quantitative Results (Table): The table shows that fine-tuning (Plan-R1 w/ GRPO) significantly improves performance across the board compared to the pre-trained model. Most importantly, it leads to a dramatic improvement in safety metrics like Drivable Area Compliance and a reduction in Speeding.
• Qualitative Results (Images): The visual examples are even more compelling. In row (a), the pre-trained model makes critical errors like driving off-road or speeding. In row (b), the fine-tuned Plan-R1 model corrects these mistakes, producing a safe and legal trajectory in the exact same scenario.

This evidence confirms that the two-stage framework is highly effective. Stage 1 provides the behavioral prior, and Stage 2 successfully aligns it with safety principles.

Conclusion

In this lesson, we have deconstructed the two-stage training framework of Plan-R1, a state-of-the-art approach to safe autonomous planning.

Key Takeaways:

• Plan-R1 employs a two-stage framework that decouples behavior learning from principle alignment, a pattern common in modern AI.
• Stage 1 (Imitation Pre-training): An autoregressive model learns to generate diverse, human-like driving behaviors by training on expert data with a next-token prediction objective. This teaches the model how to drive.
• Stage 2 (GRPO Fine-tuning): The pre-trained model is then aligned with safety principles using reinforcement learning. It uses rule-based rewards and GRPO to correct the unsafe habits learned in Stage 1, teaching the model the why of safe driving.
• This approach effectively combines the strengths of imitation learning (generating naturalistic behavior) and reinforcement learning (optimizing for explicit objectives).

Preview of the Next Lesson:

We have seen the what and why of the Plan-R1 framework. In our next lesson, we will dive into the how. Our focus will be on the fine-tuning stage as we learn to design a rule-based reward function to penalize collisions and enforce hard safety constraints. We will examine the mathematical logic used to translate abstract safety concepts into concrete reward signals that guide the GRPO algorithm.