Imitation Learning's Unsafe Replications

Part 1: The Flaw in the Foundation: Imperfect Experts

Imitation Learning (IL), also known as Behavior Cloning (BC), is a powerful and intuitive paradigm. The goal is to train a policy, $\pi$, that maps states (sensor inputs, map data) to actions (steering, acceleration) by simply minimizing the difference between the model's actions and the actions of an expert (a human driver) in the same state.

This approach is appealing because it bypasses the need to manually engineer complex rules for every possible driving scenario. Let's start with a video that introduces the core concepts of imitation learning and its application to autonomous driving.

The video makes a compelling case for IL. If humans can navigate complex traffic, the optimal behavior must be implicit in our driving data. However, this relies on a crucial assumption: that the human demonstrations are, in fact, optimal.

Let's examine the reality of this assumption by looking at the Plan-R1 paper, which is a central model in our course.

As the paper highlights, real-world "expert" data is not perfect. The authors found that over 10% of scenes in the nuPlan training data exhibit speeding. This is a concrete example of the problem: if our training data contains undesirable behaviors, an imitation learning model trained on this data will learn to replicate them. The model, aiming to minimize the error between its output and the human's, will learn that speeding is sometimes the "correct" action because that's what the expert did.

This direct replication of bad habits is the most obvious failure mode of imitation learning, but the problem runs much deeper.

Part 2: Causal Confusion and Compounding Errors

A more subtle and dangerous issue arises when the model learns the wrong cause-and-effect relationships from the data. This is known as causal confusion. The model might identify a spurious correlation in the data and treat it as a causal link, leading to brittle and illogical behavior.

The same video we watched earlier provides a perfect illustration of this phenomenon, which it calls the "inertia problem."

In the video's example, the model learns the rule: "If my previous action was 'stop', my next action should be 'stop'." This is a strong correlation in the training data because humans drive smoothly. However, it's not the true causal reason for stopping. The true cause is the state of the traffic light or the presence of a pedestrian.

When the learning agent makes a small mistake (e.g., a slight, unnecessary braking action), this error feeds back into its own input for the next time step. The model sees its own previous action was "stop" and, following its flawed logic, decides to stop again, leading to a catastrophic failure: stopping at a green light.

This compounding of errors is a fundamental challenge in imitation learning. It's formally known as covariate shift:

• The training distribution consists of states visited by the human expert.
• The test distribution consists of states visited by the learning agent itself.

Because the agent is imperfect, it will inevitably drift away from the expert's trajectories and into states it has never seen before. Since it has no training data for these novel states, it doesn't know how to act, is likely to make a larger error, and drifts even further away.

The Plan-R1 paper provides a stark visualization of this problem in practice.

This case study perfectly demonstrates the learning outcome. The IL models, trained to mimic the expert, faithfully replicate the expert's unsafe speeding.

Part 3: The Social Dilemma: The Rules of Breaking the Rules

Finally, there is a higher-level challenge. Driving is not a solitary activity performed in a vacuum; it's a multi-agent social game. Sometimes, strictly adhering to every rule can make a vehicle ineffective or unpredictable to other drivers.

This puts imitation learning in a difficult position:

• If trained on data from overly cautious, perfectly rule-abiding drivers, the model may be too timid, getting stuck in merges or at intersections.
• If trained on data from typical, "assertive" human drivers, the model will learn to speed, follow too closely, and make other potentially unsafe maneuvers.

The training data is a mixture of these behaviors, and a simple IL model has no robust way to distinguish the "good" assertiveness from the "bad" recklessness. It simply learns to replicate the average behavior, flaws and all.

Conclusion

In this lesson, we've explored the fundamental reasons why naive imitation learning is insufficient for developing safe autonomous driving systems.

Key Takeaways:

• Imitation Learning (IL) trains a model to mimic expert behavior by learning a mapping from states to actions from a dataset of demonstrations.
• Imperfect Data: Real-world human driving data is not perfect. It often contains undesirable and unsafe behaviors, such as speeding, which IL models will learn to replicate.
• Causal Confusion: Models can learn spurious correlations instead of true causal relationships. This can lead to issues like the "inertia problem," where a car learns to stay stopped because it was previously stopped, not because of the actual traffic situation.
• Covariate Shift: Small errors made by the learning agent can compound over time through a feedback loop. This causes the agent to drift into states it has never seen during training, where its behavior is undefined and likely to be erroneous.

Preview of the Next Lesson:

Now that we have a clear understanding of the problem, we can begin to explore the solution. In our next lesson, we will introduce Plan-R1's two-stage framework. We will see how it uses imitation learning in the first stage to capture human-like driving behavior, and then employs a second stage of reinforcement learning with GRPO to explicitly fine-tune the model and align it with hard-coded safety principles, effectively "unlearning" the bad habits from the initial training data.