Traditional vs. Unified Generative Planning

The Traditional Paradigm: The Modular Pipeline (AV 1.0)

The classical approach to building an autonomous vehicle (AV) system is rooted in a "divide and conquer" strategy. The complex problem of driving is broken down into a series of smaller, more manageable sub-problems, each handled by a specialized software module. This is often referred to as the "modular pipeline" or "AV 1.0."

To understand the components of this pipeline, please read the following article, which provides a clear breakdown.

As the article describes, the data flows sequentially through this pipeline:

1. Perception: Senses the world using cameras, LiDAR, etc., to answer the question, "What is around me?" This involves tasks like 3D object detection (e.g., using VoxelNet) and segmentation.
2. Prediction: Takes the output of the perception module to forecast the future behavior of other agents (vehicles, pedestrians). It answers, "What will those other agents do next?"
3. Planning: Uses the outputs from perception and prediction to chart a safe and efficient path for the ego-vehicle. It answers, "What should I do?"
4. Control: Executes the plan by sending low-level commands (e.g., acceleration, steering angle) to the vehicle's hardware.

This modular design is intuitive and allows specialized teams to work on individual components. However, it has a critical weakness. Watch this short clip from a talk by the creators of UniAD, a modern AV system, who start by describing the limitations of this traditional approach.

The key issue with the modular pipeline is error propagation. Each module is optimized independently, and an error made in an early stage (e.g., the perception module fails to detect a distant car) is passed down the chain. The prediction module cannot forecast the behavior of a car it doesn't know exists, and the planning module cannot plan to avoid it. There is no feedback mechanism for the planner to signal to the perception system that something might be wrong.

The image below visually contrasts this "Standalone Models" approach with more integrated designs.

The New Paradigm: Unified Generative Planning (AV 2.0)

The limitations of the modular pipeline led researchers to explore a new paradigm: unified end-to-end planning. This approach, often called "AV 2.0," abandons the series of hand-engineered modules in favor of a single, holistic system that learns to drive by mapping sensor inputs directly to driving decisions.

Let's hear from Alex Kendall, CEO of Wayve, a leading company in this space, as he defines the shift from AV 1.0 to AV 2.0.

Instead of optimizing for intermediate tasks like object detection accuracy, the entire system is optimized for the final planning objective. This allows the model to learn a more robust and nuanced understanding of the driving task.

The core idea is to treat driving not as a pipeline of classification and regression problems, but as a generative modeling problem. The model's goal is to generate a future trajectory that is safe, comfortable, and makes progress toward the destination.

The research paper for GenAD (Generative End-to-End Autonomous Driving) provides an excellent, technically deep comparison of these two paradigms.

Advantages of the Unified Approach

The GenAD paper highlights the key advantages of unified generative planning over the traditional modular pipeline:

1. Modeling High-Order Interactions: A modular pipeline predicts how other cars will move, then plans the ego-vehicle's path. It cannot easily model how the ego-vehicle's own tentative plan might influence other cars, which in turn influences the ego-plan. A unified generative model, by generating the entire future scene simultaneously, can capture this complex, interactive dynamic.

2. Learning Structural Priors: Real-world trajectories are not random; they are smooth and follow physical laws. A generative model can learn this "structural prior" of what a realistic future looks like, leading to more plausible and safer plans.

3. End-to-End Optimization: By training the system with a final planning loss, all intermediate representations are automatically optimized to serve the ultimate goal of driving well. This avoids issues where a component excels at its intermediate metric (e.g., perception achieves 99% detection accuracy) but still fails in a way that is critical for the downstream planner.

The diagram below from another paper illustrates this well. The more modular approaches (a, b) suffer from "Information Loss" between perception and planning, whereas the unified approach (c) integrates simulation and multi-target learning directly into the planning process.

Conclusion

Today we've drawn a clear line between the two dominant architectural philosophies in autonomous driving. The traditional modular pipeline offers interpretability and engineering simplicity but suffers from error propagation. The unified generative paradigm offers a more holistic, robust, and scalable solution by framing driving as a single end-to-end learning problem.

Key Takeaways:

This fundamental shift from a modular pipeline to a unified generative model is what enables the advanced systems we will be studying, like DAP and Plan-R1, which literally treat "planning as language modeling."

Preview of the Next Lesson:
Now that we understand why the field is moving toward unified generative planners, our next lesson will explore how these systems work. We will begin by dissecting a critical component mentioned in your syllabus: the Vector-Quantized Variational Autoencoder (VQ-VAE). We will explain its architecture and see how it provides the essential bridge for turning the continuous, high-dimensional visual world into the discrete tokens that a transformer-based planner can understand and process.