Introduction
Welcome to our final technical lesson of the course. In our journey so far, we have deconstructed modern Transformer architectures, explored advanced alignment techniques like GRPO, and designed a complete generative planner, Plan-R1, for autonomous driving. We have a model trained to be both human-like and safe, but this raises the most critical question in any engineering discipline: how do we prove that it works?
This lesson addresses that question by focusing on evaluation. Your learning goal is to distinguish between open-loop and closed-loop evaluation metrics for autonomous driving planners. We will examine why this distinction is fundamental to understanding a planner's real-world performance. We'll use two of the most important industry benchmarks, nuPlan and NAVSIM, as our case studies to see how these evaluation paradigms are implemented in practice.
1. The Fundamental Distinction: Open-Loop vs. Closed-Loop
At the heart of planner evaluation lies a crucial difference in how we treat the interaction between our autonomous vehicle (the "ego" vehicle) and its environment.
- Open-Loop Evaluation asks: Given a recorded scenario, what would the planner have done? The planner's output is compared to a ground-truth trajectory (usually from a human driver), but the world does not react to the planner's decisions. It's an offline, non-interactive analysis.
- Closed-Loop Evaluation asks: When the planner acts, what happens next? The planner is placed "in the loop" of a simulation. Its actions are executed, the state of the world is updated in response, and the planner then makes a new decision based on this new state. It is an online, interactive simulation.
The choice between these paradigms depends on what you are trying to measure. This next video provides a clear explanation of why different components of an AV stack require different evaluation methods.
Safety in the Loop: Advancing Autonomous Vehicle Safety Validation Through Simulation
In this segment from a talk by NVIDIA researchers, Apurv Tiwari explains why planning, unlike perception, fundamentally requires closed-loop evaluation.
Watch from 10:48 to 13:59. Focus on the core argument: why is the performance of a perception module 'insensitive' to the vehicle's actions, and why is a planner's performance highly sensitive to its own actions?
As the video explains, the key is the feedback loop. A planner's decision to accelerate, brake, or change lanes directly influences the future positions of other agents, which in turn must inform the planner's next decision. Open-loop evaluation breaks this feedback loop.
Key Characteristics
| Aspect | Open-Loop Evaluation | Closed-Loop Evaluation |
|---|---|---|
| Mechanism | Compares a predicted trajectory against a static, recorded log. | Executes the planner's actions within an interactive simulator. |
| Interaction | Non-interactive. Other agents replay their logged paths, ignoring the ego vehicle. | Interactive. Other agents react to the ego vehicle's simulated movements. |
| Pros | Fast, computationally cheap, perfectly repeatable, easy to scale across massive datasets. | More realistic, captures emergent behaviors and interaction failures. |
| Cons | Can be misleading. A plan might seem safe but would have caused a crash if others had reacted. | Computationally expensive, requires a high-fidelity reactive simulator, can suffer from a "sim-to-real" gap. |
Now, let's examine how these concepts are realized in a leading benchmark.
2. Case Study: nuPlan and its Metrics
The nuPlan benchmark is a large-scale, industry-standard platform designed specifically for evaluating motion planners. It was created to move beyond simple open-loop metrics and enable rigorous closed-loop testing.
First, let's understand how nuPlan's simulation works and how it defines its metrics.
NuPlan: Autonomous Driving Closed-Loop Benchmark
The documentation for the nuPlan benchmark provides a concise overview of its evaluation philosophy and a clear definition of its open-loop and closed-loop scores.
Please read the introduction and sections '2. Closed-Loop Simulation and Reactive Agent Modeling' and '3. Evaluation Metrics and Scoring Protocols'. Pay close attention to the explicit definitions of 'Open-Loop Score (OLS)' and 'Closed-Loop Scores (CLS)' and the sub-metrics that comprise each.
As you read, you'll see a direct mapping to the concepts we've discussed:
- Open-Loop Score (OLS): This score is primarily used to evaluate imitation learning models. It measures how well the planner's predicted trajectory matches the one recorded by the expert human driver.
- Typical Metrics: Average Displacement Error (ADE) and Final Displacement Error (FDE). These measure the geometric distance between the planned and logged paths.
- Closed-Loop Scores (CLS): This is a composite score aggregated from running the planner in the interactive simulator. It directly measures the outcomes of the planner's actions.
- Typical Metrics:
- Safety: Collision with objects, driving off-road.
- Progress: Making progress towards the destination.
- Comfort: Staying within acceleration and jerk limits.
- Notice how these metrics are identical in spirit to the rule-based reward functions we designed for Plan-R1. This is no coincidence; evaluation metrics and alignment rewards are two sides of the same coin.
- Typical Metrics:
nuPlan makes it clear that while OLS is useful, only CLS can truly assess the safety and feasibility of a planner in dynamic traffic. However, running millions of miles in a high-fidelity, reactive, closed-loop simulation is incredibly expensive. This has led to research into more efficient evaluation paradigms.
3. Case Study: NAVSIM's Non-Reactive Middle Ground
What if we could get the safety insights of closed-loop evaluation with the scalability of open-loop? This is the goal of NAVSIM, a benchmark that proposes a clever "middle ground."
NAVSIM introduces the concept of non-reactive simulation.
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation
The NAVSIM paper introduces a novel evaluation paradigm. These sections explain the motivation and mechanics of its non-reactive simulation.
First, read the Abstract to understand the high-level motivation. Then, read 'Section 3: NAVSIM: Non-Reactive Autonomous Vehicle Simulation' to grasp the core mechanism. Finally, examine 'Section 4.1: Alignment Between Open-Loop and Closed-Loop Evaluation' and focus on Figure 3, which shows the correlation results.
Here's the core idea of NAVSIM:
- Single Plan: For each test scenario, the planner is queried only once at the beginning to produce a trajectory for a short horizon (e.g., 4 seconds).
- Non-Reactive Unroll: This fixed trajectory is then "unrolled" in a simple simulator where all other agents just replay their logged paths. The other agents do not react to the ego vehicle's plan.
- Simulation-Based Metrics: Even without reaction, the simulator can still detect critical events. Did the planned trajectory intersect with a log-replayed vehicle? Did it leave the drivable area? Metrics like collision, off-road, and progress are computed.
This approach cleverly combines the best of both worlds:
- It's scalable like open-loop because it only requires one forward pass of the planner per scenario and a simple, non-reactive unroll.
- It provides meaningful safety metrics like closed-loop because it's based on simulation outcomes (e.g., collision), not just geometric deviation from a log (ADE/FDE).
As Figure 3 in the paper shows, NAVSIM's aggregated metric (PDMS) correlates much more strongly with true closed-loop performance than the traditional Open-Loop Score (OLS) does. This makes it a more reliable and efficient proxy for real-world performance.
Conclusion
In this lesson, we dissected the critical distinction between open-loop and closed-loop evaluation for autonomous driving planners. Understanding this difference is essential for correctly interpreting benchmark results and appreciating the trade-offs between evaluation fidelity and computational cost.
Key Takeaways:
- Open-Loop Evaluation is a non-interactive, offline check against a logged trajectory. It is scalable but can be misleading as it doesn't account for how the world reacts. Its primary metrics are geometric errors like ADE and FDE.
- Closed-Loop Evaluation is an interactive simulation where the planner's actions influence the environment, creating a feedback loop. It provides a more realistic assessment of safety and is evaluated on outcome-based metrics like collision rate and progress.
- nuPlan is a premier closed-loop benchmark that formalizes these two approaches with its Open-Loop Score (OLS) and Closed-Loop Score (CLS), demonstrating that both are used in a complete evaluation protocol.
- NAVSIM offers a pragmatic "middle ground" with its non-reactive simulation. It computes closed-loop-style safety metrics in a scalable, open-loop manner, providing a more reliable proxy for real-world performance than traditional open-loop methods.
Preview of the Final Module:
With this lesson, you have now covered all the core technical components of the course, from Transformer architectures and alignment algorithms to planning frameworks and evaluation methodologies. In our final sessions, we will synthesize this knowledge to analyze the end-to-end lifecycle of a generative planner and discuss the future research directions on the path toward embodied Artificial General Intelligence.