Reward Function Design for Safe and Comfortable Navigation

1. Designing the Progress Reward

A vehicle that is perfectly safe and law-abiding but never moves is not a useful planner. The progress reward, also known as the efficiency objective, is what incentivizes the agent to make headway towards its destination.

However, designing this reward requires care. A naive reward for simply moving could encourage the agent to drive in circles or in the wrong direction. Plan-R1 uses a more intelligent approach.

The core idea in Plan-R1 is to reward forward progress along the intended route. Here's the logic:

1. Project: For each point in the agent's predicted trajectory, find the closest point on the centerline of the reference route.
2. Accumulate: Sum the distance traveled forward along this projected path on the centerline.
3. Normalize: The final score is normalized using the total progress made by the original expert (human) driver in the dataset.

This is a crucial design choice. The reward function does not penalize the agent for deviating laterally from the expert's path (e.g., to change lanes to overtake). It only cares that the agent is making efficient progress toward the goal. This encourages effective driving, not just blind imitation.

Alternative Formulations for Progress

The Plan-R1 method is one of many ways to define progress. A broader survey of techniques can provide more context on the trade-offs involved.

As the review highlights, there are several families of progress rewards:

• Sparse Reward: A large positive reward is given only upon reaching the final goal. This is simple but can make learning very difficult, as the agent gets no feedback until the very end.
• Dense Velocity Reward: The agent is rewarded at each timestep based on its forward velocity, often clipped at the speed limit: $r_{\text{progress}}=\min (v,v_{\text{limit}})$. This provides constant feedback but can reward moving even if it's in the wrong direction.
• Distance-to-Goal Reward: The reward is based on the change in Euclidean distance to the goal. This solves the "wrong direction" problem but fails on roads, where the path is not a straight line.
• On-Route Progress Reward: This is the category Plan-R1's method falls into. It measures distance traveled along the planned route, providing a robust and meaningful signal.

The review also raises a critical point: a simple progress reward can conflict with safety. An agent might choose to take a penalty for a collision if it's smaller than the cumulative penalty for waiting, which is an unacceptable outcome. This is precisely why Plan-R1's multiplicative safety gate is so important—it makes safety an absolute prerequisite for any other objective.

2. Designing the Comfort Reward

A car that gets you to your destination quickly but does so with violent, jerky movements is not well-designed. The comfort objective aims to produce smooth trajectories that are pleasant for passengers.

In physics and engineering, the key to a smooth ride is managing not just acceleration, but its rate of change, known as jerk.

• Position: $P(t)$
• Velocity: $V(t)=\frac{dP(t)}{dt}$ (Rate of change of position)
• Acceleration: $A(t)=\frac{dV(t)}{dt}$ (Rate of change of velocity)
• Jerk: $J(t)=\frac{dA(t)}{dt}$ (Rate of change of acceleration)

Sudden changes in acceleration create a high jerk, which we feel as a "jolt". Therefore, a primary goal of a comfort reward is to minimize jerk.

Now, let's see how Plan-R1 implements this.

Plan-R1's comfort reward is a binary score. It defines thresholds for:

• Lateral acceleration
• Longitudinal acceleration
• Angular velocity
• Angular acceleration

If the predicted trajectory exceeds any of these thresholds at any point, the comfort score for the whole trajectory becomes 0. If all values stay within the thresholds, the score is 1.

This is a simple but effective approach. It doesn't provide a graded signal but instead defines a hard boundary for what is considered a "comfortable" maneuver.

Alternative Formulations for Comfort

The binary threshold is not the only way to reward comfort. A more continuous penalty can also be effective.

The review paper shows that comfort can be encouraged by directly penalizing the quantities that cause discomfort. A common formulation is a quadratic penalty on jerk:

This type of function heavily penalizes large, sudden changes in acceleration, while being more tolerant of small, gradual adjustments. Similar penalties can be applied to acceleration or sharp steering angles. The choice between a binary score (like Plan-R1) and a continuous penalty depends on the specific design goals.

3. The Complete Picture: Balancing Objectives

We have now defined all the major components of the Plan-R1 reward function. Let's put it all together.

The final reward is a combination of hard safety gates and a weighted sum of soft costs:

The weights ($w_j$) are critical hyperparameters. They allow engineers to define the desired driving style by balancing competing objectives:

• High $w_{\text{prog}}$: The agent will be more "aggressive," prioritizing reaching the goal quickly, potentially at the cost of some comfort.
• High $w_{\text{comfort}}$: The agent will be more "cautious," prioritizing a smooth ride, even if it means slower progress.
• High $w_{\text{speed}}$, $w_{\text{lane}}$: The agent will be a strict rule-follower.

Tuning these weights is a key part of the alignment process, shaping the final behavior of the autonomous vehicle.

Conclusion

In this lesson, you've learned to design the final, crucial components of a rule-based reward system for autonomous driving.

Key Takeaways:

• Progress Reward: This incentivizes efficiency. A robust method is to measure the vehicle's forward progress along the centerline of its reference route, normalized by a reference like an expert trajectory.
• Comfort Reward: This ensures a smooth ride for passengers. It is achieved by penalizing undesirable kinematics, most notably high jerk (the rate of change of acceleration), as well as high acceleration itself.
• Balancing Act: These soft costs are summed and multiplied by weights ($w_j$), which are tuned to create the desired driving "personality" by trading off between objectives like speed and smoothness.
• Safety First: Crucially, these soft rewards are only considered after a trajectory has passed the non-negotiable multiplicative safety gate.

Preview of the Next Lesson:

We have now fully deconstructed the reward function used in Plan-R1. However, the paper introduces one final, critical modification to the GRPO algorithm itself, called Variance-Decoupled GRPO (VD-GRPO). In the next lesson, we will explore why standard GRPO can be problematic in safety-critical tasks and how VD-GRPO solves this by preventing the model from reinforcing the "best of a bad bunch" of unsafe actions.