Crafting Rule-Based Reward Functions for Autonomous Driving

Designing the Lane Compliance Reward

A key aspect of good driving is staying properly positioned within a lane. In our last lesson, we saw that Plan-R1 implements "Driving area compliance" as a hard safety indicator. Let's quickly review that.

This rule—staying within the drivable area—is a hard constraint. If the car drives off the road, the trajectory is an immediate failure, and the total reward is zero.

However, we can introduce a more nuanced, "soft" reward to encourage the car to stay centered within its lane, not just on the road. This reflects a preference for good driving style rather than a strict safety necessity.

A simple and effective way to implement this is with a continuous penalty that increases as the vehicle deviates from the lane's centerline.

As the paper describes, the Centering Reward can be formulated as a linear penalty:

Let's analyze this function:

• deviation_dist is the vehicle's lateral distance from the center of its current lane.
• c_centering is a positive coefficient (a weight) that we can tune.
• The reward is negative and proportional to the distance from the center. A larger deviation results in a larger penalty.
• If the vehicle is perfectly centered (deviation_dist = 0), this reward component is 0, contributing no penalty.

This is a classic "soft" cost. It doesn't automatically zero out the entire reward. Instead, it contributes a negative value to the weighted sum, gently nudging the policy towards better lane discipline.

Designing the Speed Limit Reward

Adhering to the speed limit is another crucial aspect of rule-compliant driving. Like lane centering, this is typically handled as a soft cost. A small deviation might be acceptable, but significant or prolonged speeding should be penalized.

The Plan-R1 paper proposes an intuitive method for this.

The Plan-R1 approach is to calculate a score, $r_{\text{speed}}$, that starts at 1 (no penalty) and decreases linearly to 0 as the total "over-speed distance" increases.

Here's how you could implement the logic for this score:

1. Calculate Over-Speed Distance: For each segment of the planned trajectory, if the vehicle's speed is above the limit, calculate the distance it travels during that segment. Sum these distances to get the total_over_speed_distance.
2. Define a Threshold: Set a max_over_speed_threshold (e.g., 50 meters). This represents the maximum distance of speeding you are willing to tolerate before the reward score drops to zero.
3. Compute the Ratio: Calculate the ratio total_over_speed_distance / max_over_speed_threshold.
4. Compute the Score: The final score is $r_{\text{speed}}=1-\min (1,\text{ratio})$.
   • If there is no speeding, the ratio is 0, and the score is 1.
   • If the over-speed distance equals or exceeds the threshold, the ratio is $\ge 1$, and the score becomes 0.
   • For intermediate values, the score decreases linearly.

This formulation is effective because it accounts for both the magnitude and duration of speeding. A brief moment of high speeding or a long period of slight speeding will both contribute to the total_over_speed_distance and be penalized.

Alternative: A Quadratic Penalty

As a point of comparison, the "Responsibility-Oriented Reward Design" paper you read earlier proposes a different formulation:

This quadratic penalty is harsher. Doubling the amount you are over the speed limit quadruples the penalty. This can be useful if you want to strongly discourage any significant speeding. The choice between linear and quadratic penalties is a design decision that depends on how strictly you want to enforce the rule.

The Complete Picture: Integrating Soft Costs

Now, let's place our newly designed functions for lane and speed compliance back into the full Plan-R1 reward structure:

Here's how the system evaluates a planned trajectory:

1. Safety Gate Check: The system first checks all hard safety indicators. Does the trajectory result in a collision? Does it leave the drivable area?
   • If YES (any indicator is 0), the entire reward $R(y_t)$ is immediately 0. The soft costs are irrelevant. This enforces safety as the absolute top priority.
2. Soft Cost Calculation:
   • If NO (all safety indicators are 1), the system proceeds to calculate the soft costs.
   • It computes the penalty for lane deviation ($r_{\text{center}}$) and the penalty for speeding ($r_{\text{speed}}$), along with other soft costs like comfort and progress.
   • These scores are multiplied by their respective weights ($w_j$) and summed up. This sum becomes the final reward.

This two-part structure is elegant and effective. It allows the model to first learn the non-negotiable rules of the road and then fine-tune its behavior to be more efficient, legal, and pleasant, all guided by simple, verifiable code.

Conclusion

In this lesson, we've extended our understanding of rule-based reward design to cover "soft" costs that guide the model's driving style.

Key Takeaways:

• Lane compliance can be treated as both a hard constraint (staying on the road) and a soft cost (staying centered). A soft penalty can be a linear function of the distance from the lane centerline.
• Speed limit adherence is typically a soft cost. A common approach is a linearly decreasing reward score based on the ratio of "over-speed distance" to a maximum threshold.
• These soft rewards are calculated and summed only if the trajectory first passes the multiplicative safety gate of hard constraints (like collision avoidance).
• The use of weights ($w_j$) allows engineers to tune the trade-offs between different driving objectives, such as staying perfectly centered versus making faster progress.

Preview of the Next Lesson:

We have now designed reward functions for hard safety and for legal compliance (lanes and speed). In the next lesson, we will complete our tour of the Plan-R1 reward system by learning how to design the final two key soft costs: encouraging progress towards the goal and ensuring passenger comfort by minimizing jerk.