Rule-Based Reward Functions for Safe Reinforcement Learning

Why Rule-Based Rewards? The Philosophy of Verifiable Safety

Before we write any equations, it's important to understand why we use explicit, coded rules for safety instead of, for example, training another neural network to predict a "safety score."

This distinction is often framed as "process-oriented" versus "purpose-oriented" rewards. To understand this concept, please read the following short introduction.

For safety-critical tasks like driving, process-oriented rewards are essential. We cannot just reward the model for arriving at the destination; we must penalize it for the unsafe processes it might take to get there. Using hard-coded rules provides three main advantages:

1. Verifiability: A rule like if distance_to_obstacle < 0: reward = -1 is unambiguous and auditable. We can prove it works. A learned reward model is a black box that could have unforeseen failure modes.
2. No Reward Hacking: The agent cannot find a clever loophole to get a high reward while still being unsafe. A collision is a collision; the rule is absolute.
3. Data Efficiency: We don't need to collect massive datasets of safe and unsafe scenarios to train a separate reward model. The "reward model" is simply the set of rules we implement in code.

Designing Hard Constraints: The Multiplicative Safety Gate

The most elegant way to enforce hard safety constraints is to structure the reward function so that any single violation nullifies the entire reward. Plan-R1 achieves this with a multiplicative formula.

The paper gives us the total reward formula:

Let's break this down:

• The first part, ${\prod }_{k\in I_{\text{safe}}}{1}_{k,t}$, is a product of binary safety indicators.
• Each $1_{k,t}$ represents a specific safety rule (like "no collision"). It is 1 if the rule is satisfied and 0 if it is violated.
• Because these indicators are multiplied, if even one of them is 0, the entire product becomes 0.
• This product acts as a safety gate. The second part of the equation (the weighted sum of "soft" costs like comfort and progress) is only considered if the safety gate is 1 (i.e., all hard safety constraints are met). If the gate is 0, the total reward is 0, no matter how "comfortable" or "efficient" the trajectory was.

This lesson focuses on designing the functions for $1_{k,t}$, the hard safety indicators.

Implementing the Collision Reward

Now, let's design the most important safety indicator: collision avoidance. How do we create the function for $1_{\text{collision},t}$?

The Plan-R1 paper provides a clear and robust definition in its appendix.

Based on the reading, we can define our collision-related safety indicators. The final collision indicator is a combination of several checks.

1. No Collision with Dynamic Agents
The ego vehicle's planned trajectory must not intersect with the predicted trajectories of other moving agents (cars, pedestrians, cyclists).

• Input: The ego vehicle's planned trajectory (a sequence of bounding boxes over time) and the predicted trajectories of all other agents.
• Logic: For each future timestep t, check if the ego vehicle's bounding box intersects with any other agent's bounding box.
• Output: The indicator $1_{\text{dynamic\_collision}}$ is 0 if any intersection is found at any timestep; otherwise, it is 1.

2. No Collision with Static Obstacles
The ego vehicle must not collide with any non-moving parts of the environment.

• Input: The ego vehicle's planned trajectory and the locations/bounding boxes of all static obstacles (e.g., parked cars, traffic cones, barriers).
• Logic: For each timestep t, check if the ego vehicle's bounding box intersects with any static obstacle's bounding box.
• Output: The indicator $1_{\text{static\_collision}}$ is 0 if any intersection is found; otherwise, it is 1.

3. Driving Area Compliance
This is another critical hard constraint. The vehicle must stay on the road.

• Input: The ego vehicle's planned trajectory and a polygon representing the drivable area.
• Logic: For each point in the planned trajectory, check if it lies inside the drivable area polygon.
• Output: The indicator $1_{\text{drivable\_area}}$ is 0 if any point is outside the polygon; otherwise, it is 1.

The final safety gate is the product of all these individual checks:

If the trajectory involves running a red light, this would be another binary indicator multiplied in. If the vehicle drives off a cliff, it has violated the "drivable area" rule. This multiplicative structure is incredibly powerful for enforcing a strict hierarchy of priorities.

The practical impact of these rules is immense. The paper's ablation study shows that removing the collision term causes the model's safety score to plummet, confirming it is the most critical component.

A More General Mathematical Framework

The binary 0/1 indicators used by Plan-R1 are a "hard" implementation of a safety rule. It's also possible to use "softer" functions that penalize the model as it approaches an unsafe state, rather than only when it crosses the line.

Another paper on reward function design provides a general mathematical framework for this.

This paper proposes that any reward function can be a product of evaluation functions $e_k(f)$, where each function maps some feature $f$ to a value between 0 and 1.

The Plan-R1 binary indicators are a simple step function. But let's consider Equation 9 from the paper for a feature like distance_to_nearest_car:

Here:

• $f$ is the distance to the nearest car.
• $\tau$ is a safe distance threshold (e.g., 5 meters).
• If the distance $f$ is greater than or equal to the safe threshold $\tau$, the reward component is 1 (full reward).
• If the distance $f$ drops below the threshold, the reward component decays exponentially towards 0.

This "soft" penalty encourages the model to maintain a safety margin, not just barely avoid a collision. While Plan-R1 uses hard binary indicators for its primary safety gate, this more general framework shows how you can design nuanced rewards for various aspects of the driving process, giving you a flexible and powerful toolkit.

Conclusion

In this lesson, we moved from the high-level concept of rule-based alignment to the concrete design of functions for enforcing hard safety constraints.

Key Takeaways:

• Hard safety constraints are best implemented with a multiplicative safety gate, where a set of binary indicators are multiplied together.
• If any single safety rule (e.g., collision, off-road) is violated, its indicator becomes 0, forcing the total reward to zero. This mathematically enforces safety as the top priority.
• A collision reward function is implemented by performing geometric checks for intersections between the ego vehicle's bounding box and the bounding boxes of other agents (dynamic) and obstacles (static) at each future timestep.
• This rule-based approach is verifiable, transparent, and avoids the ambiguity and potential for reward hacking associated with learned reward models.

Preview of the Next Lesson:

We have now mastered the design of the hard safety gate $\prod 1_{k,t}$. In our next lesson, we will focus on the other half of the reward equation: designing the "soft" cost terms $\sum w_j\cdot r_j(y_t)$. You will learn to design rule-based reward functions for lane compliance, adherence to speed limits, encouraging progress, and ensuring passenger comfort.