Rule-Based Rewards for Verifiable Tasks

The Challenge: Defining a "Good" Response

In reinforcement learning, the central challenge is to provide a numerical reward signal that accurately reflects the quality of a model's output.

For open-ended, subjective tasks like writing a poem or summarizing a document, there is no single "correct" answer. To train a model on such tasks, we typically rely on Reinforcement Learning from Human Feedback (RLHF). This involves collecting vast amounts of data where humans have compared different model outputs and indicated their preferences. A separate reward model is then trained to mimic this human judgment. This process is slow, expensive, and the resulting reward model is only an approximation of true quality.

But what if a task has a clear, objective, right-or-wrong answer? This is where a more direct and powerful approach becomes possible.

Verifiable Correctness and Rule-Based Rewards

A special class of problems, which includes mathematics, logic, and code generation, exhibits verifiable correctness. This means two conditions are met:

1. An objective ground truth exists (e.g., the solution to a math problem, the expected output of a program).
2. A cheap, automated program, called a verifier, can deterministically check if a model's output matches the ground truth.

This allows us to replace the complex, learned reward model with a simple, programmatic rule-based reward function. Instead of estimating what a human might prefer, the verifier provides a factual, binary reward: 1 if the answer is correct, 0 if it is not.

Let's begin by exploring this paradigm, often called Reinforcement Learning with Verifiable Rewards (RLVR).

As the article highlights, RLVR works beautifully for math and code precisely because a verifier can provide an unambiguous reward signal. Let's see how this is applied in practice.

While a binary reward is the simplest form, these rule-based functions can also incorporate other heuristics, such as checking for correct output formatting.

The Perfect Match: Why GRPO Thrives on Verifiable Tasks

Now we can address the core question: why is GRPO the perfect algorithm for this setup? The synergy arises from two key aspects.

1. Eliminating the Critic Network

In traditional policy gradient methods like PPO, determining how to update the policy is difficult. The algorithm needs to know if the reward received for an action was better or worse than expected. This "expectation" is provided by a critic (or value) network, which is trained to estimate the expected future reward ($V(s)$) from any given state. Training this critic is notoriously difficult, computationally expensive (it's often as large as the policy model itself), and a major source of instability.

With a rule-based verifier, the reward signal is not an estimate; it is a deterministic fact. We know the ground-truth value of a completed output with certainty.

GRPO is designed to exploit this. It completely eliminates the critic network. Instead of comparing the received reward to a learned value estimate, it compares the rewards of several different outputs generated for the same prompt. The "advantage" is simply how much better or worse a given output's reward is compared to the average reward of the group.

This is the central reason why GRPO is so well-suited for verifiable tasks: the verifier provides a perfect reward signal, which allows GRPO to use a simple, stable, and efficient group-relative comparison instead of a complex and unstable critic network.

2. Handling Sparse Rewards

While the reward from a verifier is perfect, it is also sparse. For a long, multi-step math problem, the model only receives a single reward signal at the very end. It gets no feedback on which of its intermediate reasoning steps were correct or incorrect. This is a classic credit assignment problem in reinforcement learning. If the final answer is wrong, should the model reduce the probability of the entire reasoning chain, even if 99% of it was correct?

This is where the "noisy feedback" nature of RL becomes a challenge, and it's why the conservative update mechanisms in the GRPO objective function (which you studied in the last lesson) are so critical.

As the video explains, blindly penalizing an entire sequence for one small mistake can be very harmful. GRPO's design—combining group-relative advantage calculation with the clipped surrogate objective and a KL penalty—is tailored for this exact scenario. It allows the model to gently explore different reasoning paths to find those that lead to a final correct answer, without being catastrophically derailed by a single mistake in an otherwise promising attempt.

Conclusion

In this lesson, we've connected the mechanics of GRPO to the nature of the tasks it excels at. You can now explain the profound synergy between the algorithm and the problem domain.

Key Takeaways:

• Tasks with verifiable correctness (e.g., math, code) allow for cheap, deterministic rule-based reward functions (verifiers) that provide a perfect, factual reward signal.
• This perfect reward signal eliminates the need for a critic/value model, which is the most complex, expensive, and unstable component of traditional RLHF algorithms like PPO.
• GRPO is specifically designed to operate without a critic, using efficient group-relative reward comparisons instead. This makes it an ideal algorithmic match for verifiable tasks.
• The reward signal in these tasks is often sparse (all-or-nothing at the end). GRPO's conservative update mechanisms (clipping and KL divergence) are well-suited to learning effectively and stably from this type of sparse but accurate feedback.

Preview of the Next Lesson:

Now that we understand the powerful combination of GRPO and rule-based rewards, we'll examine how this is put into practice in a full training pipeline. In the next lesson, we will explore the DeepSeek-R1 "cold start" phase, where the model is first fine-tuned on chain-of-thought data to initialize the capacity for structured reasoning, preparing it for the main GRPO training stage.