Understanding the GRPO Algorithm

The GRPO Algorithm: A Step-by-Step Walkthrough

The GRPO algorithm can be broken down into a clear, iterative process. For each prompt in our training data, we perform the following four steps to calculate a loss and update our model. The diagram below provides a high-level visual map of this entire workflow, which we will dissect piece by piece.

Step 1: Group Sampling

The first step is to generate a diverse set of possible outputs for a given input query or prompt.

For a single query $q$, we use our current policy model, ${\pi }_{{\theta }_{old}}$, to generate a "group" of $G$ different output sequences. For example, the DeepSeek-R1 paper uses a group size of $G=64$. This is done using standard autoregressive sampling, but instead of taking just the single most likely output, we generate multiple varied completions.

This step is fundamental because it creates the "group" that will be used for the relative comparison in the subsequent steps. Without multiple outputs, there's nothing to compare.

Step 2: Reward Computation

Once we have our group of $G$ outputs, we need to evaluate how "good" each one is. This is done using a reward function, which assigns a scalar score, $r_i$, to each output $o_i$.

For reasoning tasks like mathematics or coding, this reward function doesn't need to be a complex, learned model. Instead, it can be a simple, rule-based system. For instance, your course notes mention:

• Math: Does the final answer in the model's output match the correct answer? (e.g., reward = 1 if correct, 0 if incorrect).
• Code: Does the generated code compile and pass a set of predefined unit tests?

This provides an objective, verifiable signal of quality for each response in the group.

Step 3: Advantage Normalization

This is the step that replaces the critic network. In the previous lesson, we learned that the advantage tells us how much better a particular action (or in this case, a whole output sequence) was compared to a baseline.

GRPO calculates this baseline empirically from the group of rewards we just computed. The process is:

1. Compute Group Statistics:

   • Calculate the mean reward of the group: $\bar{r}=\frac{1}{G}{\sum }_{i=1}^G{r}_i$
   • Calculate the standard deviation of the rewards: ${\sigma }_r=\sqrt{\frac{1}{G}{\sum }_{i=1}^G(r_i-\bar{r}{)}^2}$

2. Calculate the Advantage: For each output $o_i$, its advantage $A_i$ is its reward, normalized by the group's statistics:

   $$A_i=\frac{r_i-\bar{r}}{{\sigma }_r+ϵ}$$

   Here, $ϵ$ is a small constant to prevent division by zero.

This simple formula effectively tells us how many standard deviations better or worse a given response was than the group average. A positive advantage means the response was better than average and should be encouraged, while a negative advantage means it was worse and should be suppressed.

To see this process explained with clear visuals, please watch the following video segment.

Step 4: Clipped Policy Update

Now that we have an advantage score $A_i$ for each generated output, we can update the policy parameters $\theta$. GRPO uses an objective function that is nearly identical to PPO's, ensuring stable training updates.

The overall objective function to be maximized is:

This might look complex, but let's break down the two main components:

1. The Clipped Surrogate Objective:

   • The first part, min(...), is the core of PPO's update rule.
   • The probability ratio, $\text{ratio}(\theta )=\frac{{\pi }_{\theta }(o|q)}{{\pi }_{{\theta }_{old}}(o|q)}$, measures how much more (or less) likely the new policy ${\pi }_{\theta }$ is to produce a given output $o$ compared to the policy before the update, ${\pi }_{{\theta }_{old}}$.
   • The clipping function, clip(ratio, 1-ε, 1+ε), limits this ratio to a small window around 1.0. This is a crucial stabilization technique that prevents the policy from changing too drastically in a single step, which could lead to a collapse in performance. By taking the min of the unclipped and clipped objectives, we only apply the update-limiting "brake" when the policy is improving (positive advantage).

2. The KL Divergence Penalty:

   • The second term, $-\beta D_{KL}({\pi }_{\theta }||{\pi }_{ref})$, is an additional regularizer.
   • $D_{KL}$ is the Kullback-Leibler divergence, which measures how much the new policy ${\pi }_{\theta }$ has diverged from an original reference policy ${\pi }_{ref}$ (e.g., the model state before GRPO training began).
   • This term acts as a "leash," ensuring the model doesn't stray too far from its original, stable behavior while it learns new reasoning skills. The hyperparameter $\beta$ controls how tight this leash is.

The following resources provide excellent breakdowns of this final, crucial step.

Putting It All Together: The GRPO Loop

We have now examined each individual step of the GRPO algorithm. When put together, they form an iterative training loop where the model's policy is progressively refined.

The pseudocode in the image below offers a concise summary of this entire process, clearly outlining the initialization and the update loop.

As you can see in Algorithm 1, the process involves initializing the policy, then repeatedly sampling batches, generating groups of outputs, computing rewards and advantages, and finally updating the policy by maximizing the objective function we just discussed. This loop continues until the model's reasoning performance converges.

Conclusion

In this lesson, we have deconstructed the complete Group Relative Policy Optimization algorithm. We moved beyond the high-level concept of "critic elimination" and detailed the precise mechanics that make GRPO an effective and efficient method for reasoning alignment.

Key Takeaways:

• Group Sampling: The process starts by generating a group of $G$ diverse outputs for a single prompt using the current policy.
• Reward Computation: A rule-based or model-based reward function assigns a scalar quality score to each output in the group.
• Advantage Normalization: The advantage for each output is calculated by normalizing its reward against the group's mean and standard deviation. This serves as the critic-free baseline.
• Clipped Policy Update: The policy is updated using a PPO-style clipped surrogate objective, which uses the advantage to guide the update while ensuring stability. A KL divergence penalty further regularizes the policy, preventing it from deviating too far from a trusted reference model.

Preview of the Next Lesson:

Now that we have a solid grasp of the GRPO algorithm itself, our next lesson will investigate its powerful effects. We will explore how applying this algorithm leads to the emergence of complex reasoning abilities in models. We will examine the DeepSeek-R1 training pipeline in detail, focusing on concepts like the "cold start" phase for initializing structured reasoning, and how behaviors like self-verification and backtracking develop during the GRPO training process.