GRPO

Revisit the PPO clipped surrogate objective and its role in stabilizing policy updates.

Explain the function of the advantage estimator and value-function baseline in PPO for variance reduction.

Outline the standard RLHF pipeline to contextualize the role of the RL fine-tuning stage.

Explain the purpose of the KL-divergence penalty for regularizing the policy during RL fine-tuning.

Explain the motivation for GRPO's critic-free approach by contrasting it with PPO.

Derive the group relative advantage estimator using a batch of sampled model generations and their associated rewards.

Construct the full GRPO objective function, interpreting the roles of the clipped advantage, group normalization, and KL penalty.

Implement a Python function to compute the group relative advantage from a batch of rewards.

Implement the complete GRPO loss function in Python, incorporating the clipping mechanism and KL penalty.

Assemble a training loop to fine-tune a language model on a dummy dataset using the GRPO loss.