Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
👋
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Generative AI for Autonomous Driving
·
Module 8
Post-Training Alignment: SFT, PPO, and DPO
1
Supervised Fine-Tuning (SFT) for Instruction Following
Explain supervised fine-tuning (SFT) and its role in teaching a base model to follow instructions
2
Training Reward Models from Human Preferences
Describe how reward models are trained from pairwise human preference comparisons
3
The RLHF Pipeline: SFT, Reward Models, and PPO
Outline the standard RLHF pipeline: SFT, reward model training, and policy optimization
4
PPO Clipped Surrogate Objective and Stable Policy Updates
Describe the PPO clipped surrogate objective and why clipping stabilizes policy updates
5
KL Divergence as a Regularizer
Explain the KL divergence penalty that prevents the policy from diverging too far from the reference model
6
PPO-Based RLHF: The Four Essential Models
Identify the four models required in PPO-based RLHF: actor, critic, reward model, and reference model
7
Direct Preference Optimization (DPO): Simplified Alignment
Describe Direct Preference Optimization (DPO) and how it eliminates the need for a separate reward model
Previous module
Reinforcement Learning Foundations
Next module
GRPO and Emergent Reasoning