Introduction
Welcome back. In our last lesson, we explored how the pure reinforcement learning phase of DeepSeek-R1-Zero gives rise to emergent reasoning behaviors like self-verification. We ended with a model that is a powerful, specialized reasoner but is also unpolished—its outputs can be hard to read, mix languages, and lack general helpfulness.
This raises a critical question: How do you take this raw, powerful reasoning ability and distill it into a robust, user-friendly model? The answer lies in the third major stage of the DeepSeek-R1 training pipeline.
Our learning outcome for this lesson is to describe the rejection sampling stage of the DeepSeek-R1 pipeline and its role in creating a high-quality SFT dataset. We will see how the specialized R1-Zero model is cleverly repurposed as a data generation engine to create a massive, clean, and diverse dataset that forms the foundation for the final, polished model.
From Specialist to Generalist: The Need for Curation
The R1-Zero model, trained with GRPO, is a reasoning specialist. However, its unconstrained RL training produced several undesirable side effects:
- Poor Readability: Outputs often lack clear formatting (e.g., Markdown) and can be chaotic.
- Language Mixing: The model might mix languages like English and Chinese within a single response.
- Narrow Focus: Its skills are concentrated on reasoning tasks, with weaker performance in general areas like writing, role-playing, or factual question-answering.
To create a truly useful model, the goal is to transfer R1-Zero's reasoning prowess into a model that performs well across general tasks and communicates clearly. This requires a new dataset—one that embodies high-quality reasoning and adheres to human preferences for readability and helpfulness. The process for creating this dataset is Rejection Sampling.
The Rejection Sampling Process
Rejection sampling is a simple yet powerful technique for quality control. Instead of using every output the model generates, you generate many and "reject" the ones that don't meet a certain standard. In the context of DeepSeek-R1, this works in three main steps: Generation, Filtering, and Augmentation.
1. Generation: Using the Expert to Create Data
The process begins by using the highly capable R1-Zero model as a data factory. For a given prompt (e.g., a math problem), the model is prompted to generate multiple different reasoning paths and answers. This creates a large pool of potential training examples.
2. Filtering: The "Rejection" Step
This is the core of the process. Each generated response is evaluated against a set of criteria, and only the "best" ones are kept. The rest are rejected. This quality-control filter is crucial.
Let's watch a short video that provides a clear overview of this filtering process.
Understanding Reasoning LLMs (o1/o3, DeepSeek-R1, Gemini Thinking, Grok 3, Claude 3.7)
The video 'Understanding Reasoning LLMs' by Donato Capitella gives a concise explanation of the rejection sampling stage.
Please watch from 14:24 to 15:07. Pay close attention to the criteria used for selecting the 'highest quality outputs'.
As the video and the source paper explain, the filtering is based on two main factors:
- Correctness: The final answer must be correct. This can be verified using the same rule-based rewards from the previous RL stage (e.g., does the code compile? is the math answer right?). For more complex problems without a simple rule, the DeepSeek team even used a powerful generative model (DeepSeek-V3) as a judge.
- Readability: The chain-of-thought itself must be well-structured and easy for a human to read. Any responses with mixed languages, long chaotic paragraphs, or poorly formatted code were filtered out.
By sampling multiple responses and retaining only those that are both correct and clean, the DeepSeek team curated a high-quality dataset of about 600,000 reasoning-related training samples.
3. Augmentation: Adding General-Purpose Data
The goal is not just a better reasoner, but a better all-around model. To achieve this, the curated reasoning dataset is augmented with data for non-reasoning tasks.
- This includes examples of writing, factual question-answering, role-playing, and translation.
- The DeepSeek team collected approximately 200,000 such samples, reusing some data from the development of their base DeepSeek-V3 model.
This results in a final, diverse Supervised Fine-Tuning (SFT) dataset of about 800,000 samples.
The Role of the SFT Dataset
To understand the specific details of this stage, let's turn to the original research paper.
DeepSeek-R1: Incentivizing Reasoning in LLMs
The DeepSeek-R1 paper provides the most authoritative description of this pipeline stage. Reading it will clarify the nuances of the data collection and filtering process.
Please read Section 2.3.3, 'Rejection Sampling and Supervised Fine-Tuning'. Focus on the 'Reasoning data' and 'Non-Reasoning data' subsections to understand what was generated, what was filtered, and what was added to create the final 800k sample dataset.
The reading confirms the primary role of this stage: to create a high-quality dataset for Supervised Fine-Tuning (SFT).
Here’s the crucial part of the strategy: this newly created 800k-sample dataset is not used to further train R1-Zero. Instead, the team goes back to the original DeepSeek-V3-Base model and fine-tunes it on this new, high-quality data.
This process effectively "distills" the advanced reasoning capabilities discovered by R1-Zero into a fresh model. The new model learns the effective reasoning patterns from the curated data but does so within the structured, clean format enforced by the rejection sampling filter. The result is a model (called "DeepSeek-R1 Dev3" in the follow-up Nature paper) that has strong reasoning skills and is a good instruction-follower.
Conclusion
In this lesson, we've examined the pivotal rejection sampling stage of the DeepSeek-R1 pipeline, which bridges the gap between a raw reasoning expert and a polished, general-purpose model.
Key Takeaways:
- Purpose: The rejection sampling stage addresses the poor readability and narrow focus of the pure RL model (R1-Zero).
- Process: It uses the R1-Zero model to generate a large pool of reasoning examples, then filters them, keeping only the responses that are both correct and well-formatted.
- Dataset Creation: This process curates a high-quality SFT dataset, which is augmented with non-reasoning data to ensure the final model is a strong generalist.
- Role: The resulting dataset is used to fine-tune the original base model, effectively "distilling" the reasoning skills of R1-Zero into a new, more robust, and user-friendly model.
Preview of the Next Lesson:
We have now followed the pipeline to produce a model that is both a skilled reasoner and a good instruction-follower. But there is one final step. In our next lesson, we will cover the fourth and final stage: "Reinforcement Learning for all Scenarios." We will explore how another round of RL, this time using reward models, is used to further align the model for helpfulness and harmlessness, ensuring it is safe and reliable for real-world use.