Introduction
Welcome back! In our last lesson, we established a crucial link: the GRPO algorithm is exceptionally well-suited for tasks with verifiable correctness, like math and code, because it thrives on the simple, rule-based rewards these tasks provide. This allows us to eliminate the need for a complex and unstable critic network, which is a major bottleneck in traditional RLHF.
However, this raises a new question. A base language model, pre-trained only to predict the next word, has no inherent concept of "reasoning," "thinking," or formatting its output in a structured way. If we were to apply GRPO directly, the model would generate mostly random, unstructured text, receive a reward of zero almost every time, and struggle to learn anything meaningful. This is known as the "cold start" problem in reinforcement learning for reasoning.
Today, we'll address this problem head-on. Our learning outcome is to describe the DeepSeek-R1 cold start phase: fine-tuning on chain-of-thought data to initialize structured reasoning. We will see how a preliminary Supervised Fine-Tuning (SFT) step acts as a "warm-up," teaching the model the basic format and style of reasoning, thereby preparing it for the much more powerful GRPO training phase that follows.
The Challenge of a "Cold Start"
Imagine trying to learn a complex skill, like writing a formal research paper, by submitting random essays and only being told "correct" or "incorrect" at the end. Without any examples of what a good paper even looks like, you would be lost. The chances of you randomly producing a correctly formatted paper with a thesis, body, and conclusion are practically zero.
This is precisely the challenge a base LLM faces when put directly into a reasoning-oriented RL loop. The model's initial policy is effectively random from a reasoning perspective. This leads to two major issues:
- Extreme Reward Sparsity: The model almost never stumbles upon a correct, multi-step reasoning process, so the feedback signal is almost always zero.
- Training Instability: The few times it might get a positive reward could be due to chance, leading to unstable and inefficient learning. The model doesn't have a good "scaffolding" to build upon.
To overcome this, we need to first show the model what "thinking" looks like.
The Solution: Supervised Fine-Tuning on Chain-of-Thought Data
The DeepSeek-R1 pipeline solves the cold start problem with an initial phase of Supervised Fine-Tuning (SFT). You can think of SFT as "learning by example." Instead of learning through trial and error (RL), the model is shown a set of high-quality examples and trained to imitate them.
The key is the kind of data used: Chain-of-Thought (CoT) data. This isn't just a list of questions and final answers. A CoT example includes the entire step-by-step reasoning process a model should follow to arrive at the answer.
The goal of this phase is not to make the model a perfect reasoner. Rather, it's to initialize the model with the fundamental capacity for structured thought. It teaches the model the desired format, such as using <think> and <answer> tags, and gives it a solid foundation for the subsequent RL phase.
The following flowchart shows the entire DeepSeek-R1 training process. Notice that our focus today, the "Cold Start" SFT, is the very first step.
Anatomy of Chain-of-Thought Data
So, what does this "chain-of-thought data" actually look like, and where does it come from? Creating a large dataset of high-quality, step-by-step reasoning examples is a significant challenge. The following video provides a brilliant, practical look at one way this can be done for mathematical problems.
DeepSeek R1 Coldstart: How to TRAIN a 1.5B Model to REASON
To understand what 'chain-of-thought data' is, let's watch a practical demonstration of how such data can be created for a math task. This video shows a two-step process: first, using a deterministic 'math compiler' to generate the correct step-by-step calculations, and second, using another LLM to add natural language explanations to these correct steps.
Please watch from 12:04 to 18:25. Focus on how the process ensures the mathematical correctness of the steps first, and only then uses a language model to generate the 'thinking' narrative. This separation of concerns is key to creating reliable, high-quality training data for the cold start phase.
As the video demonstrates, the resulting data provides a complete example that looks something like this:
User: Calculate 8895 + 631 / -659 + -952
Assistant:<think>
Problem: What is 8895 + 631 / -659 + -952?
Reasoning:
- Begin by addressing the division operation as per PEMDAS... 631 / -659 is approximately -0.9575.
- Next, substitute this value back into the expression... 8895 - 0.9575 - 952.
- Perform the subtractions from left to right...
</think>
<answer>
The result is 7942.0425.
</answer>
This is the kind of structured example the model learns to imitate during the SFT cold start phase.
The Purpose of the Cold Start
Now that we understand what the cold start phase is, let's solidify why it's so critical. Why not just start with RL and let the model figure it out? The DeepSeek team found that this initial SFT step provides two major benefits.
DeepSeek R1 Coldstart: How to TRAIN a 1.5B Model to REASON
This segment from the 'DeepSeek R1 Coldstart' video directly addresses why the researchers added this SFT step before the main RL training.
Watch from 09:35 to 11:38. Pay close attention to the two key technical motivations discussed: 1) accelerating the convergence of the RL process, and 2) improving the clarity and coherence of the model's generated chain of thought.
As the video and the paper highlight, the cold start phase is designed to "prevent the early unstable cold start phase of RL training." Beyond the technical necessity, there's also a product-focused motivation.
DeepSeek-R1 Paper - Appendix on Cold Start
The paper's appendix provides further insight, framing the cold start not just as a technical necessity but also as a choice to improve the final product's user experience.
Read the section 'B.3.2. DeepSeek-R1 Cold Start'. Note the 'product-driven' motivation: training the model to adopt a first-person perspective ('I' instead of 'we') to make the reasoning process more engaging for users and to ensure language consistency.
In essence, the cold start SFT phase serves a dual purpose:
- Technical: It provides a stable foundation for RL, accelerating training and ensuring the model generates coherent, structured output.
- Product: It aligns the model's output style with a more intuitive, first-person narrative that improves the end-user experience.
Conclusion
In this lesson, we have dissected the "cold start" phase of the DeepSeek-R1 training pipeline. You can now describe this critical first step and explain its importance in developing advanced reasoning models.
Key Takeaways:
- The "cold start" problem describes the instability and inefficiency of applying reinforcement learning directly to a base model for complex reasoning tasks.
- The solution is an initial Supervised Fine-Tuning (SFT) phase using a small but high-quality set of Chain-of-Thought (CoT) data.
- This SFT phase initializes the model to produce structured reasoning (e.g., using
<think>tags) and adopt a specific reasoning style. - The primary goals are to accelerate and stabilize the subsequent, more powerful GRPO training phase and to align the model's output for a better user experience.
Preview of the Next Lesson:
Now that the model has been "warmed up," it is ready for the main event. In our next lesson, we will explore the second stage of the pipeline: the reasoning-oriented reinforcement learning phase. We will examine how, starting from this SFT checkpoint, GRPO incentivizes the model to autonomously discover and refine sophisticated reasoning behaviors like self-verification and backtracking.