Introduction
Welcome back! In our previous lesson, we addressed the "cold start" problem in training reasoning models. We saw how an initial Supervised Fine-Tuning (SFT) phase on Chain-of-Thought (CoT) data "warms up" the model, teaching it the basic structure and format of a reasoning process, such as using <think> and </think> tags.
Now that the model knows what reasoning looks like, we can move to the crucial next stage: teaching it how to reason effectively. Our learning outcome for this lesson is to explain how reasoning behaviors such as self-verification and backtracking emerge during GRPO training on R1-Zero.
We will explore one of the most fascinating aspects of modern AI: how complex, seemingly intelligent behaviors can arise "organically" from a simple reinforcement learning process. The model is not explicitly taught to check its work or correct its mistakes. Instead, it discovers these strategies on its own, driven by the pure incentive of maximizing its reward.
The Setup: Pure Reinforcement Learning
The model we're focusing on is DeepSeek-R1-Zero. The "Zero" is significant—it signifies that this model's reasoning capabilities are developed through pure reinforcement learning, with zero reliance on human-annotated reasoning paths during this phase.
The setup is as follows:
- Starting Point: We begin with the model that has completed the "cold start" SFT. It knows how to structure a response but isn't yet a skilled reasoner.
- Task: The model is given problems from verifiable domains like mathematics and coding.
- Reward Signal: This is the key. The reward is entirely rule-based and outcome-driven.
- For a math problem, the reward is positive if the final answer is correct, and zero otherwise.
- For a coding problem, the reward is positive if the code compiles and passes all unit tests, and zero otherwise.
Crucially, the model gets no credit for the "thinking process" itself, only for the final result. This simple, binary reward signal is the only incentive the model has to improve.
The Emergence of Complex Behavior
So, how does a model go from this simple setup to developing sophisticated strategies like checking its own work? It's a powerful feedback loop driven by GRPO. The following video provides an excellent overview of the results of this process.
How to Train LLMs to "Think" (o1 & DeepSeek-R1)
Let's start by observing the results. This video segment from 'How to Train LLMs to "Think"' by Shaw Talebi discusses the performance of R1-Zero and showcases an example of its emergent reasoning behavior.
Please watch from 19:58 to 22:30. Pay close attention to two key observations: The graph showing that the model's responses get longer as training progresses. The example where the model performs 'verification' and 'reflection' steps, even though it was never explicitly told to do so.
As you saw, two things happen: the model's thought process gets longer, and it starts exhibiting behaviors like self-verification. Let's break down how this occurs.
1. The Incentive for Longer Chains of Thought
At the beginning of RL training, the model's reasoning attempts are short and often incorrect. However, by random chance (stochastic sampling), it will occasionally generate a slightly longer, more detailed chain of thought that happens to lead to the correct answer.
- The Reward: This successful trajectory receives a positive reward.
- The GRPO Update: The GRPO algorithm then increases the probability of the entire sequence of tokens that led to this success.
- The Feedback Loop: As the model is updated, it becomes more likely to generate longer chains ofthought. This, in turn, increases its chances of solving problems correctly, which leads to more rewards, further reinforcing the behavior.
This creates a self-perpetuating cycle where the model "learns" that generating more thinking tokens is a good strategy for maximizing reward.
2. The Emergence of Self-Verification and Backtracking
The emergence of self-verification is even more remarkable. It follows the same principle: strategies that lead to correct answers are reinforced.
Imagine the model generates a line of reasoning, makes a calculation error, but then, by chance, generates tokens like: "Wait, let's double-check that step." It then corrects the error, proceeds along a new path, and arrives at the correct final answer.
Because the final outcome was correct, the entire successful trajectory—including the "Wait, let's double-check" part—receives a positive reward and is reinforced by GRPO. The model isn't "aware" that it's verifying its work. It has simply learned, through trial and error, that token sequences corresponding to reflection and correction are part of a high-reward strategy.
To get a deeper understanding of how the RL mechanism enables this, let's watch a short clip that explains how long-term rewards are propagated back to every action (token choice) the model makes.
Paper: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
The video 'Paper: DeepSeek-R1' by Umar Jamil provides a clear explanation of how reinforcement learning assigns credit for long-term success to early actions. This is the core mechanism behind emergent reasoning.
Watch from 34:37 to 37:28. The speaker uses a simple analogy of a cat finding meat to explain how the reward signal is propagated back through a trajectory. This helps understand why not just the final correct answer, but the entire reasoning path, including verification steps, gets reinforced.
The researchers at DeepSeek called this phenomenon the "aha moment," where the model appeared to discover these reflective strategies on its own.
DeepSeek-R1 Paper: Emergence of Reasoning
Let's read the researchers' own description of this phenomenon from the DeepSeek-R1 paper.
Please read Section 2.3, focusing on the paragraphs after the template description, and Appendix C.2 (the text below Figure 9). Notice how the paper quantifies this emergent behavior by tracking the frequency of reflective words like 'wait,' 'verify,' and 'mistake' during training.
Why Pure RL is Crucial for Emergent Reasoning
A key insight from the DeepSeek-R1 paper is that these novel reasoning strategies emerge precisely because the RL training is unconstrained.
- Supervised Fine-Tuning (SFT) teaches the model to imitate human reasoning. This is useful for the cold start, but if used exclusively, it can limit the model to known human problem-solving patterns.
- Reinforcement Learning (RL), especially with a simple outcome-based reward, allows the model to explore the vast space of possible reasoning strategies. It is free to discover novel, and potentially more effective, ways of thinking that a human might not have provided in an SFT dataset.
This is the key difference that can elevate a model from a "stochastic parrot" that merely mimics training data to a genuine problem-solver.
How to Train LLMs to "Think" (o1 & DeepSeek-R1)
This final video segment frames the discussion in a broader context, drawing a powerful analogy to AlphaGo.
Watch from 30:16 to 32:50. Pay attention to how reinforcement learning enabled AlphaGo to surpass human-level performance by discovering novel strategies, and how the same principle applies to LLMs for reasoning.
Conclusion
In this lesson, we've unpacked the fascinating process by which complex reasoning behaviors emerge from a simple reinforcement learning setup.
Key Takeaways:
- Emergent behavior is the development of sophisticated strategies (like self-verification and backtracking) without explicit instruction.
- It is driven by a positive feedback loop in RL: strategies that happen to lead to correct outcomes are rewarded and reinforced by GRPO, making them more likely to occur in the future.
- The model learns that generating longer chains of thought and incorporating reflective/corrective language are high-utility patterns for maximizing reward in verifiable tasks.
- This process is enabled by unconstrained exploration during RL, which allows the model to discover novel reasoning strategies beyond what it could learn from imitating human examples alone.
Preview of the Next Lesson:
We now have DeepSeek-R1-Zero, a model that is an incredibly powerful, specialized reasoner but may lack general helpfulness or politeness. So, what's next? In our upcoming lesson, we will examine the rejection sampling stage of the DeepSeek-R1 pipeline. You will learn how R1-Zero is used as a data generation engine to create a massive, high-quality dataset, which is then used to train the final, more balanced, and user-facing DeepSeek-R1 model.