Value vs. Policy-Based RL

Hello! Welcome to the final lesson in our "Artificial Intelligence: Search and Reasoning" module.

In our last session, we established the fundamental concepts of Reinforcement Learning (RL): the agent, environment, states, actions, rewards, and the agent's strategy, or policy. We learned that the agent's goal is to find an optimal policy ( $π^{*}$ ) that maximizes its long-term cumulative reward.

Today, we will address the crucial next question: How does an agent actually learn this optimal policy? We'll discover that there are two primary philosophical approaches to this problem. By the end of this lesson, you will be able to distinguish between value-based and policy-based reinforcement learning methods.

1. Two Paths to a Goal

Imagine you're trying to navigate a maze. You could approach this in two ways:

The Value-Based Approach: You could create a "heat map" of the maze, where you assign a score (a "value") to every position based on how close it is to the exit. To navigate, you would simply move from a lower-scoring position to an adjacent higher-scoring position. You are learning how good each position is.
The Policy-Based Approach: You could ignore the scores and instead draw arrows on the floor at every intersection, telling you which direction to turn. You are directly learning what action to take.

These two strategies represent the fundamental split between value-based and policy-based methods in Reinforcement Learning.

This diagram provides a high-level map of where these methods fit within the broader landscape of RL. Today, we focus on the two branches of "Model-free methods."

Classification of Reinforcement Learning Methods — A flowchart showing how Reinforcement Learning methods are classified. Model-free methods, which learn through trial and error, are divided into value-based and policy-based approaches.

2. The Core Distinction: Judging vs. Acting

Let's start with a clear, conceptual reading to frame this distinction.

Chapter 2: Value Based Reinforcement Learning vs Policy Based Reinforcement Learning

This article provides an excellent introduction to the core difference between the two methods using a Minesweeper analogy. It asks the fundamental question: should an agent learn to judge the value of each move, or should it learn the moves themselves?

Please read the 'Introduction' section of the article. Focus on the central question that separates the two strategies: 'Should I learn how valuable each move is... or should I directly learn what moves to make...?'

As the article highlights, the difference boils down to what the agent learns:

Value-Based Methods: The agent first learns a value function. This function estimates the expected cumulative reward from being in a particular state, or from taking a particular action in a state. The policy is then derived from these values (e.g., always choose the action with the highest value).
Policy-Based Methods: The agent directly learns a policy function. This function is a direct mapping from a state to an action (or a probability distribution over actions). The goal is to optimize this policy function directly without first needing to learn a value function.

3. Deep Dive: Q-Networks and Policy Gradients

To make this more concrete, let's see how these two approaches work in practice, especially when we use neural networks to approximate the functions. The following video provides a very friendly and intuitive explanation.

A friendly introduction to deep reinforcement learning, Q-networks and policy gradients

This video by Serrano.Academy explains how neural networks can be used to implement both value-based and policy-based methods. It will help you visualize the difference between a 'Q-network' (value-based) and a 'policy network' (policy-based).

Please watch the following segments: Introduction to the methods (14:53 - 16:22): This part introduces the two ways neural networks can solve RL problems: as a value network (Q-network) or a policy network. How Value Networks Train (21:32 - 25:47): Focus on the core idea. The network takes a state as input and outputs values for actions. It's trained to make these values consistent with the Bellman equation, which defines the relationship between a state's value and its neighbors' values. How Policy Networks Train (25:47 - 29:30): Focus on the different training logic here. The agent tries a path, gets a final score, and then goes back. Actions on paths with high scores are 'encouraged', and actions on paths with low scores are 'discouraged'. This directly tunes the policy.

Let's summarize the key ideas from the video.

Value-Based Learning (Q-Networks)

What it learns: A value function, often the state-action value function $Q (s, a)$ . This function tells you the expected return if you start in state $s$ , take action $a$ , and follow the optimal policy thereafter.
How it acts: The policy is often implicit. Given a state $s$ , the agent computes $Q (s, a)$ for all possible actions $a$ and chooses the action that has the maximum Q-value. This is a greedy policy.
Analogy: The agent becomes a "critic" that is excellent at judging the quality of every possible move from any position. It acts by always making the move it judges to be the best.
Example Algorithm: Q-Learning.

Policy-Based Learning (Policy Gradients)

What it learns: A policy function $π_{θ} (a ∣ s)$ directly. The neural network's parameters ( $θ$ ) define the policy. The network takes a state $s$ as input and outputs a probability for each action $a$ .
How it acts: The agent acts by sampling from the probability distribution output by its policy network. If the policy says "move right" with 90% probability and "move left" with 10%, it will most likely move right, but sometimes it will explore by moving left.
Analogy: The agent is an "actor" that develops an instinct or a reflex for what to do in any situation. Learning involves refining these instincts based on whether they led to good or bad outcomes.
Example Algorithm: REINFORCE.

Test your understanding!

A chess-playing AI is being trained.

If the AI learns by estimating a score for every possible move on the board and then picking the move with the highest score, what kind of method is it using?

Now, consider another AI that learns a direct probability for moving each piece to each possible square (e.g., "in this board state, there is a 70% chance the best move is Pawn to e4"). What kind of method is this?

Show answer

The first AI, which estimates a score (a value) for each move, is using a value-based method.
The second AI, which directly learns a probability distribution over moves, is using a policy-based method.

4. Head-to-Head Comparison

Now that you understand the two approaches, let's put them side-by-side and compare their strengths and weaknesses.

Feature	Value-Based Methods	Policy-Based Methods
Primary Goal	Learn a value function (e.g., $Q (s, a)$ ).	Learn a policy function ( $π (a ∥ s)$ ) directly.
Policy	Implicit. The policy is derived from the value function (e.g., choose the action with the max Q-value).	Explicit. The policy is the function being learned and optimized.
Action Spaces	Best for discrete and small action spaces. Finding the `max` over actions can be hard if there are too many.	Can handle continuous and high-dimensional action spaces naturally.
Policy Nature	Often deterministic (with exploration hacks like $ϵ$ -greedy). The agent always exploits the best-known action.	Naturally stochastic (probabilistic). The policy outputs probabilities, which allows for exploration to be part of the policy itself.
Sample Efficiency	Generally more sample-efficient. "Off-policy" algorithms like Q-learning can learn from old experiences (experience replay).	Generally less sample-efficient. "On-policy" algorithms often need new samples generated from the current policy for each update.
Stability	Learning can be unstable. A small change in a Q-value can drastically change the policy (e.g., a different action now has the max value).	Smoother updates. The probabilities of actions change gradually, leading to more stable learning.

Reinforcement Learning: Exploring Policy vs. Value-Based Methods

For a more detailed breakdown of these differences, this article provides a concise summary.

Skim the sections 'Key Differences Between Policy and Value-Based Methods' and the 'Conclusion'. This will reinforce the points made in the table above, such as differences in sample efficiency and handling of action spaces.

Conclusion

You have now reached the end of this lesson and the "Artificial Intelligence: Search and Reasoning" module. You've journeyed from fundamental search algorithms to the complex, uncertain world of Reinforcement Learning.

Key Takeaways from this Lesson:

There are two main approaches to solving RL problems: value-based and policy-based.
Value-based methods learn to be a good "critic." They estimate the value of states or actions and derive a policy from these estimates. They are sample-efficient but struggle with continuous action spaces.
Policy-based methods learn to be a good "actor." They directly learn a policy that maps states to actions. They can handle continuous actions and learn stochastic policies but are often less sample-efficient.
These two approaches have distinct trade-offs in terms of stability, sample efficiency, and applicability to different types of problems.

Understanding this distinction is crucial because it forms the basis for nearly all advanced RL algorithms. In fact, many state-of-the-art methods, such as those used to train large language models, are Actor-Critic methods. These are hybrid approaches that combine a policy-based "actor" with a value-based "critic" to get the best of both worlds. This is a topic for more advanced study, but you now have the foundational knowledge to understand how it works

Recommended resources