Policy Gradient vs. Value-Based Methods

Introduction

In our last lesson, we explored Q-learning, a canonical value-based method. The core strategy was to learn an action-value function, $Q(s,a)$, which estimates the long-term reward for taking action $a$ in state $s$, and then use this function to derive a policy, typically by acting greedily. This approach is powerful, but it relies on an indirect method for finding the best policy: first find the values, then infer the actions. What if we could learn the policy directly?

This lesson introduces a fundamentally different and powerful class of algorithms: policy gradient methods. Instead of learning value functions, we will directly parameterize the policy itself, ${\pi }_{\theta }(a|s)$, and use optimization techniques to find the parameters $\theta$ that maximize the expected return. You will learn the theoretical motivation for this approach, its advantages over value-based methods, especially in complex scenarios, and the core mathematical principle—the Policy Gradient Theorem—that makes it possible. We will then see how this theorem is put into practice, culminating in an understanding of Actor-Critic methods, a hybrid approach that combines the strengths of both policy-based and value-based learning.

The Limits of Value-Based Methods

Value-based methods like Q-learning are highly effective, but they face challenges in certain situations. Considering these challenges provides the motivation for exploring a direct policy-based approach.

1. Continuous or High-Dimensional Action Spaces: The core of control in Q-learning involves finding the action that maximizes the Q-value: $\arg {\max }_{a}Q(s,a)$. If the action space is discrete and small, this is a simple lookup. However, if the action space is continuous (e.g., the precise torque to apply to a robotic arm) or exceptionally large, this maximization step becomes a complex optimization problem in its own right, which must be solved at every single step.

2. Stochastic Policies: Value-based methods typically produce deterministic policies (or $ϵ$-greedy variants). In some environments, the optimal policy is genuinely stochastic. For instance, in a game of rock-paper-scissors, the optimal policy is to play each action with a probability of 1/3. A deterministic policy would be easily exploited. Policy-based methods can naturally learn truly stochastic policies.

3. Policy Complexity vs. Value Function Complexity: In some problems, the optimal policy might be much simpler to represent than its corresponding value function. Imagine a robot needing to move left when a specific light turns on. The policy ("if light, go left") is simple. The value function, which must quantify the long-term reward of every possible state-action pair, might be incredibly complex.

The Core Idea: Direct Policy Parameterization

The central idea of policy gradient methods is to treat reinforcement learning as a direct optimization problem. We define the policy as a parameterized function ${\pi }_{\theta }(a|s)$, where $\theta$ is a vector of parameters (e.g., the weights of a neural network). This function outputs the probability of taking action $a$ in state $s$.

Our goal is to find the parameters $\theta$ that maximize a performance objective, $J(\theta )$, which is typically the expected total return. We can then use gradient ascent to update the parameters in the direction that increases this objective:

$${\theta }_{k+1}={\theta }_k+\alpha {\nabla }_{\theta }J({\theta }_k)$$

The entire challenge now boils down to one question: How do we compute the policy gradient, ${\nabla }_{\theta }J(\theta )$? The environment's dynamics and the reward function are unknown, so we can only rely on experience sampled by following the policy.

The Policy Gradient Theorem

The solution comes from a fundamental result known as the Policy Gradient Theorem. This theorem provides a way to calculate an analytical expression for the policy gradient ${\nabla }_{\theta }J(\theta )$ that we can estimate from samples. While the full derivation is involved, the final result is elegant and intuitive. For any differentiable policy ${\pi }_{\theta }$, the gradient of the objective $J(\theta )$ is given by:

$${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{{\pi }_{\theta }}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(a|s)\cdot Q^{{\pi }_{\theta }}(s,a)\right]$$

Let's break down this crucial equation:

• ${\mathbb{E}}_{{\pi }_{\theta }}[\cdot ]$ indicates that we are taking an expectation over trajectories of states and actions generated by following our current policy ${\pi }_{\theta }$.
• ${\nabla }_{\theta }\log {\pi }_{\theta }(a|s)$ is the score function. This is the gradient of the log-probability of the action we actually took. From a mathematical perspective (using the "log-derivative trick"), this term allows us to express the gradient of an expectation as the expectation of a gradient. Intuitively, this vector tells us which direction to move $\theta$ in to make the action $a$ more likely.
• $Q^{{\pi }_{\theta }}(s,a)$ is the familiar action-value function: the expected return for taking action $a$ in state $s$ and then following policy ${\pi }_{\theta }$ thereafter. It acts as a weighting term, telling us the long-term consequence of that action.

The intuition is powerful: We sample an action $a$ from our policy in state $s$. We then see how "good" that action was by estimating its Q-value. If $Q^{{\pi }_{\theta }}(s,a)$ is high (a good action), we update our parameters $\theta$ in the direction of ${\nabla }_{\theta }\log {\pi }_{\theta }(a|s)$ to make that action more probable. If $Q^{{\pi }_{\theta }}(s,a)$ is low or negative (a bad action), the update moves $\theta$ in the opposite direction, making that action less probable.

REINFORCE: Monte Carlo Policy Gradient

The simplest algorithm that stems directly from the Policy Gradient Theorem is called REINFORCE. It's a Monte Carlo method, meaning it learns from complete episodes.

The algorithm works as follows:

1. Initialize the policy parameters $\theta$ arbitrarily.
2. Generate a full episode by following the current policy ${\pi }_{\theta }$: $s_0,a_0,r_1,s_1,a_1,\dots ,s_{T-1},a_{T-1},r_T$.
3. For each time step $t$ in the episode, calculate the return from that point onward: $G_t={\sum }_{k=t+1}^T{\gamma }^{k-t-1}{r}_k$.
4. This return $G_t$ is used as an unbiased, sampled estimate of $Q^{{\pi }_{\theta }}(s_t,a_t)$.
5. Update the policy parameters by taking a step in the direction of the estimated gradient: $$\theta \leftarrow \theta +\alpha {\gamma }^t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)G_t$$

While simple and unbiased, REINFORCE suffers from the classic problem of Monte Carlo methods: high variance. The return $G_t$ is the sum of many random reward variables, making it a very noisy estimate of the true Q-value. An action that was good in the long run might, by chance, be part of an episode that ended poorly, and it would be incorrectly penalized. This high variance leads to slow and unstable learning.

Reducing Variance: Baselines and Actor-Critic Methods

The key to making policy gradients practical is to reduce the variance of the gradient estimate. A common and powerful technique is to introduce a baseline, $b(s)$, which is subtracted from the Q-value term. The policy gradient can be rewritten as:

$${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{{\pi }_{\theta }}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(a|s)\cdot (Q^{{\pi }_{\theta }}(s,a)-b(s))\right]$$

As long as the baseline $b(s)$ does not depend on the action $a$, this transformation is exact and does not change the expectation of the gradient (it introduces no bias). However, a well-chosen baseline can dramatically reduce the variance of the estimate.

A natural and effective choice for the baseline is the state-value function, $b(s)=V^{{\pi }_{\theta }}(s)$. This leads to a new term in our update:

$$A^{{\pi }_{\theta }}(s,a)=Q^{{\pi }_{\theta }}(s,a)-V^{{\pi }_{\theta }}(s)$$

This is called the Advantage Function. It measures how much better taking action $a$ is compared to the average action from state $s$. Intuitively, we don't care how good a state is in absolute terms; we only care about whether an action we took was better or worse than average for that state. Using the advantage leads to much more stable learning.

This brings us to Actor-Critic methods. These methods explicitly learn both a policy and a value function. They are a hybrid that combines the best of policy-based and value-based approaches.

• The Actor is the policy ${\pi }_{\theta }(a|s)$. It decides which actions to take. Its parameters $\theta$ are updated using the policy gradient, guided by feedback from the critic.
• The Critic is a value function estimator, for example $V_{\varphi }(s)$ with parameters $\varphi$. It evaluates the actions taken by the actor by learning the value function of the current policy. It doesn't take actions; it only "criticizes" the actor.

In a typical Actor-Critic loop:

1. The actor takes an action $a_t$ in state $s_t$. The environment returns a reward $r_{t+1}$ and next state $s_{t+1}$.
2. The critic evaluates the "surprise" or TD-error: ${\delta }_t=r_{t+1}+\gamma V_{\varphi }(s_{t+1})-V_{\varphi }(s_t)$. This TD-error is an unbiased estimate of the advantage function!
3. The critic's parameters $\varphi$ are updated to minimize this error (e.g., using TD learning).
4. The actor's parameters $\theta$ are updated using the TD-error as the advantage signal: $$\theta \leftarrow \theta +\alpha {\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t){\delta }_t$$

This structure allows for online, step-by-step learning and is far more sample-efficient and stable than the vanilla REINFORCE algorithm. The diagram below illustrates the conceptual difference between the direct optimization loop of policy gradients and the value-fitting loop of value-based methods. Actor-Critic effectively merges these two paradigms.

Conclusion

This lesson introduced a new perspective on reinforcement learning, shifting from learning values to learning policies directly. This approach, grounded in the Policy Gradient Theorem, offers compelling advantages, particularly for problems with continuous action spaces or where stochastic policies are optimal.

Here are the key takeaways:

• Policy Gradient (PG) methods directly parameterize a policy ${\pi }_{\theta }(a|s)$ and optimize the parameters $\theta$ to maximize expected return using gradient ascent.
• They differ from value-based methods (like Q-learning) which learn a value function and derive a policy implicitly. PG methods are often more stable and better suited for continuous action spaces.
• The Policy Gradient Theorem provides a theoretical foundation, showing that the policy gradient is the expected product of the score function (${\nabla }_{\theta }\log {\pi }_{\theta }(a|s)$) and the action-value function ($Q^{{\pi }_{\theta }}(s,a)$).
• REINFORCE is the basic Monte Carlo implementation of this theorem but suffers from high variance.
• Actor-Critic methods significantly improve upon this by using a learned value function (the critic) to provide a low-variance estimate of the advantage (the advantage function), which then guides the updates to the policy (the actor).

We have now covered the three main families of model-free RL algorithms: value-based (Q-learning), policy-based (REINFORCE), and the hybrid Actor-Critic approach. These form the building blocks for most advanced algorithms in use today.

In the next lessons, we will shift our focus from algorithms to the practical engineering challenges of deploying and managing machine learning models in production, starting with the principles of MLOps.