The Policy Gradient Theorem: Derivation and Significance

1. The Challenge: Optimizing with a Black-Box Reward

In supervised learning, you have a clear loss function (like cross-entropy) that you can directly differentiate with respect to your model's weights. In RL, our objective is to maximize the expected total reward, $J(\theta )=\mathbb{E}[R(\tau )]$, where $\tau$ is a trajectory generated by our policy ${\pi }_{\theta }$.

The problem is that the reward doesn't come from a nice, differentiable function. It comes from interacting with an environment—be it a video game, a physical robot, or even a human providing feedback on an LLM's response. You can't backpropagate a gradient through a game engine or a person's opinion.

This video introduces this core challenge, setting the stage for why we need a new mathematical tool.

As the video highlights, the path for gradients is broken. We need a way to estimate the gradient ${\nabla }_{\theta }J(\theta )$ without differentiating the reward function itself.

2. The Policy Gradient Theorem: Derivation and Statement

The Policy Gradient Theorem provides an elegant solution to this problem. It gives us an expression for the policy gradient that we can compute, transforming an intractable problem into a practical one that can be solved with sampling.

The derivation relies on a clever piece of calculus known as the log-derivative trick. This trick allows us to rewrite the gradient in the form of an expectation, which can then be estimated using a Monte Carlo approach (i.e., by averaging over samples from our policy).

The following video provides an excellent, step-by-step derivation.

To solidify your understanding, let's review the same derivation in a written format from OpenAI's "Spinning Up" series, a highly respected resource in RL education.

The Theorem

As you've now seen in two forms, the Policy Gradient Theorem states that the gradient of the policy performance $J(\theta )$ is:

Let's break this down:

• ${\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[\dots ]$: This is an expectation over trajectories $\tau$ sampled by running our policy ${\pi }_{\theta }$ in the environment. This means we can estimate this value by running the policy, collecting a batch of trajectories, and averaging the results.
• $R(\tau )$: This is the total reward for an entire trajectory $\tau$. It acts as a weight.
• ${\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)$: This is the gradient of the log-probability of taking action $a_t$ in state $s_t$. This term is crucial because it's a vector in the parameter space that tells us how to change $\theta$ to make the action $a_t$ more (or less) likely at state $s_t$.

In short, the theorem tells us that to get the policy gradient, we can sample a trajectory, see what total reward we got, and then nudge our policy parameters to make that entire sequence of actions more likely if the reward was good, and less likely if the reward was bad.

3. The Significance of the Theorem

So, why is this theorem so important? It has three profound implications that unlock modern reinforcement learning.

1. It Makes the Gradient Computable

The most critical contribution is that it transforms an intractable gradient into a sample-based estimate. We don't need to know the inner workings of the environment ($p(s^{\prime }|s,a)$) or differentiate it. All we need is to be able to:

1. Run our policy ${\pi }_{\theta }$ to generate trajectories.
2. Calculate the log-probabilities of the actions we took, $\log {\pi }_{\theta }(a|s)$, which is a standard output for any probabilistic model like one using a softmax layer.
3. Observe the total reward $R(\tau )$.

This is why policy gradients are so versatile and can be applied to complex environments like video games or robotics simulators.

2. It Provides an Intuitive Update Rule

The formula directly translates to an intuitive learning rule: "reinforce good behavior."

• If $R(\tau )$ is high and positive, the gradient step will increase the log-probabilities of all actions taken in that trajectory.
• If $R(\tau )$ is negative or low, the gradient step will decrease the log-probabilities.

This aligns perfectly with the goal of finding a policy that maximizes reward.

3. It Decouples the Policy from the Environment Dynamics

The gradient calculation, ${\nabla }_{\theta }J(\theta )$, depends on ${\pi }_{\theta }$ and the rewards, but not on the gradient of the state distribution, ${\nabla }_{\theta }{d}^{\pi }(s)$. This is a subtle but vital point. The distribution of states you visit depends on your policy, and if you had to calculate how that distribution changes when your policy changes, the problem would be much harder. The theorem cleverly sidesteps this.

This short reading emphasizes this crucial point.

This decoupling is what makes policy gradients a general-purpose tool for optimization in sequential decision-making problems.

Conclusion

In this lesson, we have moved from evaluating policies to directly optimizing them. We confronted the challenge of optimizing for black-box rewards and saw how the Policy Gradient Theorem provides a powerful and elegant solution.

Key Takeaways:

• Policy-based methods directly optimize the parameters $\theta$ of a policy ${\pi }_{\theta }(a|s)$ to maximize expected reward.
• The primary challenge is that the reward function is often a non-differentiable "black box," making standard gradient descent impossible.
• The Policy Gradient Theorem provides a computable expression for the gradient of the expected reward: $${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\left(\sum _{t=0}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)\right)R(\tau )\right]$$
• Significance: The theorem's main contribution is transforming an intractable analytical gradient into an expectation that can be estimated by sampling trajectories from the policy. This allows us to use gradient ascent to improve policies in any environment where we can run the policy and observe rewards.

Preview of the Next Lesson:
The theorem gives us the gradient, but how do we practically use it in an algorithm? In our next lesson, we will cover the REINFORCE algorithm, the most direct and intuitive implementation of the policy gradient theorem. We will see how it uses sampled episode returns to estimate the policy gradient and update the agent's policy. We will also begin to uncover a major weakness of this simple approach—high variance—which will motivate the more advanced algorithms we will study later, such as PPO and GRPO.