Importance Sampling in Off-Policy Policy Gradients

1. From On-Policy to Off-Policy: The Core Problem

The core challenge of off-policy learning is a distributional mismatch. The policy gradient theorem, which we use to update our policy ${\pi }_{\theta }$, relies on an expectation calculated over trajectories generated by that same policy ${\pi }_{\theta }$.

If we have a dataset $\mathcal{D}$ collected by a different policy, say ${\pi }_{\text{behavior}}$, we can't just plug those samples into the equation above. The data comes from a different probability distribution, so the expectation would be incorrect. We need a way to correct for this mismatch.

2. The Solution: Importance Sampling

The statistical technique designed to solve exactly this problem is called Importance Sampling (IS). It provides a way to estimate the expected value of a function under one distribution, using samples drawn from another.

The following video provides an excellent and intuitive explanation of the core mathematical trick behind importance sampling.

The Importance Sampling Ratio

As the video explained, the central insight of importance sampling is to re-weight the samples. If we want to compute the expectation of a function $f(x)$ under a target distribution $p(x)$ using samples from a proposal (or behavior) distribution $q(x)$, we can do so with the following identity:

The term $\frac{p(x)}{q(x)}$ is the importance sampling ratio (or importance weight). It "corrects" for the fact that we are sampling from the "wrong" distribution, $q(x)$. It works by:

• Up-weighting samples that were more likely under our target distribution $p(x)$ than under the behavior distribution $q(x)$.
• Down-weighting samples that were less likely under $p(x)$.

3. Applying Importance Sampling to Policy Gradients

Now, let's apply this directly to our reinforcement learning problem. In our case:

• The target distribution $p(x)$ is our new policy we want to evaluate or improve, ${\pi }_{\theta }(a|s)$.
• The behavior distribution $q(x)$ is the old policy that collected the data, $\beta (a|s)$.

The following reading provides a clear derivation of the off-policy policy gradient objective using exactly this principle.

As the article shows, the gradient of our off-policy objective becomes:

Here, the importance sampling ratio is ${\rho }_t(\theta )=\frac{{\pi }_{\theta }(a_t|s_t)}{\beta (a_t|s_t)}$. This single ratio allows us to take a batch of data collected by the behavior policy $\beta$ and use it to calculate an unbiased estimate of the gradient for our target policy ${\pi }_{\theta }$. This means we can finally reuse old data, making our learning process much more sample-efficient.

4. The Catch: The Problem of High Variance

Importance sampling is powerful, but it comes with a significant risk: high variance. If the target policy ${\pi }_{\theta }$ becomes very different from the behavior policy $\beta$, the importance ratio ${\rho }_t(\theta )$ can become extremely large or close to zero.

• If ${\pi }_{\theta }(a|s)$ is large but $\beta (a|s)$ is small, the ratio blows up. A single data point can have an enormous effect on the gradient update, leading to instability.
• If ${\pi }_{\theta }(a|s)$ is small but $\beta (a|s)$ is large, the ratio becomes tiny, and the data point is effectively ignored.

The following resources explain and visualize this critical issue.

This variance problem is the central challenge that modern policy gradient algorithms like TRPO and PPO were designed to solve.

5. The Solution: Trust Region Methods (A Preview)

If the variance of the importance sampling estimate explodes when the new and old policies are too different, the solution is intuitive: don't let the policies get too different.

This is the core idea behind Trust Region Policy Optimization (TRPO) and its more popular successor, Proximal Policy Optimization (PPO). They use the importance sampling ratio to enable off-policy updates but add a constraint or a penalty to ensure the new policy ${\pi }_{\theta }$ stays within a "trust region" of the old policy $\beta$ that collected the data.

Let's watch a brief clip that introduces this concept.

As the video explains, TRPO frames the problem as:

• Maximize the surrogate objective (which uses the importance sampling ratio).
• Subject to the constraint that the KL divergence between the old and new policies is less than some small value $\delta$.

This ensures that we get the sample efficiency benefits of off-policy learning while controlling the variance introduced by the importance sampling ratio.

Conclusion

In this lesson, we established the motivation and mechanism for off-policy policy gradient methods, which are far more sample-efficient than their on-policy counterparts.

Key Takeaways:

• On-policy algorithms like REINFORCE are sample-inefficient because they discard data after each update.
• Off-policy algorithms solve this by learning from a dataset collected by a behavior policy ($\beta$) while optimizing a target policy (${\pi }_{\theta }$).
• The importance sampling ratio, ${\rho }_t(\theta )=\frac{{\pi }_{\theta }(a_t|s_t)}{\beta (a_t|s_t)}$, is a correction factor that reweights the contribution of each sample to give an unbiased estimate of the policy gradient.
• The main drawback of importance sampling is high variance when the target and behavior policies diverge significantly.
• Modern algorithms like TRPO and PPO manage this variance by constraining the policy update, ensuring the new policy stays in a "trust region" close to the old one.

Preview of the Next Lesson:
We have now assembled all the foundational pieces of reinforcement learning needed for this course. In our next module, we will move from these general RL concepts to the specific algorithms used to align modern large language models. We'll begin with a deep dive into the full RLHF (Reinforcement Learning from Human Feedback) pipeline, starting with Supervised Fine-Tuning (SFT) and Reward Modeling, before finally tackling Proximal Policy Optimization (PPO) itself. You will see the clipped importance sampling ratio at the very heart of the PPO objective function.