Variational Inference: Approximating Posteriors

Introduction

In our last lesson, we explored the architecture of autoencoders and saw how they excel at learning compressed, non-linear representations of data. We ended by noting a key limitation: while the encoder learns to map inputs to a latent space, this space is often unstructured. Simply picking a random point from the latent space and feeding it to the decoder is unlikely to produce a meaningful output. The model learns a mapping from the data, not a complete generative distribution for the data.

Today, we transition from simple reconstruction to true generative modeling. To do this, we need to adopt a probabilistic perspective. Instead of encoding an input as a single point in the latent space, what if we encoded it as a distribution? This shift forces us to confront a fundamental challenge in Bayesian machine learning: the problem of intractable posteriors. Our objective is to understand this problem and derive the core idea of variational inference (VI), the mathematical technique that provides a powerful and elegant solution. This concept is the "Variational" in Variational Autoencoder and is central to modern generative AI.

The Probabilistic View: Latent Variables and Intractable Posteriors

Let's formalize our goal. We want to build a generative model of our data $x$. A powerful way to do this is to assume that the data is generated by some underlying, unobserved latent variables $z$. This defines a generative process:

1. A latent variable $z$ is sampled from a prior distribution $p(z)$. We often choose something simple, like a standard normal distribution, $\mathcal{N}(0,I)$.
2. An observable data point $x$ is sampled from a likelihood distribution $p(x|z)$. This distribution is conditioned on the latent variable and is often complex, modeled by a neural network (our decoder).

Together, these define the joint distribution $p(x,z)=p(x|z)p(z)$. We can write down and evaluate the functional forms of $p(z)$ and $p(x|z)$, and therefore the joint.

However, a critical task is inference: given an observed data point $x$, what were the likely latent variables $z$ that generated it? We want to compute the posterior distribution $p(z|x)$. Using Bayes' theorem, we know:

The numerator, which is the joint distribution $p(x,z)$, is tractable. The problem lies in the denominator, $p(x)$, known as the marginal likelihood or evidence. To compute it, we must integrate (or "marginalize") over all possible latent variables:

This integral is the source of our difficulties. Why? In any non-trivial model, the latent space $z$ is high-dimensional, and the likelihood $p(x|z)$ is a complex, non-linear function (like a deep neural network). The integral has no analytical closed-form solution and is computationally intractable to approximate numerically.

The following video provides a concrete example of a simple generative model and demonstrates exactly why this marginalization step is intractable.

Since we cannot compute $p(z|x)$ directly, we cannot perform exact inference. This is the central problem that variational inference was developed to solve.

The Variational Solution: From Intractable Problem to Tractable Optimization

The core idea behind variational inference is this: if we cannot compute the true posterior $p(z|x)$, let's approximate it with another, simpler distribution $q_{\varphi }(z|x)$ that we can control. This distribution, called the variational posterior or surrogate posterior, is chosen from a tractable family of distributions, such as the Gaussian family. It has its own set of parameters, $\varphi$, which we can optimize.

Our goal is now to make $q_{\varphi }(z|x)$ as "close" as possible to the true, intractable posterior $p(z|x)$. From Module 4, you'll recall that a standard way to measure the "distance" between two probability distributions is the Kullback-Leibler (KL) divergence.

So, our objective becomes finding the parameters $\varphi$ that minimize the KL divergence between our approximation and the true posterior:

Now, we seem to have hit the same wall again. The formula for KL divergence involves the very distribution we deemed intractable:

This expression still requires us to evaluate $p(z|x)$. This is where a remarkable mathematical rearrangement comes into play, leading us to a tractable objective. The following video meticulously derives this result.

The Evidence Lower Bound (ELBO)

As derived in the video, by expanding the KL divergence and rearranging terms, we arrive at one of the most important identities in modern machine learning:

where $\mathcal{L}(\varphi ,\theta )$ is the Evidence Lower Bound (ELBO). Let's analyze this equation:

1. $\log p(x)$: The log of the model evidence. For a given data point $x$, this is a fixed value. It does not depend on the parameters $\varphi$ of our variational distribution.
2. $D_{KL}(\dots )$: The KL divergence between our approximation $q$ and the true posterior $p$. We know this is always non-negative ($D_{KL}\ge 0$).
3. $\mathcal{L}(\varphi ,\theta )$: The ELBO. Because the KL divergence is non-negative, the ELBO is always less than or equal to the log evidence: $\mathcal{L}(\varphi ,\theta )\le \log p(x)$. Hence its name: a "lower bound" on the evidence.

From this identity, the grand insight of variational inference emerges: since $\log p(x)$ is constant with respect to $\varphi$, maximizing the ELBO is equivalent to minimizing the KL divergence.

We have successfully transformed our original, intractable goal (minimizing a KL divergence that contains $p(z|x)$) into a new, tractable one: maximizing the ELBO. The ELBO is defined as:

This expression is tractable because it only involves the joint distribution $p_{\theta }(x,z)$ and our approximation $q_{\varphi }(z|x)$, both of which we can evaluate. We have sidestepped the need to compute the marginal $p(x)$ entirely.

For a deeper, text-based walkthrough of this derivation, the following article is excellent. It derives the same identity without relying on Jensen's inequality, providing a more direct and intuitive proof.

This act of approximating a complex probability distribution by optimizing a simpler, "variable" one is what gives variational inference its name. It's a technique rooted in a branch of mathematics called variational calculus, which deals with optimizing functionals (functions of functions).

Conclusion

In this lesson, we have confronted a core challenge in probabilistic modeling and established the fundamental principles of its solution. We have moved from the deterministic world of standard autoencoders into a probabilistic framework, setting the stage for building true generative models.

Key Takeaways:

• Intractable Posterior: In a latent variable model, computing the posterior distribution $p(z|x)$ is often intractable due to the high-dimensional integral required to calculate the marginal likelihood, $p(x)$.
• The Variational Approach: Variational Inference (VI) solves this by introducing a simpler, tractable distribution $q_{\varphi }(z|x)$ (the variational posterior) to approximate the true posterior.
• The Optimization Goal: The goal of VI is to make the approximation $q_{\varphi }(z|x)$ as close as possible to the true posterior $p(z|x)$ by minimizing the KL divergence between them.
• The Evidence Lower Bound (ELBO): By rearranging the definition of KL divergence, we arrive at the identity $\log p(x)=D_{KL}(q\Vert p)+\text{ELBO}$. This shows that maximizing the ELBO is equivalent to minimizing the KL divergence.
• From Inference to Optimization: VI elegantly transforms an intractable inference problem into a tractable optimization problem. We can now use techniques like gradient descent to find the parameters $\varphi$ that maximize the ELBO.

Preview of the Next Lesson:

We now have the theoretical tool—variational inference—needed to build a generative autoencoder. In our next lesson, we will integrate these ideas into the autoencoder architecture to derive the Variational Autoencoder (VAE). We will formulate the VAE's objective function using the ELBO and uncover a final, clever challenge: how do you backpropagate gradients through a sampling operation? The solution, known as the reparameterization trick, is a cornerstone of training VAEs and many other modern generative models.