Deriving and Interpreting the Evidence Lower Bound (ELBO)

1. The Central Problem: Intractable Posteriors

Let's begin by formalizing the generative model that a VAE assumes. We imagine that our data, $x$, is generated by a two-step process:

1. A latent variable $z$ is sampled from a simple prior distribution, $p(z)$. Typically, this is a standard multivariate Gaussian, $p(z)=\mathcal{N}(0,I)$.
2. A data point $x$ is sampled from a conditional distribution $p_{\theta }(x|z)$. This is the "decoder" or "generative network," parameterized by $\theta$.

The joint probability is thus $p_{\theta }(x,z)=p_{\theta }(x|z)p(z)$. In this framework, our primary goal is often inference: given a data point $x$, what is the distribution of latent variables $z$ that could have generated it? This is the posterior distribution, $p_{\theta }(z|x)$. Using Bayes' theorem, we can write:

The problem lies in the denominator, $p_{\theta }(x)=\int p_{\theta }(x|z)p(z)dz$, which is the marginal likelihood of the data, also known as the "evidence". For a model where the decoder $p_{\theta }(x|z)$ is a complex neural network, this integral is computationally intractable. It requires integrating over all possible latent variables $z$, a high-dimensional and complex space. Since we cannot compute the evidence $p_{\theta }(x)$, we cannot compute the true posterior $p_{\theta }(z|x)$.

2. The Variational Solution: Introducing an Approximation

Variational inference provides an elegant solution: if the true posterior $p_{\theta }(z|x)$ is intractable, we approximate it with a simpler, tractable distribution $q_{\varphi }(z|x)$. This distribution is called the variational posterior or approximate posterior.

We choose a family of distributions for $q$ (e.g., Gaussians) and then use a neural network, called the "encoder" and parameterized by $\varphi$, to map an input $x$ to the parameters of this distribution. For instance, the encoder might output the mean ${\mu }_{\varphi }(x)$ and variance ${\Sigma }_{\varphi }(x)$ of a Gaussian, so that $q_{\varphi }(z|x)=\mathcal{N}(z;{\mu }_{\varphi }(x),{\Sigma }_{\varphi }(x))$.

The goal then becomes to find the parameters $\varphi$ that make our approximation $q_{\varphi }(z|x)$ as "close" as possible to the true posterior $p_{\theta }(z|x)$. As we learned in the last lesson, the natural way to measure the dissimilarity between two distributions is the KL divergence. We want to minimize $D_{KL}(q_{\varphi }(z|x)\Vert p_{\theta }(z|x))$.

The following reading explains this setup from a probabilistic modeling perspective.

3. Deriving the Evidence Lower Bound (ELBO)

We've established that our objective is to minimize the KL divergence between the approximate and true posteriors. However, computing this KL divergence directly still requires knowing the true posterior $p_{\theta }(z|x)$, which is intractable. This seems like a catch-22.

Let's see what happens when we expand the definition of this KL divergence:

Using Bayes' rule, we can rewrite $\log p_{\theta }(z|x)=\log p_{\theta }(x,z)-\log p_{\theta }(x)$. Substituting this in:

Since $\log p_{\theta }(x)$ does not depend on $z$, we can pull it out of the expectation:

Rearranging this equation to solve for the log-evidence, $\log p_{\theta }(x)$, gives us the fundamental identity of variational inference:

Let's define the second term as the Evidence Lower Bound (ELBO):

Our fundamental identity becomes:

We know that KL divergence is always non-negative ($D_{KL}\ge 0$). This implies that $\log p_{\theta }(x)\ge \text{ELBO}$. The ELBO is a lower bound on the log-evidence of our data. Because the gap between the ELBO and the log-evidence is the very KL divergence we want to minimize, maximizing the ELBO is equivalent to minimizing the KL divergence between our approximation and the true posterior.

The ELBO is something we can actually compute and optimize! It depends only on $p_{\theta }(x,z)$ (which is just $p_{\theta }(x|z)p(z)$) and $q_{\varphi }(z|x)$, both of which are defined by our model.

The following resources walk through this derivation in detail.

An Alternative Derivation with Jensen's Inequality

There is another, more direct way to derive the ELBO using Jensen's inequality, which you may find more concise given your mathematical background. For any concave function $f$ (like the logarithm) and random variable $Y$, Jensen's inequality states that $\mathbb{E}[f(Y)]\le f(\mathbb{E}[Y])$.

We start with the log-evidence and introduce our variational distribution $q_{\varphi }(z|x)$:

Now, we apply Jensen's inequality. Since $\log$ is concave, we can move it inside the expectation, which flips the equality to an inequality:

The right-hand side is exactly our definition of the ELBO. This directly shows that the ELBO is a lower bound on the log-evidence.

4. Interpreting the Terms of the ELBO

So, our objective is to maximize the ELBO. To gain intuition, let's rearrange it into a more interpretable form.

Starting with our definition:

We expand the joint probability $\log p_{\theta }(x,z)=\log p_{\theta }(x|z)+\log p(z)$:

Now, we group the terms:

The second term is exactly the definition of $D_{KL}(q_{\varphi }(z|x)\Vert p(z))$. So, the final, most common form of the ELBO is:

Maximizing the ELBO means maximizing the reconstruction term while minimizing the KL divergence. In practice, we minimize the negative ELBO as our loss function:

Let's analyze these two terms:

1. Reconstruction Loss: This term pushes the decoder ($\theta$) to learn to reconstruct the input $x$ accurately from latent codes $z$ sampled from the encoder's output. Maximizing the log-likelihood $\log p_{\theta }(x|z)$ is equivalent to minimizing a reconstruction loss. For example, if the decoder assumes a Gaussian output, this becomes a mean squared error (MSE) loss. If it assumes a Bernoulli output (for binary images), this becomes a binary cross-entropy (BCE) loss.

2. KL Regularizer: This term measures the divergence between the approximate posterior $q_{\varphi }(z|x)$ and the prior $p(z)$. It acts as a regularizer, forcing the distributions produced by the encoder to stay close to the prior distribution (e.g., a standard normal). This encourages the latent space to be well-structured and continuous, which is crucial for being able to generate new data by sampling from the prior. We can compute this term analytically using the formula we derived in the previous lesson.

The VAE training dynamic is a trade-off between these two terms. The model must learn to encode information into $z$ to reconstruct $x$ well, but not so well that the distribution of $z$ strays too far from the simple prior.

Conclusion

In this lesson, we have derived the central objective function for training Variational Autoencoders from first principles. We have connected the practical problem of generative modeling with the theoretical framework of variational inference.

Key Takeaways:

• Generative modeling with latent variables often leads to an intractable posterior $p(z|x)$ because the evidence $p(x)$ is impossible to compute.
• Variational inference approximates the true posterior with a simpler, tractable distribution $q_{\varphi }(z|x)$ (the encoder).
• The objective is to minimize the KL divergence between the approximate and true posteriors, which is equivalent to maximizing the Evidence Lower Bound (ELBO).
• The ELBO is composed of two competing terms:
  • A reconstruction term that encourages the decoder to faithfully reconstruct the input data.
  • A regularization term (a KL divergence) that forces the encoder's output distribution to match a simple prior, ensuring a well-structured latent space.
• The final VAE loss function is the negative ELBO: $\text{Loss}=\text{Reconstruction Loss}+\text{KL Regularizer}$.

You now have a complete picture of the VAE objective. In our next lesson, we will briefly explore a subtle but important detail: the difference between forward and reverse KL divergence and its implications for how models like VAEs learn, a phenomenon often described as "mode-seeking" versus "mode-covering." This will be our final piece of theory before we move on to implementation.