Bayesian Model Comparison

1. From Parameter Posterior to Model Posterior

So far, we have focused on the posterior distribution of a model's parameters, $\theta$, given the data, $D$: $P(\theta |D)$. For model comparison, we elevate this idea to the level of the models themselves. Our goal is to calculate the posterior probability of a model, $M$, given the data: $P(M|D)$.

Using Bayes' theorem once again, we can write this as:

• Model Prior $P(M)$: Our prior belief that model $M$ is the correct one, before seeing any data. For example, we might assign a higher prior probability to a simpler model.
• Model Evidence $P(D|M)$: The probability of observing the data $D$ given the model $M$. This is the central quantity in Bayesian model comparison. It's also called the marginal likelihood.

The marginal likelihood is the key. To understand how it's calculated and what it represents, let's watch a short video.

As the video explained, the marginal likelihood $P(D|M)$ is calculated by "averaging out" or "integrating out" the parameters $\theta$ of the model:

Don't worry too much about solving the integral. The key intuition is that the model evidence is the average probability of the data, averaged over all possible parameter values weighted by their prior probabilities. It answers the question: "How well does this model, as a whole, predict the observed data?"

2. Model Evidence as an Automatic Occam's Razor

This brings us to a beautiful and powerful aspect of Bayesian model comparison. The marginal likelihood $P(D|M)$ has a built-in penalty for unnecessary complexity. This is often called a Bayesian "Occam's Razor," which favors the simplest explanation that fits the data well.

How does this work? Let's consider three models:

• Model 1 (Simple): A very simple model (e.g., a constant value). It can only explain a very narrow range of datasets.
• Model 2 (Just Right): A model with medium complexity that happens to be a good fit for our specific dataset.
• Model 3 (Complex): A very complex model (e.g., a high-order polynomial). It is so flexible it can explain a vast range of different datasets.

Because a complex model can generate many different possible datasets, its predictive probability is "spread thin" across all of them. So, for any one specific dataset, the probability $P(D|M_3)$ is relatively low. The simple model is not flexible enough to explain our data, so its evidence $P(D|M_1)$ is also low. The model that is "just complex enough" to explain the data and not much else will have its predictive probability concentrated on datasets like ours, resulting in the highest model evidence.

This image provides a great visual for this concept.

To deepen this intuition, the following video provides a brilliant explanation.

3. Comparing Models: Bayes Factors and Posterior Odds

Now that we understand model evidence, comparing two models, $M_1$ and $M_2$, is straightforward.

Bayes Factor

The most common way to compare two models is to compute the ratio of their model evidences. This ratio is called the Bayes Factor, $B{F}_{12}$.

The Bayes Factor tells you how much the data has shifted your belief in favor of one model over the other.

• If $B{F}_{12}>1$, the data provides more evidence for $M_1$ than for $M_2$.
• If $B{F}_{12}<1$, the data provides more evidence for $M_2$ than for $M_1$.
• If $B{F}_{12}\approx 1$, the data does not distinguish between the two models.

To make interpretation easier, statisticians have proposed scales to categorize the strength of evidence.

For a more detailed interpretation scale, let's consult this guide.

Posterior Odds

The Bayes Factor tells us what the data says. But what if we had prior beliefs about the models themselves? This is where Posterior Odds come in.

The relationship is simple and elegant:

If we have no reason to prefer one model over the other before seeing the data, we can set the prior odds to 1 ($P(M_1)=P(M_2)$). In this common case, the posterior odds are equal to the Bayes factor. This is why the Bayes factor is so often the primary focus of analysis.

4. Practical Computation and Challenges

This all sounds great in theory, but as we noted, computing the marginal likelihood integral is often very difficult. Let's briefly look at how it's done in practice.

To summarize the practical aspects:

• The Challenge: The marginal likelihood is hard to compute because it involves a high-dimensional integral. It's also very sensitive to the choice of priors on the parameters ($\theta$).
• Analytical Solutions: For simple conjugate models (like the Beta-Binomial coin-flip example), we can sometimes calculate the Bayes factor exactly.
• Numerical Methods: For most real-world models in machine learning, we rely on numerical methods like Sequential Monte Carlo (SMC), which can estimate the marginal likelihood as part of their sampling process.
• Shortcuts: For nested models (e.g., testing if a regression coefficient is exactly zero), the Savage-Dickey density ratio provides an easy way to compute the Bayes factor by comparing the prior and posterior distributions at that specific value.

Conclusion

Congratulations on completing the final lesson of the Bayesian methods module! You've moved from the fundamentals of Bayes' theorem to the sophisticated application of Bayesian model comparison.

Key Takeaways:

• Bayesian model comparison is framed as calculating the posterior probability of a model, $P(M|D)$.
• The key quantity is the marginal likelihood or model evidence, $P(D|M)$, which measures how well a model predicts the data on average over all its possible parameter values.
• Model evidence naturally embodies Occam's Razor, automatically penalizing models for unnecessary complexity.
• The Bayes factor is the ratio of two models' evidences and quantifies the support from the data for one model over another.
• Posterior odds combine the Bayes factor with our prior beliefs about the models, giving a complete Bayesian comparison.

Preview of the Next Module:

In this module, we have explored a comprehensive framework for reasoning about uncertainty and evidence. While powerful, full Bayesian inference can be computationally intensive. Many of the most widely used machine learning models today are trained not by analyzing a full posterior, but by optimizing a single objective.

In our next module, Optimization for Machine Learning, we will dive into this world. We'll begin by exploring the cornerstone of this process: the loss function. You will see deep connections to what you've learned here; for example, minimizing certain loss functions is mathematically equivalent to performing the Maximum Likelihood or MAP estimations we've already studied. This will bridge your knowledge from the probabilistic world to the optimization-centric view of machine learning.