Maximum A Posteriori (MAP) Estimation: Fundamentals

Recommended resources

The freely available resources this lesson will direct you to.

Hello! Welcome to your next lesson in our journey through the theoretical foundations of AI.

In our last few lessons, we've focused on Maximum Likelihood Estimation (MLE), culminating in the derivation of MLEs for the parameters of a Gaussian distribution. You've seen that the core idea of MLE is to find the parameter values that make our observed data most probable. This approach is powerful, but it relies solely on the data at hand.

Today, we move to a more general and often more robust principle: Maximum a Posteriori (MAP) estimation. Your learning outcome for this lesson is to formulate the principle of MAP estimation. We will explore how MAP extends MLE by incorporating prior beliefs into the estimation process, a foundational concept in Bayesian statistics. This will give you a more powerful lens through which to view parameter estimation and will directly connect to many advanced machine learning concepts.

1. From MLE to MAP: The Need for Prior Knowledge

Maximum Likelihood Estimation works wonderfully when we have a large amount of data. However, with small or unrepresentative datasets, it can lead to questionable or extreme conclusions.

To see why, let's consider a simple coin-flipping experiment. Imagine you flip a coin 3 times and get heads every time. What is the MLE for the probability of getting heads, $θ$ ? As you derived for the Bernoulli distribution, it would be $θ^_{M L E} = 3/3 = 1$ . This estimate implies that tails are impossible, which seems overly confident based on just three flips. Our intuition suggests that the coin is probably closer to fair, and we just got a slightly unusual result.

This is where MAP estimation comes in. It provides a formal, mathematical way to incorporate this kind of "prior intuition." The following video provides an excellent introduction to this idea.

Bayesian Maximum Aposteriori Estimation (MAP): Extending Maximum Likelihood Estimation

This video from Steve Brunton's channel begins by highlighting the fragility of MLE with small datasets using the exact coin-flip example we just discussed. This serves as a strong motivation for why we need a different approach.

Please watch the first part of the video, from 00:00 to 04:11. Focus on understanding how MLE can produce 'bad' estimates from 'unlucky' data draws and how the proposed solution involves using Bayes' theorem to incorporate prior knowledge.

The key takeaway is that MLE can be sensitive to small sample sizes and can lead to overfitting. MAP estimation addresses this by introducing a mechanism to temper the influence of the data with our prior beliefs.

2. The Bayesian Framework: Parameters as Random Variables

To incorporate prior beliefs mathematically, we must make a fundamental conceptual shift.

In the Frequentist framework (which underlies MLE), the parameter $θ$ is considered a fixed, unknown constant. We use data to find a point estimate for this constant.
In the Bayesian framework (which underlies MAP), the parameter $θ$ is treated as a random variable. It has its own probability distribution, which reflects our uncertainty about its true value.

This shift allows us to talk about the probability of the parameter itself. Our beliefs are captured in two key distributions:

Prior Distribution, $p (θ)$ : This distribution represents our belief about the parameter $θ$ before we observe any data. For the coin example, we might choose a prior that places a high probability on $θ$ being close to 0.5.
Posterior Distribution, $p (θ ∣ D)$ : This is our updated belief about $θ$ after observing the data $D$ . It is the combination of our prior belief and the evidence provided by the data.

The goal of Bayesian inference is to calculate this posterior distribution, which represents our complete knowledge about the parameter. MAP estimation is then the process of finding the single most probable value of the parameter according to this posterior distribution.

The following resource provides a clear explanation of this Bayesian perspective.

7.5: Maximum A Posteriori Estimation

These notes from a Stanford course, titled 'Maximum A Posteriori Estimation', introduce the Bayesian framework and the intuition behind treating the parameter as a random variable.

Read the sections 'Maximum A Posteriori (MAP) Estimation' (7.5.1) and 'Intuition' (7.5.1.1). Focus on the distinction between the Frequentist and Bayesian frameworks and how the concept of a 'belief distribution' (the prior) is introduced.

3. Formulating the MAP Estimate using Bayes' Theorem

The bridge from our prior belief to our posterior belief is Bayes' Theorem. For a parameter $θ$ and data $D$ , it is written as:

p (θ ∣ D) = \frac{p ( D ∣ θ ) p ( θ )}{p ( D )}

Let's break this down:

$p (θ ∣ D)$ is the Posterior: The probability of the parameter given the data. This is what we want to maximize.
$p (D ∣ θ)$ is the Likelihood: The probability of the data given the parameter. This is the same likelihood function, $L (θ; D)$ , that we maximized in MLE.
$p (θ)$ is the Prior: Our initial belief about the distribution of the parameter.
$p (D)$ is the Evidence (or Marginal Likelihood): The probability of the data, averaged over all possible parameter values. It is calculated as $p (D) = \int p (D ∣ θ) p (θ) d θ$ .

The Maximum a Posteriori (MAP) estimate is the value of $θ$ that maximizes the posterior probability distribution.

θ^_{M A P} = ar g θ max p (θ ∣ D)

Now, let's substitute the Bayes' theorem formula into this expression:

θ^_{M A P} = ar g θ max \frac{p ( D ∣ θ ) p ( θ )}{p ( D )}

A crucial insight for optimization is that the denominator, $p (D)$ , does not depend on $θ$ . It is a normalizing constant that ensures the posterior distribution integrates to 1. Since it is constant with respect to $θ$ , it does not affect the location of the maximum. Therefore, we can remove it from the maximization problem.

This simplifies the MAP formulation significantly:

θ^_{M A P} = ar g θ max [p (D ∣ θ) p (θ)]

The following resources walk through this derivation formally and provide further intuition.

Bayesian Maximum Aposteriori Estimation (MAP): Extending Maximum Likelihood Estimation

Let's return to the Steve Brunton video, which now formalizes the MAP principle using Bayes' theorem.

Watch from 04:11 to 10:32. This section clearly derives the MAP objective function and explains why the evidence term, P(x), can be ignored during the maximization over the parameter heta.

7.5: Maximum A Posteriori Estimation

The Stanford notes also provide a concise and rigorous mathematical derivation of the MAP estimator. This will solidify your understanding of the formula.

Read the subsection 'Derivation' (7.5.1.2). It formally presents the definition of the MAP estimator and shows how it arises from maximizing the posterior.

Visually, the MAP estimate is simply the mode (the peak) of the posterior distribution, as shown in the image below.

MAP Estimate for Bernoulli Parameter with Beta Prior — This graph shows the posterior distribution of a coin's probability of heads, $p$, given 7 heads and 3 tails, using a Beta(2,2) prior. The MAP estimate is the value of $p$ at the peak of this curve, $\hat{p}_{MAP} \approx 0.67$, which is less extreme than the MLE of 0.7.

The Log-Posterior

Just as with MLE, it is often mathematically more convenient to maximize the logarithm of the objective function. Since the logarithm is a strictly increasing function, maximizing $lo g (f (x))$ is equivalent to maximizing $f (x)$ . Applying this to the MAP objective:

θ^_{M A P} = ar g θ max lo g [p (D ∣ θ) p (θ)]

Using the properties of logarithms, this becomes:

θ^_{M A P} = ar g θ max [lo g (p (D ∣ θ)) + lo g (p (θ))]

This final form is very revealing. The MAP objective is the sum of the log-likelihood (the same term we maximized in MLE) and the log-prior. The log-prior term acts as a regularizer, penalizing parameter values that are considered less likely according to our prior belief. This formally connects Bayesian estimation to the concept of regularization that you've likely encountered in your practical machine learning work.

Test your understanding!

You are trying to find the MAP estimate for a parameter $θ$ . You have calculated the log-likelihood as $l (θ; D) = - (θ - 5)^{2}$ and you have a prior belief that $θ$ is close to 0, which you model with a log-prior of $lo g (p (θ)) = - 2 θ^{2}$ . What is the MAP objective function you need to maximize? What is the qualitative difference between the value that maximizes this objective versus the value that maximizes the log-likelihood alone?

Show answer

The MAP objective is the sum of the log-likelihood and the log-prior:

J_{M A P} (θ) = l (θ; D) + lo g (p (θ)) = - (θ - 5)^{2} - 2 θ^{2}

The MLE would simply maximize $l (θ; D) = - (θ - 5)^{2}$ , which is maximized at $θ^_{M L E} = 5$ .

The MAP estimate will maximize $J_{M A P} (θ)$ . The term $- 2 θ^{2}$ from the prior penalizes large values of $θ$ . Therefore, the MAP estimate will be "pulled away" from the MLE of 5 and shifted closer to 0, balancing the evidence from the data (which points to 5) with the prior belief (which favors 0).

4. MAP and its Relation to MLE

The connection between MAP and MLE is deep. As we've seen, MAP = MLE + Prior. This means that if we choose a specific type of prior, MAP estimation can become equivalent to MLE.

This happens when we use a uniform prior. A uniform prior, $p (θ) = c$ for some constant $c$ , expresses maximal uncertainty or "no preference" for any particular parameter value. In this case, the log-prior, $lo g (p (θ)) = lo g (c)$ , is also a constant. Since constants don't affect maximization, the MAP objective becomes:

θ^_{M A P} = ar g θ max [lo g (p (D ∣ θ)) + constant] = ar g θ max lo g (p (D ∣ θ)) = θ^_{M L E}

Thus, MLE can be viewed as a special case of MAP estimation with a uniform prior.

Bayesian Maximum Aposteriori Estimation (MAP): Extending Maximum Likelihood Estimation

The final part of the Brunton video summarizes this important relationship and discusses the role of the prior.

Watch from 10:32 to 12:45. This part explicitly discusses that MLE is a special case of MAP when the prior is 'maximally uninformative' (i.e., uniform).

Finally, for mathematical rigor, it's important to be precise about the notation used in Bayesian analysis. The following resource clarifies two key points: the meaning of the proportionality symbol ( $\propto$ ) and the difference between probability and probability density, which is crucial when dealing with continuous parameters.

CPSC 440: Machine Learning

These slides from a UBC course on Machine Learning contain excellent 'digressions' that clarify common points of confusion in Bayesian statistics. Given your goal of high mathematical rigor, these are essential reading.

Please review the two sections titled 'Digression: “Proportional to” (∝) Notation' and 'Digression2: “Probability” vs. “Probability Density”'. These will clarify why we can write p( heta | D) \propto p(D | heta) p( heta) and the formal meaning of p( heta) when heta is a continuous variable.

Conclusion

In this lesson, you have formulated the principle of Maximum a Posteriori estimation. You've made the conceptual leap from the frequentist world of MLE to the Bayesian world of priors and posteriors, providing you with a more general and powerful framework for statistical estimation.

Key Takeaways:

MAP estimation extends MLE by incorporating prior beliefs about parameters, making it more robust, especially with small datasets.
This is achieved by adopting a Bayesian perspective, where parameters are treated as random variables that have their own probability distributions.
The MAP estimate is the mode of the posterior distribution $p (θ ∣ D)$ .
Using Bayes' Theorem, the MAP objective is to find the $θ$ that maximizes the product of the likelihood and the prior: $θ^_{M A P} = ar g max_{θ} [p (D ∣ θ) p (θ)]$ .
In log-space, this is equivalent to maximizing the sum of the log-likelihood and the log-prior. The log-prior acts as a regularizer.
MLE is a special case of MAP estimation where the prior distribution is uniform.

Preview of the Next Lesson:
With the principle of MAP now firmly established, our next step is to apply it. In the next lesson, we will derive the MAP estimate for the parameter of a Bernoulli distribution using a Beta prior. This is a classic and foundational example that will make these concepts concrete and prepare you for applying MAP in more complex models.