MAP Estimation for Bernoulli Parameter with Beta Prior

Hello! Welcome back to our series on the mathematical foundations of AI.

In our last lesson, you formulated the principle of Maximum a Posteriori (MAP) estimation. We established that MAP extends Maximum Likelihood Estimation (MLE) by incorporating prior beliefs about a parameter $\theta$. The goal is to maximize the posterior distribution, which we found is equivalent to maximizing the sum of the log-likelihood and the log-prior:

Today, we will put this principle into practice. Your learning outcome is to derive the MAP estimate for a Bernoulli parameter with a Beta prior and compare it to the MLE result. This is a classic and fundamentally important example in Bayesian statistics, and mastering this derivation will give you a concrete understanding of how prior beliefs mathematically combine with data.

1. The Components: Bernoulli Likelihood and Beta Prior

To derive the MAP estimate, we first need to define its two components: the likelihood of our data and the prior distribution for the parameter.

The Likelihood

Let's consider a series of $N$ independent Bernoulli trials (e.g., coin flips). We observe $n_H$ successes (heads) and $n_T$ failures (tails), where $N=n_H+n_T$. The parameter we want to estimate is $\theta$, the probability of success. The likelihood function for this data, which you've worked with before, is given by the Binomial probability mass function:

Since we are maximizing with respect to $\theta$, the binomial coefficient $\left(\genfrac{}{}{0ex}{}{N}{n_H}\right)$ is a constant and can be ignored. Thus, the likelihood is proportional to:

The Prior: Introducing the Beta Distribution

Next, we need to choose a prior distribution, $p(\theta )$, that reflects our belief about $\theta$ before seeing any data. Since $\theta$ is a probability, its value must lie in the interval $[0,1]$. A highly flexible and mathematically convenient probability distribution for variables on this interval is the Beta distribution.

The Beta distribution is defined by two positive shape parameters, $\alpha$ and $\beta$, which are called hyperparameters. Its probability density function (PDF) is:

where $B(\alpha ,\beta )$ is the Beta function, which acts as a normalizing constant to ensure the total probability is 1. For our purposes, we can write this as:

The hyperparameters $\alpha$ and $\beta$ allow us to encode our prior beliefs. For example:

• Beta(1, 1) is a uniform distribution, expressing no prior preference.
• Beta(5, 5) is a bell-shaped curve centered at 0.5, expressing a belief that $\theta$ is likely near 0.5.
• Beta(5, 2) would be skewed towards higher values of $\theta$.

The following video provides a concise introduction to the Beta distribution.

2. Conjugate Priors: A Mathematical Convenience

An extremely useful property in Bayesian analysis is conjugacy. A prior distribution is said to be conjugate to a likelihood function if the resulting posterior distribution belongs to the same family of distributions as the prior.

As it turns out, the Beta distribution is the conjugate prior for the Bernoulli/Binomial likelihood. This means:

If Prior is Beta($\theta ;\alpha ,\beta$) and Likelihood is Bernoulli/Binomial($D|\theta$), then the Posterior will also be a Beta distribution.

This is not just a neat trick; it simplifies Bayesian analysis immensely by providing a closed-form expression for the posterior. Instead of complex numerical integration, we can find the posterior by simply updating the hyperparameters of our prior.

As the video shows, if our prior is Beta($\alpha, \beta$) and our data has $n_H$ successes and $n_T$ failures, the posterior distribution is Beta($\alpha + n_H, \beta + n_T$). This is a powerful result.

3. Deriving the MAP Estimate

Now we have all the pieces to perform the derivation. Our goal is to find the $\theta$ that maximizes the posterior distribution. Since we know the posterior is Beta($\alpha + n_H, \beta + n_T$), the MAP estimate is simply the mode (the peak) of this posterior Beta distribution.

While we could look up the formula for the mode of a Beta distribution, your goal is to derive it from first principles using the MAP objective function.

Let's walk through the key steps outlined in the resource for clarity:

1. Form the log-posterior:
   We start with the objective ${\hat{\theta }}_{MAP}=\arg {\max }_{\theta }[\log (p(D|\theta ))+\log (p(\theta ))]$.

   • Log-Likelihood: $\log (p(D|\theta ))=n_H\log \theta +n_T\log (1-\theta )+\text{constant}$.
   • Log-Prior: $\log (p(\theta ))=(\alpha -1)\log \theta +(\beta -1)\log (1-\theta )+\text{constant}$.

   Combining these (and ignoring constants that don't depend on $\theta$), the function to maximize is:

   $$J(\theta )=(n_H+\alpha -1)\log \theta +(n_T+\beta -1)\log (1-\theta )$$

2. Differentiate with respect to $\theta$:

   $$\frac{dJ}{d\theta }=\frac{n_H+\alpha -1}{\theta }-\frac{n_T+\beta -1}{1-\theta }$$

3. Set the derivative to zero and solve for $\theta$:

   $$\frac{n_H+\alpha -1}{\theta }=\frac{n_T+\beta -1}{1-\theta }$$ $$(1-\theta )(n_H+\alpha -1)=\theta (n_T+\beta -1)$$ $$(n_H+\alpha -1)-\theta (n_H+\alpha -1)=\theta (n_T+\beta -1)$$ $$n_H+\alpha -1=\theta (n_T+\beta -1+n_H+\alpha -1)$$

   Rearranging and noting $N=n_H+n_T$, we get our final result:

   $${\hat{\theta }}_{MAP}=\frac{n_H+\alpha -1}{N+\alpha +\beta -2}$$

This is the MAP estimate for the parameter of a Bernoulli distribution with a Beta($\alpha, \beta$) prior.

4. Comparison with MLE

Now, let's formally compare the MAP and MLE results.

• MLE: ${\hat{\theta }}_{MLE}=\frac{n_H}{N}$
• MAP: ${\hat{\theta }}_{MAP}=\frac{n_H+\alpha -1}{N+\alpha +\beta -2}$

The following reading provides a fantastic discussion of this comparison.

Based on this resource and our derivation, we can draw several key conclusions:

1. Interpretation of Hyperparameters: The MAP formula provides a beautiful intuition. The estimate is like the MLE, but we add $\alpha -1$ "pseudo-counts" to the number of heads and $\beta -1$ "pseudo-counts" to the number of tails. The prior acts as if we had already observed $\alpha +\beta -2$ imaginary trials.

2. Regularization: The prior acts as a regularizer. If we have a small dataset (e.g., 3 flips, 3 heads), the MLE is an extreme ${\hat{\theta }}_{MLE}=1$. A reasonable prior (e.g., Beta(2,2), which adds 1 pseudo-head and 1 pseudo-tail) gives a more moderate ${\hat{\theta }}_{MAP}=(3+1)/(3+2)=0.8$. The prior "pulls" the estimate away from extreme values, preventing overfitting on small data.

3. Asymptotic Equivalence: As the amount of data grows (i.e., $N\to \infty$), the influence of the fixed hyperparameters $\alpha$ and $\beta$ diminishes. The $n_H$ and $N$ terms dominate the numerator and denominator, respectively. Therefore:

   $$\underset{N\to \infty }{\lim }{\hat{\theta }}_{MAP}=\underset{N\to \infty }{\lim }\frac{n_H+\alpha -1}{N+\alpha +\beta -2}\approx \frac{n_H}{N}={\hat{\theta }}_{MLE}$$

   With enough data, the evidence from the data overwhelms the prior belief, and the MAP and MLE estimates converge.

4. MLE as a Special Case: If we choose a Beta(1,1) prior, which is a uniform distribution over $[0,1]$, this represents a "maximally uninformative" prior belief. Let's plug $\alpha =1$ and $\beta =1$ into our MAP formula:

   $${\hat{\theta }}_{MAP}=\frac{n_H+1-1}{N+1+1-2}=\frac{n_H}{N}={\hat{\theta }}_{MLE}$$

   This formally proves what we discussed in the last lesson: MLE is a special case of MAP with a uniform prior.

Conclusion

In this lesson, you have successfully moved from the abstract principle of MAP to a concrete, rigorous derivation for a fundamental problem in statistics. This example is a cornerstone of Bayesian inference and illustrates the elegant interplay between prior knowledge and observed data.

Key Takeaways:

• The Beta distribution is the conjugate prior for the Bernoulli/Binomial likelihood, meaning a Beta prior combined with a Bernoulli likelihood yields a Beta posterior.
• The MAP estimate for a Bernoulli parameter $\theta$ with a Beta($\alpha, \beta$) prior is the mode of the posterior distribution, which we derived to be ${\hat{\theta }}_{MAP}=\frac{n_H+\alpha -1}{N+\alpha +\beta -2}$.
• The MAP estimate can be interpreted as an MLE with $\alpha -1$ pseudo-successes and $\beta -1$ pseudo-failures added to the data.
• The prior acts as a regularizer, making MAP estimates more robust than MLEs for small datasets.
• As the amount of data increases, the MAP estimate converges to the MLE estimate.

Preview of the Next Lesson:
We have now concluded our exploration of parameter estimation, covering MLE and MAP. We will now turn to another major pillar of statistical inference: hypothesis testing. In the next lesson, you will learn to formulate the framework for statistical hypothesis testing, including null and alternative hypotheses, and understand the different types of errors we can make.