MLE for Bernoulli Distribution

Introduction

In our previous lesson, we explored Bayesian inference and saw how it provides a complete posterior probability distribution for a parameter, such as a coin's bias $\theta$. This distribution represents our full belief, including our uncertainty. However, in many practical scenarios, we need to commit to a single best estimate for that parameter.

This lesson introduces a powerful and widely used method for finding such an estimate: Maximum Likelihood Estimation (MLE). Instead of the Bayesian approach of finding the posterior $P(\theta |D)$, MLE asks a more direct question: "What value of the parameter $\theta$ makes the data we observed, $D$, the most probable?" We will answer this question from first principles, culminating in a formal derivation of the MLE for the parameter of a Bernoulli distribution—the mathematical model for a single coin flip. This result is not just a statistical curiosity; it forms the bedrock of how we define and optimize many machine learning models.

The Principle of Maximum Likelihood

The core idea of MLE is to find the parameter values that maximize the likelihood of the observed data. To formalize this, we first define the likelihood function.

Given a dataset $D=\{y^{(1)},y^{(2)},\dots ,y^{(N)}\}$ of $N$ independent and identically distributed (i.i.d.) samples drawn from a distribution with parameter(s) $\theta$, the likelihood of the data is the joint probability of observing all these samples:

When we view this expression as a function of the parameter $\theta$ for a fixed dataset $D$, we call it the likelihood function, often denoted as $L(\theta ;D)$. The goal of MLE is to find the value of $\theta$ that maximizes this function. We use the $\arg \max$ notation to express this:

The notation $\hat{\theta }$ ("theta-hat") signifies an estimate of the true parameter $\theta$.

The following resource provides a clear introduction to this core concept.

From Likelihood to Log-Likelihood

Let's apply this to a concrete example: a sequence of $N$ coin flips, where each flip is a Bernoulli trial. Let's denote a "heads" outcome as $y=1$ and "tails" as $y=0$. The parameter of our Bernoulli distribution is $\varphi$, the probability of getting heads, so $P(Y=1)=\varphi$.

The probability mass function (PMF) for a single Bernoulli trial can be written compactly as:

Verify for yourself that this expression gives $\varphi$ for $y=1$ and $(1-\varphi )$ for $y=0$.

The likelihood function for $N$ i.i.d. trials is the product of the individual probabilities:

This product can be simplified by grouping the terms. If we have $N_H$ heads (where $y^{(i)}=1$) and $N_T$ tails (where $y^{(i)}=0$) in our dataset, the likelihood becomes:

Now, we face a practical challenge. Maximizing this function requires taking its derivative. Differentiating a product is cumbersome, and for large $N$, the product of probabilities (which are less than 1) can become vanishingly small, leading to numerical underflow on a computer.

Here is a question for you: What common mathematical function transforms products into sums, and how might that help us here?

This is the standard practice in MLE. We work with the log-likelihood function, denoted $\ell (\varphi ;D)$. Since the logarithm is a strictly monotonically increasing function, the value of $\varphi$ that maximizes the likelihood also maximizes the log-likelihood.

Let's derive the log-likelihood for our Bernoulli case:

Using the properties of logarithms ($\log (ab)=\log a+\log b$ and $\log (a^b)=b\log a$), we get:

This expression is much more tractable.

Deriving the Maximum Likelihood Estimate

We can now find the MLE by taking the derivative of the log-likelihood function with respect to $\varphi$, setting it to zero, and solving for $\varphi$.

Step 1: Differentiate $\ell (\varphi ;D)$ with respect to $\varphi$.
Using the chain rule, we know that $\frac{d}{dx}\log (x)=\frac{1}{x}$.

Step 2: Set the derivative to zero.
To find the value of $\varphi$ that maximizes the function, we find the critical point where the gradient is zero.

Step 3: Solve for $\hat{\varphi }$.

Finally, isolating $\hat{\varphi }$ gives us the Maximum Likelihood Estimate:

This elegant result confirms our intuition: the most likely value for the probability of heads is simply the proportion of heads observed in the data—the sample mean.

The following video and reading provide excellent, step-by-step walkthroughs of this exact derivation. Reviewing them will help solidify your understanding.

Conclusion

In this lesson, we established a cornerstone of statistical estimation and machine learning. We moved from the Bayesian concept of a full posterior distribution to the frequentist-style goal of finding a single best point estimate for a model parameter.

Key Takeaways:

• Maximum Likelihood Estimation (MLE) seeks the parameter value $\hat{\theta }$ that maximizes the likelihood function $L(\theta ;D)=p(D|\theta )$.
• We use the log-likelihood for computational stability (avoiding underflow) and mathematical convenience (turning products into sums).
• Maximizing the log-likelihood is equivalent to maximizing the likelihood because the logarithm is a monotonically increasing function.
• For a Bernoulli distribution, the MLE of the success probability $\varphi$ is the sample mean: the number of successes divided by the total number of trials (${\hat{\varphi }}_{\text{MLE}}=N_H/N$).

Preview of the Next Lesson:

The process of MLE involves defining and then optimizing an objective function—the log-likelihood. In machine learning, we more commonly frame optimization as minimizing a loss function. This is just a simple transformation: maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood.

In our next lesson, we will make a critical connection: we will show that minimizing the negative log-likelihood for a Bernoulli model is mathematically identical to minimizing the binary cross-entropy loss, a function you will encounter constantly in training neural networks for classification tasks. This will be our first major bridge from a classical statistical principle to a core component of modern deep learning.