Maximum Likelihood Estimation for Simple Models

The 5-Step Recipe for Finding MLEs

The process of deriving a Maximum Likelihood Estimator analytically can be broken down into five clear steps. We'll use this recipe for both of our examples today.

1. Define the Model and its Probability Function: Start with the probability mass function (PMF) for discrete data or the probability density function (PDF) for continuous data of a single observation, $f(x_i;\theta )$, where $\theta$ is the parameter we want to estimate.
2. Construct the Likelihood Function: Assuming the data points are independent and identically distributed (IID), the likelihood of observing the entire dataset is the product of the individual probabilities: $L(\theta )={\prod }_{i=1}^{n}f(x_i;\theta )$.
3. Construct the Log-Likelihood Function: Take the natural logarithm of the likelihood function. This converts the product into a more manageable sum: $\ell (\theta )=\log L(\theta )={\sum }_{i=1}^n\log f(x_i;\theta )$.
4. Differentiate and Set to Zero: Find the derivative of the log-likelihood function with respect to the parameter $\theta$. Set this derivative equal to zero to find the point of maximum likelihood: $\frac{d\ell (\theta )}{d\theta }=0$.
5. Solve for the Parameter: Algebraically solve the equation from the previous step for $\theta$. The solution, denoted $\hat{\theta }$, is the Maximum Likelihood Estimator.

Now, let's put this recipe into practice.

Example 1: MLE for the Bernoulli Distribution

The Bernoulli distribution models a single trial with a binary outcome: success (1) or failure (0). The single parameter to estimate is $p$, the probability of success. Think of this as modeling a coin flip, a user clicking on an ad, or a patient responding to a treatment.

Let's derive the MLE for $p$. The following video provides a clear, step-by-step walkthrough of the entire process. I've broken down the key sections for you to watch.

As a summary of what you just watched, we start with the PMF for a single observation $x_i$:
$f(x_i;p)=p^{x_i}(1-p{)}^{1-x_i}$

The likelihood for $n$ observations is:
$L(p)={\prod }_{i=1}^n{p}^{x_i}(1-p{)}^{1-x_i}=p^{\sum x_i}(1-p{)}^{n-\sum x_i}$

And the log-likelihood is:
$\ell (p)=\log (p){\sum }_{i=1}^n{x}_i+\log (1-p)\left(n-{\sum }_{i=1}^n{x}_i\right)$

Now for the calculus. The next part of the video shows how to differentiate this function and solve for $p$.

The derivative of the log-likelihood with respect to $p$ is:
$\frac{d\ell (p)}{dp}=\frac{\sum x_i}{p}-\frac{n-\sum x_i}{1-p}$

Setting this to zero and solving for $p$ gives us the Maximum Likelihood Estimator, $\hat{p}$:
${\hat{p}}_{\text{MLE}}=\frac{{\sum }_{i=1}^n{x}_i}{n}=\bar{x}$

This is a powerful result. It provides a formal justification for what feels intuitive: the best estimate for the probability of success in a population is simply the proportion of successes you observed in your sample.

Example 2: MLE for the Normal Distribution

Now, let's tackle a continuous distribution. The Normal (or Gaussian) distribution is defined by two parameters: the mean $\mu$ and the variance ${\sigma }^2$. We'll derive the MLEs for both. Given your engineering background, you've likely seen this distribution countless times; now we'll prove why the sample mean and variance are its natural estimators.

The math for the Normal distribution is more involved, but the logic is identical. We will again lean on a StatQuest video to make the derivation clear.

The log-likelihood function for the Normal distribution, after simplification, is:
$\ell (\mu ,{\sigma }^2)=-\frac{n}{2}\log (2\pi )-\frac{n}{2}\log ({\sigma }^2)-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2$
This expression is much easier to differentiate than the original likelihood function. Since we have two parameters, we will need to take partial derivatives—one with respect to $\mu$ and another with respect to ${\sigma }^2$.

Finding the MLE for the Mean ($\mu$)

Let's find the estimator for the mean first.

When we take the partial derivative with respect to $\mu$ and set it to 0, we get:
$\frac{\partial \ell }{\partial \mu }=\frac{1}{{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu )=0$
Solving for $\mu$ gives us the incredibly familiar result:
${\hat{\mu }}_{\text{MLE}}=\frac{{\sum }_{i=1}^n{x}_i}{n}=\bar{x}$
Once again, the MLE for the population mean $\mu$ is simply the sample mean $\bar{x}$.

Finding the MLE for the Variance (${\sigma }^2$)

Now let's do the same for the variance.

Taking the partial derivative with respect to ${\sigma }^2$ and setting it to 0 yields:
$\frac{\partial \ell }{\partial {\sigma }^2}=-\frac{n}{2{\sigma }^2}+\frac{1}{2({\sigma }^2{)}^2}{\sum }_{i=1}^n(x_i-\mu {)}^2=0$
When we solve this for ${\sigma }^2$ (and plug in our already-derived estimator $\hat{\mu }=\bar{x}$), we get:
${\hat{\sigma }}_{\text{MLE}}^2=\frac{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}{n}$
This is the average of the squared differences from the sample mean. You might have seen a version of sample variance with $n-1$ in the denominator. That is the unbiased sample variance, a concept we will explore later. For now, the key insight is that MLE provides a direct derivation for this form of the variance estimator.

Summary and When to Go Numerical

In many cases for common distributions, we can find a closed-form solution for the MLE, as we did today. The following article provides a nice summary table of these solutions.

However, for more complex models, such as mixture models or many neural network architectures, solving $\frac{d\ell (\theta )}{d\theta }=0$ analytically is impossible. In those situations, we rely on numerical optimization algorithms, like Gradient Descent, to iteratively find the parameter values that minimize the Negative Log-Likelihood. This connects directly back to your goal of understanding how ML models are trained.

Conclusion

Congratulations on working through these derivations! This is a significant step in connecting formal statistical theory to practical data modeling.

Key Takeaways:

• We have a 5-step recipe to derive Maximum Likelihood Estimators: define the model, build the likelihood and log-likelihood, differentiate, and solve.
• For a Bernoulli distribution, the MLE of the success probability $p$ is the sample proportion of successes.
• For a Normal distribution, the MLE of the mean $\mu$ is the sample mean, and the MLE of the variance ${\sigma }^2$ is the sample variance with a denominator of $n$.
• The MLE framework provides a rigorous, principled foundation for using these intuitive sample statistics to estimate population parameters.

Preview of the Next Lesson:

This lesson concludes our module on statistical inference and estimation. You've learned how to formulate hypotheses, construct likelihood functions, understand the principle of MLE, and now apply it.

Our next module, "Optimization for Machine Learning," is the perfect follow-up. We will begin by formally defining the role of a loss function in quantifying model error. Having just seen how minimizing the Negative Log-Likelihood is a primary goal in MLE, you are perfectly positioned to understand how loss functions drive the entire training process in machine learning.