MLE for Gaussian Parameters

Hello! Welcome to your next lesson on the theoretical foundations of AI.

In our last session, you successfully derived the Maximum Likelihood Estimator (MLE) for a Bernoulli distribution. This gave you a complete, hands-on application of the MLE framework for a discrete distribution with a single parameter. We followed the core "recipe": formulating the log-likelihood, differentiating, and solving for the parameter that maximizes the probability of our observed data.

Today, we will build directly on that foundation. Your learning outcome is to derive the MLE for the parameters (mean $\mu$ and variance ${\sigma }^2$) of a Gaussian distribution. This is a cornerstone of classical statistics and a fundamental building block for many machine learning models.

This lesson will introduce a key new element: optimizing a function with respect to multiple parameters. This will require you to apply your knowledge of multivariate calculus by computing partial derivatives and solving a system of first-order conditions, just as we discussed in Modules 3 and 4.

1. The Gaussian Distribution and its Log-Likelihood

Let's begin. The Gaussian (or Normal) distribution is arguably the most important continuous probability distribution. Its probability density function (PDF) for a random variable $X$ with mean $\mu$ and variance ${\sigma }^2$ is:
$f(x|\mu ,{\sigma }^2)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)$
Our goal is to find the values for the parameters $\mu$ and ${\sigma }^2$ that best explain a given dataset.

Assume we have a dataset of $n$ observations, $D=\{x_1,x_2,\dots ,x_n\}$, which we assume are independent and identically distributed (i.i.d.) according to a Gaussian distribution with unknown $\mu$ and ${\sigma }^2$.

The following video provides a concise and rigorous derivation of the log-likelihood function for this scenario. It follows the exact same logical steps we used for the Bernoulli distribution: write the joint probability as a product (the likelihood), and then take the logarithm to simplify it into a sum.

Summarizing the Log-Likelihood Function

As derived in the video, the process unfolds as follows:

1. Likelihood Function: The joint PDF for our $n$ i.i.d. samples is the product of the individual PDFs:
   $L(\mu ,{\sigma }^2;D)={\prod }_{i=1}^n\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\right)$
   This can be grouped into:
   $L(\mu ,{\sigma }^2;D)={\left(\frac{1}{2\pi {\sigma }^2}\right)}^{n/2}\exp \left(-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2\right)$

2. Log-Likelihood Function: Taking the natural logarithm simplifies this expression significantly:
   $l(\mu ,{\sigma }^2;D)=\log (L(\mu ,{\sigma }^2;D))$
   $l(\mu ,{\sigma }^2;D)=\log \left({\left(\frac{1}{2\pi {\sigma }^2}\right)}^{n/2}\right)+\log \left(\exp \left(-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2\right)\right)$
   Applying logarithm rules, this becomes:
   $l(\mu ,{\sigma }^2;D)=-\frac{n}{2}\log (2\pi {\sigma }^2)-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2$
   Or, equivalently, by separating the ${\sigma }^2$ term:
   $l(\mu ,{\sigma }^2;D)=-\frac{n}{2}\log (2\pi )-\frac{n}{2}\log ({\sigma }^2)-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2$
   This final expression is our objective function. We need to find the $\mu$ and ${\sigma }^2$ that maximize it.

2. Deriving the MLE for Mean ($\mu$) and Variance (${\sigma }^2$)

To maximize a function of two variables, we find the partial derivatives with respect to each variable and set them to zero. This gives us a system of two equations with two unknowns ($\mu$ and ${\sigma }^2$), which we can then solve.

This procedure is demonstrated clearly in the remainder of the video and also laid out in the following text. The text offers a very clean, equation-based summary of the derivation.

Let's perform the derivation step-by-step.

Step 1: Differentiate with respect to $\mu$ and solve

We take the partial derivative of $l(\mu ,{\sigma }^2;D)$ with respect to $\mu$, treating ${\sigma }^2$ as a constant. The first two terms of the log-likelihood do not depend on $\mu$, so their derivatives are zero.

$$\frac{\partial l}{\partial \mu }=\frac{\partial }{\partial \mu }\left[-\frac{1}{2{\sigma }^2}\sum _{i=1}^n(x_i-\mu {)}^2\right]$$

Using the chain rule:

$$\frac{\partial l}{\partial \mu }=-\frac{1}{2{\sigma }^2}\sum _{i=1}^{n}2(x_i-\mu )\cdot (-1)=\frac{1}{{\sigma }^2}\sum _{i=1}^n(x_i-\mu )$$

Now, set the partial derivative to zero to find the critical point:

$$\frac{1}{{\sigma }^2}\sum _{i=1}^n(x_i-\mu )=0$$

Assuming ${\sigma }^2\ne 0$, this simplifies to:

$$\sum _{i=1}^n(x_i-\mu )=0⟹\sum _{i=1}^n{x}_i-\sum _{i=1}^n\mu =0$$ $$\sum _{i=1}^n{x}_i-n\mu =0⟹n\mu =\sum _{i=1}^n{x}_i$$

This gives us the MLE for the mean, ${\hat{\mu }}_{MLE}$:

$${\hat{\mu }}_{MLE}=\frac{1}{n}\sum _{i=1}^n{x}_i$$

This is a very intuitive result: the most likely value for the population mean is the sample mean of the data.

Step 2: Differentiate with respect to ${\sigma }^2$ and solve

Next, we take the partial derivative of $l(\mu ,{\sigma }^2;D)$ with respect to ${\sigma }^2$, treating $\mu$ as a constant.

$$\frac{\partial l}{\partial {\sigma }^2}=\frac{\partial }{\partial {\sigma }^2}\left[-\frac{n}{2}\log (2\pi )-\frac{n}{2}\log ({\sigma }^2)-\frac{1}{2{\sigma }^2}\sum _{i=1}^n(x_i-\mu {)}^2\right]$$

Let's differentiate term by term:

$$\frac{\partial l}{\partial {\sigma }^2}=0-\frac{n}{2}\cdot \frac{1}{{\sigma }^2}-\left(\sum _{i=1}^n(x_i-\mu {)}^2\right)\cdot \frac{\partial }{\partial {\sigma }^2}\left(\frac{1}{2{\sigma }^2}\right)$$

Since $\frac{d}{dx}(x^{-1})=-x^{-2}$, we have $\frac{\partial }{\partial {\sigma }^2}\left(\frac{1}{2{\sigma }^2}\right)=\frac{1}{2}(-1)({\sigma }^2{)}^{-2}=-\frac{1}{2({\sigma }^2{)}^2}$.

$$\frac{\partial l}{\partial {\sigma }^2}=-\frac{n}{2{\sigma }^2}+\frac{1}{2({\sigma }^2{)}^2}\sum _{i=1}^n(x_i-\mu {)}^2$$

Set this to zero and solve for ${\sigma }^2$:

$$\frac{1}{2({\sigma }^2{)}^2}\sum _{i=1}^n(x_i-\mu {)}^2=\frac{n}{2{\sigma }^2}$$

Multiply both sides by $2({\sigma }^2{)}^2$:

$$\sum _{i=1}^n(x_i-\mu {)}^2=n{\sigma }^2$$

This gives us the MLE for the variance. To get the final estimator, we plug in the value we already found for the mean, ${\hat{\mu }}_{MLE}$:

$${\hat{\sigma }}_{MLE}^2=\frac{1}{n}\sum _{i=1}^n(x_i-{\hat{\mu }}_{MLE}{)}^2$$

The maximum likelihood estimate for the variance is the sample variance, calculated using the MLE for the mean.

Test your understanding!

Suppose you measure the height of 4 people from a certain population and get the following data (in cm): $D=\{170,160,180,170\}$. Assuming the heights are normally distributed, what are the maximum likelihood estimates for the mean and variance of the population's height?

Show answer

First, we calculate the MLE for the mean, ${\hat{\mu }}_{MLE}$, which is the sample mean:

$${\hat{\mu }}_{MLE}=\frac{170+160+180+170}{4}=\frac{680}{4}=170\text{ cm}$$

Next, we use this value to calculate the MLE for the variance, ${\hat{\sigma }}_{MLE}^2$:

$${\hat{\sigma }}_{MLE}^2=\frac{1}{4}\left[(170-170{)}^2+(160-170{)}^2+(180-170{)}^2+(170-170{)}^2\right]$$ $${\hat{\sigma }}_{MLE}^2=\frac{1}{4}\left[0^2+(-10{)}^2+(10{)}^2+0^2\right]=\frac{1}{4}[0+100+100+0]=\frac{200}{4}=50{\text{ cm}}^2$$

So, the MLEs are ${\hat{\mu }}_{MLE}=170$ and ${\hat{\sigma }}_{MLE}^2=50$.

A Note on Rigor and Bias

For full mathematical rigor, one would need to compute the Hessian matrix (the matrix of second-order partial derivatives) and check that it is negative definite at this critical point, confirming it is a maximum. For the Gaussian log-likelihood, this is indeed the case.

It is also important to note a key property of the variance estimator we just derived. The estimator ${\hat{\sigma }}_{MLE}^2=\frac{1}{n}\sum (x_i-\hat{\mu }{)}^2$ is a biased estimator of the true population variance. The unbiased estimator, which you may have seen elsewhere, uses a denominator of $n-1$. However, the MLE procedure correctly yields the $1/n$ version. While biased, ${\hat{\sigma }}_{MLE}^2$ is consistent, meaning the bias approaches zero as the sample size $n$ grows.

Conclusion

Excellent work. You have now successfully derived the Maximum Likelihood Estimators for the two parameters of a Gaussian distribution. This process reinforced the MLE recipe and extended it to a multi-parameter, continuous case.

Key Takeaways:

• The MLE framework is general and applies to both discrete and continuous distributions with any number of parameters.
• For multi-parameter problems, finding the MLE involves taking partial derivatives of the log-likelihood with respect to each parameter and solving the resulting system of equations.
• The MLE for the mean ($\mu$) of a Gaussian distribution is the sample mean.
• The MLE for the variance (${\sigma }^2$) is the sample variance with a $1/n$ denominator, which is a biased but consistent estimator.

Preview of the Next Lesson:
So far, our estimation has been driven purely by the data. The principle of Maximum Likelihood finds the parameters that make the observed data most probable. In our next lesson, we will introduce Maximum a Posteriori (MAP) estimation. MAP extends MLE by allowing us to incorporate prior beliefs about the parameters into the estimation process. This is a crucial step toward Bayesian inference and will provide a richer framework for statistical modeling.