The Principle of Maximum Likelihood Estimation

Hello! Welcome to your next lesson on the theoretical foundations of AI.

In our last lesson, we established the criteria for evaluating statistical estimators: unbiasedness, consistency, and efficiency. We now know what makes a "good" estimator, but we haven't yet discussed a general method for finding them. This lesson addresses that very question.

Your learning outcome today is to formulate the principle of Maximum Likelihood Estimation (MLE). MLE is arguably the most important and widely used method for parameter estimation in statistics and machine learning. We will define its core components, understand its mathematical formulation, and, to satisfy your preference for rigor, explore its deep connection to information theory.

1. The Core Principle: What Parameters Make My Data Most Probable?

Imagine you have a set of observed data points. You assume these points were generated by a probability distribution from a certain family (e.g., Gaussian, Bernoulli), but you don't know the specific parameters of that distribution (e.g., the mean $\mu$ and variance ${\sigma }^2$, or the probability of success $p$).

The central idea of Maximum Likelihood Estimation is simple yet powerful: Let's choose the parameter values that make our observed data the most probable, or "most likely," to have occurred.

To formalize this, we need to define a function that quantifies this "likelihood." The following video provides an excellent introduction to this concept, explaining how we construct the likelihood function from our data and why we want to maximize it.

From Probability to Likelihood

As the video explained, the key is to shift our perspective. Let's make this concrete.
Suppose we have a set of $n$ i.i.d. data points $x_1,x_2,\dots ,x_n$, drawn from a distribution with a probability density/mass function $f(x;\theta )$, where $\theta$ is the unknown parameter we want to estimate.

• The joint probability of observing this specific dataset is, due to independence, the product of the individual probabilities:
  $P(x_1,\dots ,x_n;\theta )={\prod }_{i=1}^{n}f(x_i;\theta )$

• The likelihood function, denoted $L(\theta |\mathbf{x})$, is mathematically identical to this joint probability function. However, the interpretation is reversed. We treat the data $\mathbf{x}=(x_1,\dots ,x_n)$ as fixed and view this expression as a function of the parameter $\theta$.
  $L(\theta |\mathbf{x})=f(\mathbf{x};\theta )={\prod }_{i=1}^{n}f(x_i;\theta )$

The principle of maximum likelihood estimation states that our best estimate for $\theta$ is the value that maximizes this function. We call this the Maximum Likelihood Estimate (MLE), denoted ${\hat{\theta }}_{MLE}$.

2. A Practical Improvement: The Log-Likelihood

Maximizing the likelihood function $L(\theta |\mathbf{x})$ directly can be challenging. It's a product of many terms, which can be numerically unstable (leading to underflow if probabilities are small) and difficult to differentiate.

Fortunately, there is a simple and elegant solution: we can maximize the natural logarithm of the likelihood instead. This is valid because the logarithm is a monotonically increasing function. This means that if $L({\theta }_A)>L({\theta }_B)$, then $\log (L({\theta }_A))>\log (L({\theta }_B))$. The value of $\theta$ that maximizes the likelihood is the same value that maximizes the log-likelihood.

Let's continue with the same video to see this explained.

As the video demonstrated, the log-likelihood function, denoted $l(\theta |\mathbf{x})$ or $\log L(\theta |\mathbf{x})$, is:
$l(\theta |\mathbf{x})=\log \left({\prod }_{i=1}^{n}f(x_i;\theta )\right)={\sum }_{i=1}^n\log f(x_i;\theta )$

This transformation from a product to a sum makes finding the maximum much easier, as sums are far simpler to differentiate than products. The MLE is therefore found by solving:
${\hat{\theta }}_{MLE}=\arg {\max }_{\theta }l(\theta |\mathbf{x})=\arg {\max }_{\theta }{\sum }_{i=1}^n\log f(x_i;\theta )$

In practice, for differentiable functions, this is usually done by taking the derivative of the log-likelihood with respect to $\theta$, setting it to zero, and solving for $\theta$. This connects directly to the optimization theory we covered in Module 4.

3. The Theoretical Justification: Minimizing KL Divergence

So far, the principle of MLE seems intuitive. But is it theoretically sound? The answer is a resounding yes, and its justification lies in information theory, specifically in the concept of Kullback-Leibler (KL) Divergence.

KL Divergence, $D_{KL}(P||Q)$, measures how one probability distribution $P$ differs from a second, reference probability distribution $Q$. A key property is that $D_{KL}(P||Q)\ge 0$, and $D_{KL}(P||Q)=0$ if and only if $P$ and $Q$ are identical.

In our estimation problem, there is a "true" but unknown data-generating distribution, let's call it $P_{{\theta }^{\ast }}$. We are trying to find the best approximation for it from our family of model distributions, $\{P_{\theta }\}$. Our goal is to find the $\theta$ that makes our model distribution $P_{\theta }$ "closest" to the true distribution $P_{{\theta }^{\ast }}$. KL Divergence gives us a way to measure this closeness.

The following text provides a rigorous derivation showing that minimizing the KL divergence between the true distribution and our model is equivalent to maximizing the likelihood.

Let's walk through the logic from the article. Our goal is to find the $\theta$ that minimizes $D_{KL}(P_{{\theta }^{\ast }}||P_{\theta })$.

1. Decompose the KL Divergence: The definition of KL divergence is an expectation taken with respect to the true distribution $P_{{\theta }^{\ast }}$:
   $D_{KL}(P_{{\theta }^{\ast }}||P_{\theta })={\mathbb{E}}_{x\sim P_{{\theta }^{\ast }}}\left[\log \frac{P_{{\theta }^{\ast }}(x)}{P_{\theta }(x)}\right]$
   We can split the logarithm:
   $D_{KL}(P_{{\theta }^{\ast }}||P_{\theta })={\mathbb{E}}_{x\sim P_{{\theta }^{\ast }}}[\log P_{{\theta }^{\ast }}(x)]-{\mathbb{E}}_{x\sim P_{{\theta }^{\ast }}}[\log P_{\theta }(x)]$
   The first term, ${\mathbb{E}}_{x\sim P_{{\theta }^{\ast }}}[\log P_{{\theta }^{\ast }}(x)]$, is the negative entropy of the true distribution. It does not depend on our parameter $\theta$, so it's a constant with respect to our minimization problem. Therefore, minimizing the KL divergence is equivalent to minimizing the second term, or equivalently, maximizing its positive:
   $\arg {\min }_{\theta }{D}_{KL}(P_{{\theta }^{\ast }}||P_{\theta })\equiv \arg {\max }_{\theta }{\mathbb{E}}_{x\sim P_{{\theta }^{\ast }}}[\log P_{\theta }(x)]$

2. Approximate the Expectation: We cannot calculate this expectation directly because we don't know the true distribution $P_{{\theta }^{\ast }}$. However, we have a sample $(x_1,\dots ,x_n)$ drawn from it! By the Law of Large Numbers (which we studied in Module 7), we know that the sample mean converges to the true expectation. We can therefore use our sample to approximate the expectation:
   ${\mathbb{E}}_{x\sim P_{{\theta }^{\ast }}}[\log P_{\theta }(x)]\approx \frac{1}{n}{\sum }_{i=1}^n\log P_{\theta }(x_i)$

3. The Final Step: Substituting this approximation back into our maximization problem, we get:
   $\hat{\theta }\approx \arg {\max }_{\theta }\frac{1}{n}{\sum }_{i=1}^n\log P_{\theta }(x_i)$
   Since the $\frac{1}{n}$ term is a positive constant, it doesn't change the location of the maximum. This leaves us with:
   ${\hat{\theta }}_{MLE}=\arg {\max }_{\theta }{\sum }_{i=1}^n\log P_{\theta }(x_i)$
   This is precisely the log-likelihood function we derived earlier from intuition. This rigorous connection demonstrates that MLE is not just a heuristic; it is a principled method for finding the model that best approximates the true data-generating process in an information-theoretic sense.

Conclusion

In this lesson, we have formulated the principle of Maximum Likelihood Estimation, a cornerstone of modern statistics and machine learning.

Key Takeaways:

• Principle: MLE finds the parameter values $\theta$ that maximize the probability (likelihood) of observing the given data.
• Likelihood Function: For i.i.d. data, the likelihood $L(\theta |\mathbf{x})$ is the product of the individual probabilities/densities of each data point, viewed as a function of $\theta$.
• Log-Likelihood: In practice, we maximize the log-likelihood $l(\theta |\mathbf{x})$, which turns the product into a computationally convenient sum, without changing the location of the maximum.
• Theoretical Foundation: MLE is equivalent to finding the model distribution $P_{\theta }$ that is "closest" to the true, unknown data-generating distribution $P_{{\theta }^{\ast }}$, where closeness is measured by the Kullback-Leibler Divergence.

Preview of the Next Lesson:
We now have the principle. In the next lesson, we will apply it. You will learn how to derive the MLE for the parameters of a Bernoulli distribution. This will be our first practical application, where we use calculus to take the derivative of the log-likelihood function, set it to zero, and solve for the parameter, arriving at a familiar and intuitive result.