Understanding Maximum Likelihood Estimation

The Core Idea: What is "Most Likely"?

Imagine you have a set of data points, for instance, the heights of a sample of people. You might hypothesize that height in the population follows a normal distribution (a "bell curve"). But a normal distribution is defined by two parameters: its mean ($\mu$, the center of the curve) and its standard deviation ($\sigma$, the spread of the curve). Which values of $\mu$ and $\sigma$ create the specific bell curve that best represents your data?

Maximum Likelihood Estimation provides a formal way to answer this. The core idea is to "try out" many different possible parameter values and, for each set, ask: "Given this specific bell curve, how likely is it that we would have observed the exact data points that we did?" We then choose the parameters that make our observed data most likely.

The following video from StatQuest provides an excellent and intuitive visual explanation of this process using an example of mouse weights.

As the video demonstrates, MLE is essentially a search for the peak of a "likelihood mountain." The location of that peak gives us our best estimates for the model parameters.

Let's visualize this with a slightly different view. Look at the graph below. The blue dots are our observed data points. We are considering four different possible probability distributions, represented by the curves f₁, f₂, f₃, and f₄, that might have generated this data.

By just looking, which curve seems most plausible? Most people would choose f₃ (the pink curve). Why? Because our data points cluster near its center, where the curve is highest. This means that under the distribution f₃, observing these data points is relatively likely. In contrast, under f₄ (the teal curve), some of our data points fall in the tails of the distribution, where they are very unlikely to occur. MLE is the mathematical formalization of this intuition.

From "Magic" to Models

It's important to clarify why we need to assume a model (like a normal distribution or a binomial process) in the first place. Without a model, the "most likely" explanation for our data is always a trivial one. For example, if we flip a coin and get "Heads", the MLE without any restrictions would be a "magic coin" that always lands on Heads. If we observed the sequence H, T, H, the MLE would be a magic coin that always produces that exact sequence. This isn't useful because it doesn't generalize.

The power of MLE comes from introducing realistic constraints through a probability model. We assume our data comes from a known type of distribution (e.g., a normal distribution for heights, a binomial distribution for coin flips) but with unknown parameters. MLE then finds the best parameters within the rules of that model.

The next video explains this crucial point very clearly, using a coin-flipping example and then extending it to the normal distribution for height.

The Likelihood Function and the Log-Likelihood Trick

Now, let's formalize this a bit, connecting it to concepts from your CIS background. As you've just seen, for a set of independent data points ($x_1,x_2,\dots ,x_n$), the total likelihood of observing all of them is the product of their individual likelihoods:

$L(\theta |\text{data})=P(x_1|\theta )\times P(x_2|\theta )\times \cdots \times P(x_n|\theta )={\prod }_{i=1}^{n}P(x_i|\theta )$

Here, $\theta$ (theta) represents the set of all parameters for our model (e.g., $\theta =(\mu ,\sigma )$ for a normal distribution). The expression $L(\theta |\text{data})$ is the likelihood function.

Multiplying many small probabilities together can cause a problem known as numerical underflow, where the result becomes too small for a computer to store accurately. To get around this, we use a clever mathematical trick: we work with the natural logarithm of the likelihood.

The following reading explains this transformation and connects MLE directly to the field of machine learning.

Because the logarithm is a monotonically increasing function, the value of $\theta$ that maximizes the likelihood function $L(\theta )$ is the exact same value that maximizes the log-likelihood function $\log (L(\theta ))$. This trick converts a large product into a more manageable sum:

$\log (L(\theta ))={\sum }_{i=1}^n\log (P(x_i|\theta ))$

This log-likelihood is what is actually maximized in practice. The image below illustrates this point. The top plots (A, B) show a likelihood function, and the bottom plots (C, D) show the corresponding log-likelihood function. Notice that while their shapes are different, the peak—the maximum value—occurs at the exact same parameter value (in this case, $\lambda \approx 1.7$).

Finding the Maximum and Why We Care

So how do we find this maximum point? The traditional method involves calculus—taking the derivative of the log-likelihood function with respect to the parameters, setting it to zero, and solving. However, as your preference is to avoid deep calculus, the more relevant answer for modern applications is that we use computers. For all but the simplest models, the maximum is found using numerical optimization algorithms. The computer iteratively searches the "parameter space" to find the values that maximize the log-likelihood function.

This brings us to why MLE is so important.

As you read, MLE is the backbone of statistical estimation for several reasons:

• Universality: It provides a consistent method for fitting a vast range of models, from the linear and logistic regression models we've mentioned to the complex survival models we will study later.
• Optimality: The parameter estimates found via MLE have excellent statistical properties, especially with large sample sizes. They are generally the most precise estimates possible.
• Inference: The theory behind MLE also gives us a way to calculate standard errors and confidence intervals for our parameter estimates, which is crucial for hypothesis testing. The shape of the log-likelihood function itself tells us about our uncertainty: a sharp peak means we are very certain about our estimate, while a flat peak means we are uncertain.

Conclusion

In this lesson, you've learned the principle behind how we fit statistical models to data. Maximum Likelihood Estimation is a powerful and intuitive framework that sits at the heart of both classical statistics and modern machine learning.

Here are the key takeaways:

• The Goal of MLE: To find the values for a model's parameters that make the observed data most probable.
• The Process: We define a probability model for our data, which in turn defines a likelihood function. We then find the parameter values that maximize this function.
• The Log-Likelihood Trick: In practice, we maximize the log-likelihood because it's computationally more stable (turning products into sums) and mathematically easier to handle, while giving the exact same result for the parameter estimates.
• The Connection: MLE provides the theoretical foundation for fitting many important models, including logistic regression. The process of minimizing the negative log-likelihood is conceptually identical to minimizing a loss function in machine learning.

This lesson marks the end of our first module on the foundations of biostatistics and epidemiology. You now have a conceptual toolkit that includes measures of disease, measures of association, and the fundamental principle for fitting statistical models.

In the next lesson, we will begin Module 2, where we will apply these concepts to real-world scenarios. We'll start by examining the design, strengths, and weaknesses of cross-sectional studies, one of the most common types of study in epidemiology.