Likelihood Function Formulation

Hello! Welcome to your next lesson in the "Statistical Inference and Estimation" module.

In our last lesson, we explored the Method of Moments (MoM), an intuitive technique for estimating a model's parameters by equating sample moments (like the mean) with their theoretical population counterparts. It gave us a solid justification for why we often use sample statistics as our "best guess" for population parameters.

Today, we'll dive into a different, more powerful, and arguably the most important principle in statistical inference and machine learning: Maximum Likelihood Estimation (MLE). The core idea of MLE is to ask: "Given the data we have observed, what parameter values for our model make the data most plausible?"

Your learning outcome for this lesson is to formulate the likelihood function for a given dataset and a probabilistic model. This is the crucial first step in the MLE process. We'll focus on setting up the problem correctly today, and in the next lesson, we'll learn how to solve it.

1. Probability vs. Likelihood: A Critical Distinction

Before we build the function, we need to be very clear about the difference between two often-confused terms: probability and likelihood.

Probability asks: "Given a known model (e.g., a fair coin with $p = 0.5$ ), what is the chance of seeing a particular outcome?" Here, the parameters are fixed, and the data is variable.
Likelihood flips this around. It asks: "Given the data we have already collected (e.g., we observed 8 heads in 10 flips), how plausible are different parameter values (e.g., is $p = 0.5$ or $p = 0.8$ a better explanation for this data)?" Here, the data is fixed, and the parameters are variable.

This distinction is fundamental to MLE. The following short video from StatQuest provides an excellent visual explanation.

In Statistics, Probability is not Likelihood.

This video clearly illustrates the difference between probability and likelihood using a normal distribution. Notice how probability is the area under the curve, while likelihood is the height of the curve at a specific, observed data point.

Watch from 2:35 to 4:41. Focus on the visual distinction and the change in perspective: probability is about predicting data from a model, while likelihood is about evaluating a model given data.

To reinforce this, consider the following image.

Probability vs. Likelihood Comparison — This image provides a side-by-side comparison of 'Probability' and 'Likelihood' using a normal distribution curve. Probability is the area under the curve for a range of outcomes, while likelihood is the value of the probability density function at a specific observed data point, viewed as a function of the model parameters.

In MLE, our goal is to find the parameter value(s) that maximize the likelihood function.

2. Formulating the Likelihood Function

So, how do we write down this "likelihood function"? It's essentially the joint probability of observing our entire dataset, but we treat it as a function of the model's parameters.

Let's say we have a dataset $X = {x_{1}, x_{2}, \dots, x_{n}}$ and a model with parameter(s) $θ$ . We want to find the value of $θ$ that maximizes the probability of observing $X$ , which we write as $P (X ∣ θ)$ .

A crucial assumption we make in most ML contexts is that our data points are Independent and Identically Distributed (IID).

Independent: The value of one observation does not influence the value of another.
Identically Distributed: All observations are drawn from the same underlying probability distribution.

Because of the independence assumption, the joint probability of the entire dataset is simply the product of the probabilities of each individual data point. This gives us the likelihood function:

$L (θ ∣ X) = P (X ∣ θ) = \prod_{i = 1}^{n} P (x_{i} ∣ θ)$

Here, $P (x_{i} ∣ θ)$ is the probability mass function (PMF) for a discrete distribution or the probability density function (PDF) for a continuous distribution, evaluated at the data point $x_{i}$ .

The following reading explains this setup clearly and formally.

Introduction to Maximum Likelihood Estimation (MLE)

This article from DataCamp provides a clear, step-by-step derivation of the general likelihood function.

Read 'Step 1: Define probability model' and 'Step 2: Writing the likelihood function'. Pay close attention to how the assumption of conditional independence simplifies the joint probability from a complex chain rule expansion into a simple product of individual probabilities.

3. A Better Alternative: The Log-Likelihood Function

Working with the likelihood function directly can be difficult. Multiplying many small probabilities (numbers between 0 and 1) can lead to extremely small results, causing numerical instability (floating-point underflow) in computers.

A brilliant mathematical trick solves this: instead of maximizing the likelihood, we can maximize its logarithm, which we call the log-likelihood function.

$ℓ (θ ∣ X) = lo g L (θ ∣ X) = lo g (\prod_{i = 1}^{n} P (x_{i} ∣ θ)) = \sum_{i = 1}^{n} lo g P (x_{i} ∣ θ)$

Why is this allowed? The logarithm is a monotonically increasing function. This means that if $a > b$ , then $lo g (a) > lo g (b)$ . Therefore, the value of $θ$ that maximizes $L (θ)$ is the exact same value of $θ$ that maximizes $ℓ (θ)$ .

This transformation gives us two huge advantages:

It turns products into sums, which are computationally much more stable.
It simplifies the mathematics required for optimization, as sums are much easier to differentiate than products (a key step we'll see in the next lesson).

The next segment of the Stanford lecture video explains this concept of using argmax and the log-likelihood trick brilliantly.

Stanford CS109 Probability for Computer Scientists I M.L.E. I 2022 I Lecture 21

This video explains why we can switch to maximizing the log-likelihood instead of the likelihood itself. It introduces the argmax notation and highlights the mathematical and computational benefits of this switch.

Watch from 18:44 to 25:15. Focus on three key ideas: (1) The concept of argmax (finding the argument that maximizes a function, not the maximum value itself), (2) why the argmax of a function is the same as the argmax of its log, and (3) how the log turns products into sums.

4. Examples: Formulating Likelihood Functions

Let's now apply this process to formulate the likelihood and log-likelihood functions for a few common distributions. This is the core skill for this lesson.

We will use the same Stanford lecture video, which provides excellent step-by-step examples. For today, your goal is to follow along with the setup of the likelihood and log-likelihood functions. Don't worry about the final steps where the derivative is taken to find the maximum—that's for our next lesson.

Example 1: Bernoulli Distribution

The Bernoulli distribution models a single trial with a binary outcome (e.g., 0 or 1, fail or success). The parameter is $p$ , the probability of success. The PMF has a clever form that you may have seen before: $P (x ∣ p) = p^{x} (1 - p)^{1 - x}$ .

Let's see how to build the likelihood function for a series of Bernoulli trials.

Stanford CS109 Probability for Computer Scientists I M.L.E. I 2022 I Lecture 21

This segment demonstrates how to formulate the likelihood and log-likelihood for the Bernoulli distribution.

Watch from 46:19 to 50:15. Follow the derivation of the likelihood function as a product and how it transforms into a sum for the log-likelihood. Note the use of the p^x (1-p)^(1-x) form of the PMF, which is essential for making the math work.

Likelihood Function: $L (p ∣ X) = \prod_{i = 1}^{n} p^{x_{i}} (1 - p)^{1 - x_{i}}$
Log-Likelihood Function: $ℓ (p ∣ X) = \sum_{i = 1}^{n} [x_{i} lo g (p) + (1 - x_{i}) lo g (1 - p)]$

Example 2: Poisson Distribution

The Poisson distribution models the number of events occurring in a fixed interval, with a rate parameter $λ$ . Its PMF is $P (x ∣ λ) = \frac{e ^{- λ} λ ^{x}}{x !}$ .

Stanford CS109 Probability for Computer Scientists I M.L.E. I 2022 I Lecture 21

Now, let's do the same for the Poisson distribution. The process is identical: start with the PMF, form the product for the likelihood, and take the log to get the sum.

Watch from 37:45 to 43:30. Again, focus on the first few steps: writing the likelihood of a single data point, constructing the likelihood for all data points as a product, and simplifying it into the log-likelihood sum. You can stop when the derivative is introduced.

Likelihood Function: $L (λ ∣ X) = \prod_{i = 1}^{n} \frac{e ^{- λ} λ ^{x_{i}}}{x _{i} !}$
Log-Likelihood Function: $ℓ (λ ∣ X) = \sum_{i = 1}^{n} [- λ + x_{i} lo g (λ) - lo g (x_{i}!)]$

Test your understanding!

In the previous lesson, you found the Method of Moments estimator for the Exponential distribution. The PDF for an Exponential distribution with rate parameter $λ$ is $f (x ∣ λ) = λ e^{- λ x}$ for $x \geq 0$ .

Given a sample of data $X = {x_{1}, x_{2}, \dots, x_{n}}$ , what are the likelihood and log-likelihood functions for $λ$ ?

Show answer

Likelihood Function (Product of PDFs):
$L (λ ∣ X) = i = 1 \prod n f (x_{i} ∣ λ) = i = 1 \prod n λ e^{- λ x_{i}}$
We can simplify this by grouping the terms:
$L (λ ∣ X) = λ^{n} e^{- λ \sum_{i = 1}^{n} x_{i}}$
Log-Likelihood Function (Sum of log-PDFs):
$ℓ (λ ∣ X) = lo g (L (λ ∣ X)) = lo g (λ^{n} e^{- λ \sum x_{i}})$
Using log rules ( $lo g (ab) = lo g (a) + lo g (b)$ and $lo g (a^{b}) = b lo g (a)$ ):
$ℓ (λ ∣ X) = lo g (λ^{n}) + lo g (e^{- λ \sum x_{i}})$ $ℓ (λ ∣ X) = n lo g (λ) - λ i = 1 \sum n x_{i}$

Conclusion

Great work today! You've taken the first, most important step in understanding Maximum Likelihood Estimation. This principle is at the heart of how many machine learning models, from simple linear regression to complex neural networks, are "trained."

Key Takeaways:

Likelihood is a measure of how plausible a set of model parameters are, given the data you've observed. It's a function of the parameters for fixed data.
The Likelihood Function $L (θ ∣ X)$ is formed by taking the product of the probabilities (or probability densities) of each individual data point, based on the IID assumption.
The Log-Likelihood Function $ℓ (θ ∣ X)$ is the logarithm of the likelihood function. We use it because it converts products to sums, making it more computationally stable and mathematically convenient, without changing where the maximum occurs.
The process to formulate the log-likelihood is:
1. Identify the correct probability distribution for your data.
2. Write down its PMF or PDF, $P (x_{i} ∣ θ)$ .
3. The log-likelihood is the sum of the logs of this function over all data points: $\sum lo g P (x_{i} ∣ θ)$ .

Preview of the Next Lesson:

Now that we know how to formulate the log-likelihood function, how do we find the parameter values that maximize it? In the next lesson, Applying Maximum Likelihood Estimation (MLE), we will use the calculus skills you've reviewed. We'll take the derivative of the log-likelihood function with respect to the parameters, set it to zero, and solve to find the Maximum Likelihood Estimators.

Recommended resources

1. Probability vs. Likelihood: A Critical Distinction

2. Formulating the Likelihood Function

3. A Better Alternative: The Log-Likelihood Function

4. Examples: Formulating Likelihood Functions

Example 1: Bernoulli Distribution

Example 2: Poisson Distribution

Conclusion