MLE vs. OLS for Linear Models with Gaussian Noise

Introduction

Welcome to the next lesson in our exploration of statistical inference. In our previous lesson, we derived the Generalized Least Squares (GLS) estimator, which we proved is the Best Linear Unbiased Estimator (BLUE) when the errors have a general, known covariance matrix $C$ . That derivation was powerful because it made no assumptions about the distribution of the errors, only about their first and second moments (mean and covariance).

Today, we will add a crucial layer to our linear model: a specific distributional assumption. We will assume the errors are not just uncorrelated with constant variance, but are also drawn from a Gaussian (Normal) distribution. This assumption allows us to connect two major estimation frameworks: Ordinary Least Squares (OLS) and Maximum Likelihood Estimation (MLE).

Our primary goal is to compare the properties of the MLE and OLS estimators for the linear model with Gaussian noise. You will discover that under this specific condition, these two seemingly different approaches lead to identical conclusions for the regression coefficients, a result that highlights a deep and elegant unity in statistical theory. We will also uncover a subtle but important difference in how they estimate the model's error variance.

1. The Linear Model with Gaussian Noise

Let's begin by formally stating the model. We start with the standard linear model:
$y = X β + ϵ$
where $y$ is the $n \times 1$ vector of observations, $X$ is the $n \times p$ design matrix, $β$ is the $p \times 1$ vector of unknown parameters, and $ϵ$ is the $n \times 1$ vector of errors.

Now, we add the crucial assumption: the errors are independent and identically distributed (i.i.d.) according to a Gaussian distribution with mean 0 and variance $σ^{2}$ .
$ϵ_{i} \sim N (0, σ^{2}) for i = 1, \dots, n$
This implies that the vector of errors $ϵ$ follows a multivariate normal distribution: $ϵ \sim N (0, σ^{2} I)$ .

From the properties of the Gaussian distribution, a linear transformation of a Gaussian random variable is also Gaussian. Thus, the conditional distribution of $y_{i}$ given $x_{i}$ is:
$y_{i} ∣ x_{i}, β, σ^{2} \sim N (x_{i}^{T} β, σ^{2})$
The probability density function (PDF) for a single observation $y_{i}$ is therefore:
$p (y_{i} ∣ x_{i}, β, σ^{2}) = \frac{1}{2 π σ ^{2}} exp (- \frac{( y _{i} - x _{i}^{T} β ) ^{2}}{2 σ ^{2}})$
This PDF is the fundamental building block for constructing the Maximum Likelihood Estimator.

2. Deriving the MLE: Equivalence with OLS

Our goal is to find the parameters $(β, σ^{2})$ that maximize the likelihood of observing our data $(y, X)$ . The first step is to write down the likelihood function.

The video below provides a clear, step-by-step derivation showing how maximizing the likelihood under the Gaussian noise assumption leads directly to the least squares solution.

Least Squares as a Maximum Likelihood estimator

This video, 'Least Squares as a Maximum Likelihood estimator' from Ben Lambert's channel, formally derives the MLE for the coefficients of a linear model. It clearly demonstrates that the objective function for MLE becomes equivalent to the objective function for OLS when we assume Gaussian errors.

Watch the entire video (0:00 - 7:37). Pay close attention to these key steps: How the likelihood for a single observation is constructed from the Gaussian PDF. How the joint likelihood for the whole dataset is formed as a product. How taking the logarithm simplifies the problem. The final step where differentiating the log-likelihood with respect to the parameter beta yields a condition identical to the one found in OLS.

Let's formalize the derivation from the video using matrix notation, which you are comfortable with.

Likelihood Function: Since the observations are independent, the joint likelihood is the product of the individual PDFs:
$L (β, σ^{2} ∣ y, X) = i = 1 \prod n p (y_{i} ∣ x_{i}, β, σ^{2}) = (\frac{1}{2 π σ ^{2}})^{n /2} exp (- \frac{1}{2 σ ^{2}} i = 1 \sum n (y_{i} - x_{i}^{T} β)^{2})$
In vector form, this is:
$L (β, σ^{2}) = (2 π σ^{2})^{- n /2} exp (- \frac{1}{2 σ ^{2}} (y - X β)^{T} (y - X β))$
Log-Likelihood Function: Taking the natural logarithm simplifies the expression significantly:
$lo g L (β, σ^{2}) = - \frac{n}{2} lo g (2 π σ^{2}) - \frac{1}{2 σ ^{2}} (y - X β)^{T} (y - X β)$
Maximizing with respect to $β$ : To find the MLE for $β$ , we need to maximize this function. Notice that the first term, $- \frac{n}{2} lo g (2 π σ^{2})$ , does not depend on $β$ . Therefore, maximizing the log-likelihood with respect to $β$ is equivalent to minimizing the term:
$SSE (β) = (y - X β)^{T} (y - X β) = \sum_{i = 1}^{n} (y_{i} - x_{i}^{T} β)^{2}$
This is precisely the Sum of Squared Errors objective function from Ordinary Least Squares!

Since the optimization problem is identical, the solution must be identical.
$β^_{M L E} = β^_{O L S} = (X^{T} X)^{- 1} X^{T} y$

This is a profound result. It shows that the OLS method, which seems to be a simple heuristic based on minimizing squared distances, is in fact the "most likely" estimator under the common and natural assumption of Gaussian noise.

To solidify this concept, the following slides offer a concise summary of this derivation and the geometric interpretation.

Linear Regression and the Bias Variance Tradeoff

These slides from a Berkeley lecture on Linear Regression provide a condensed, matrix-based derivation of the equivalence between MLE and OLS. They also connect it to the geometric view of OLS as a projection.

Please review the slides from 'Rewriting with Matrix Notation' through 'Geometric Interpretation' (slides 9-13). Focus on how the log-likelihood is simplified to show that maximizing it is the same as minimizing the squared error. The final derivation of the 'Normal Equations' should be very familiar.

3. Comparing Estimator Properties

Now that we have established the equivalence of the estimators for $β$ , let's perform a detailed comparison of their properties. This is where a key difference emerges.

3.1 Properties of the Coefficient Estimator $β^$

Property	OLS Estimator ( $β^_{O L S}$ )	MLE Estimator ( $β^_{M L E}$ )	Comparison
Formula	$(X^{T} X)^{- 1} X^{T} y$	$(X^{T} X)^{- 1} X^{T} y$	Identical.
Bias	Unbiased. Proven via Gauss-Markov theorem: $E [β^_{O L S}] = β$ .	Unbiased, because it is identical to the OLS estimator.	Identical (both are unbiased).
Variance	$Var (β^_{O L S}) = σ^{2} (X^{T} X)^{- 1}$	$Var (β^_{M L E}) = σ^{2} (X^{T} X)^{- 1}$	Identical.
Optimality	BLUE (Best Linear Unbiased Estimator). This holds even without the Gaussian assumption.	MVUE (Minimum Variance Unbiased Estimator). It achieves the Cramér-Rao Lower Bound.	The MLE's optimality claim is stronger. MVUE means it has the lowest variance among all unbiased estimators, not just linear ones. This stronger claim is possible because we made a distributional (Gaussian) assumption.

3.2 Properties of the Error Variance Estimator $σ^^{2}$

Here is where the two methods diverge. Let's find the MLE for $σ^{2}$ . We return to the log-likelihood function:
$lo g L = - \frac{n}{2} lo g (2 π) - \frac{n}{2} lo g (σ^{2}) - \frac{1}{2 σ ^{2}} (y - X β)^{T} (y - X β)$
First, we plug in our MLE for $β$ , $β^_{M L E}$ . Then we take the derivative with respect to $σ^{2}$ and set it to zero:
$\frac{\partial l o g L}{\partial ( σ ^{2} )} = - \frac{n}{2 σ ^{2}} + \frac{1}{2 ( σ ^{2} ) ^{2}} (y - X β^_{M L E})^{T} (y - X β^_{M L E}) = 0$
Solving for $σ^{2}$ gives the MLE for the variance:
$σ^_{M L E}^{2} = \frac{( y - X β ^ _{M L E} ) ^{T} ( y - X β ^ _{M L E} )}{n} = \frac{SSE}{n}$
Now, let's compare this to the standard unbiased estimator for $σ^{2}$ used in the context of OLS.

Property	OLS-associated Estimator ( $σ^_{O L S}^{2}$ )	MLE Estimator ( $σ^_{M L E}^{2}$ )	Comparison
Formula	$\frac{SSE}{n - p}$	$\frac{SSE}{n}$	Different. The denominators are $n - p$ vs. $n$ , where $p$ is the number of parameters in $β$ .
Bias	Unbiased. It is specifically constructed to be so: $E [σ^_{O L S}^{2}] = σ^{2}$ .	Biased. Its expectation is $E [σ^_{M L E}^{2}] = \frac{n - p}{n} σ^{2}$ , which is slightly less than the true $σ^{2}$ .	The OLS variance estimator is unbiased by design. The MLE variance estimator is biased, though it becomes asymptotically unbiased as $n \to \infty$ . The $n - p$ term is a degrees of freedom correction. We "lose" $p$ degrees of freedom by estimating the $p$ parameters in $β$ .

The following video provides an excellent visual summary of this entire comparison, including the difference in the variance estimators.

MLE vs OLS | Maximum likelihood vs least squares in linear regression

This video, 'MLE vs OLS', provides a fantastic conceptual and visual comparison. It demonstrates how changing the regression line parameters affects both the Sum of Squared Residuals (OLS) and the joint likelihood (MLE), showing they are optimized at the same point. It also explicitly discusses the difference in the variance estimators.

Watch the entire video. Focus on: The visual connection between minimizing the distance to the line (OLS) and maximizing the height on the normal distribution curves (MLE). The demonstration at (8:21) showing that both methods find the same optimal parameters. The crucial explanation at the end (12:31) about the difference in the denominators for the variance estimate (n vs n-2 in the video's simple regression example).

The image below, taken from the video, perfectly encapsulates the main idea. Minimizing the sum of squared residuals (the $Y - Y_{es t}$ terms) is equivalent to finding the line that maximizes the product of the heights of the Gaussian distributions centered on that line.

OLS vs MLE with Gaussian Noise Visualization — A visual representation comparing OLS and MLE. OLS minimizes the sum of squared vertical distances (residuals). MLE, under Gaussian noise, places a normal distribution at each predicted point on the line and seeks to maximize the joint probability of the observed data, which corresponds to maximizing the product of the probability densities (heights of the curves at the data points).

Conclusion

In this lesson, we established a critical link between two foundational estimation techniques by adding the assumption of i.i.d. Gaussian noise to the linear model.

Key Takeaways:

Equivalence for $β$ : For a linear model with Gaussian noise, the Maximum Likelihood Estimator for the regression coefficients $β$ is identical to the Ordinary Least Squares estimator.
Stronger Optimality: This equivalence means that under Gaussian noise, the OLS estimator is not just the Best Linear Unbiased Estimator (BLUE), but the Minimum Variance Unbiased Estimator (MVUE)—the best of all unbiased estimators.
Difference for $σ^{2}$ : The MLE for the error variance $σ^{2}$ is $\frac{SSE}{n}$ and is biased. The standard unbiased estimator used with OLS is $\frac{SSE}{n - p}$ , which corrects for the degrees of freedom used to estimate $β$ .
The Power of Assumptions: This lesson highlights how adding distributional assumptions (like Gaussian noise) can provide deeper insights and stronger theoretical guarantees (like MVUE) for our estimators.

You have now seen how frequentist estimation can be approached from both a geometric/algebraic perspective (OLS) and a probabilistic one (MLE), and how they beautifully converge under the right conditions.

Preview of the Next Lesson:

We are now ready to transition into the third major paradigm of statistical inference: Bayesian Inference. In the upcoming module, we will start treating parameters not as fixed constants but as random variables themselves. You will learn how to formally combine prior knowledge about a parameter with the likelihood from observed data to form an updated posterior belief. This will lead us to powerful new estimators like the MMSE and MAP estimators and provide a completely new way of thinking about uncertainty and inference.

Recommended resources