Proof of the Gauss-Markov Theorem

1. Setting the Stage: The Linear Model and its Assumptions

Before we can prove the theorem, we must be precise about the model and the "rules of the game"—the assumptions under which the theorem is valid.

Recall the general linear model in matrix form:
$\mathbf{y}=\mathbf{X}\mathbf{\beta }+\mathbf{ϵ}$
where:

• $\mathbf{y}$ is the $n\times 1$ vector of observed outcomes.
• $\mathbf{X}$ is the $n\times p$ matrix of predictors (the design matrix).
• $\mathbf{\beta }$ is the $p\times 1$ vector of unknown coefficients we want to estimate.
• $\mathbf{ϵ}$ is the $n\times 1$ vector of unobserved random errors.

The OLS estimator for $\mathbf{\beta }$ is given by the formula you've seen before, derived from the normal equations:
$\hat{\mathbf{\beta }}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$

The Gauss-Markov theorem holds if a specific set of assumptions about this model are true. Please read the following section from StatLect to familiarize yourself with these assumptions.

To summarize, the Gauss-Markov assumptions are:

1. Zero Mean Error: The expected value of the error term, conditional on the predictors, is zero.
   $E[\mathbf{ϵ}|\mathbf{X}]=0$
   This means that, on average, the model is correctly specified.

2. Spherical Errors: The errors are homoscedastic (have constant variance, ${\sigma }^2$) and uncorrelated.
   $\text{Cov}(\mathbf{ϵ}|\mathbf{X})={\sigma }^2\mathbf{I}$
   where $\mathbf{I}$ is the $n\times n$ identity matrix. From your mechanical engineering background, you can think of this as assuming the measurement noise has the same magnitude regardless of the system's state and that each measurement's error is independent of the others.

3. Full Rank Design Matrix: The matrix $\mathbf{X}$ has full column rank ($rank(\mathbf{X})=p$). This means there is no perfect multicollinearity among the predictors, ensuring that $({\mathbf{X}}^T\mathbf{X}{)}^{-1}$ is well-defined.

Crucially, notice what is not assumed: we have not assumed that the errors $\mathbf{ϵ}$ follow a Normal distribution. This is a key difference from the assumptions often made when deriving properties of Maximum Likelihood Estimators.

2. Deconstructing BLUE: Linearity and Unbiasedness

The acronym BLUE stands for Best Linear Unbiased Estimator. Let's prove the "L" and "U" parts first.

• Linear: An estimator is linear if it is a linear function of the observed outcomes $\mathbf{y}$. Looking at the OLS formula, $\hat{\mathbf{\beta }}=[({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T]\mathbf{y}$, we can see it is the product of a matrix (which is fixed for a given $\mathbf{X}$) and the vector $\mathbf{y}$. This is a linear transformation, so $\hat{\mathbf{\beta }}$ is a linear estimator.

• Unbiased: An estimator is unbiased if its expected value equals the true parameter value. We need to prove $E[\hat{\mathbf{\beta }}]=\mathbf{\beta }$.

The following reading provides a concise proof for both of these properties.

Let's walk through the proof of unbiasedness together, as it's a foundational step:

1. Start with the definition of $\hat{\mathbf{\beta }}$:
   $\hat{\mathbf{\beta }}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$
2. Substitute $\mathbf{y}=\mathbf{X}\mathbf{\beta }+\mathbf{ϵ}$:
   $\hat{\mathbf{\beta }}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T(\mathbf{X}\mathbf{\beta }+\mathbf{ϵ})$
3. Distribute the term:
   $\hat{\mathbf{\beta }}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{X}\mathbf{\beta }+({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}$
4. The first term simplifies, since $({\mathbf{X}}^T\mathbf{X}{)}^{-1}({\mathbf{X}}^T\mathbf{X})=\mathbf{I}$:
   $\hat{\mathbf{\beta }}=\mathbf{I}\mathbf{\beta }+({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}=\mathbf{\beta }+({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}$
5. Now, take the expectation. Since $\mathbf{\beta }$ and $\mathbf{X}$ are fixed (non-random), and using Assumption 1 ($E[\mathbf{ϵ}]=0$):
   $E[\hat{\mathbf{\beta }}]=E[\mathbf{\beta }]+E[({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}]=\mathbf{\beta }+({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^{T}E[\mathbf{ϵ}]=\mathbf{\beta }+0=\mathbf{\beta }$
   This completes the proof that $\hat{\mathbf{\beta }}$ is an unbiased estimator of $\mathbf{\beta }$.

3. The Main Event: Proving "Best"

Now we arrive at the heart of the theorem: proving that the OLS estimator is "Best".

What Does "Best" Mean?

In the context of estimators, "best" means having the minimum variance. Since $\mathbf{\beta }$ is a vector, its "variance" is a covariance matrix, $\text{Var}(\hat{\mathbf{\beta }})$. For one estimator $\hat{\mathbf{\beta }}$ to be "better" than another estimator $\tilde{\mathbf{\beta }}$, we require that the matrix $\text{Var}(\tilde{\mathbf{\beta }})-\text{Var}(\hat{\mathbf{\beta }})$ is positive semi-definite.

This is a generalization of one scalar being greater than or equal to another. It ensures that for any linear combination of the coefficients (e.g., $c_1{\beta }_1+c_2{\beta }_2$), the variance of the estimate from OLS is less than or equal to the variance from the alternative estimator.

The Proof Strategy

Our strategy is as follows:

1. Define an arbitrary alternative linear unbiased estimator, $\tilde{\mathbf{\beta }}$.
2. Derive a condition that this estimator must satisfy to be unbiased.
3. Calculate the covariance matrix of $\tilde{\mathbf{\beta }}$.
4. Show that this covariance matrix is equal to the covariance matrix of the OLS estimator, $\hat{\mathbf{\beta }}$, plus a positive semi-definite matrix.

The following video provides an excellent walkthrough of the matrix algebra involved in this proof.

Let's formalize the steps from the video with our notation.

Step 1: Define an Alternative Estimator
Let $\tilde{\mathbf{\beta }}$ be any other linear estimator. We can write it as a linear function of $\mathbf{y}$:
$\tilde{\mathbf{\beta }}=\mathbf{Ay}$
To easily compare it to the OLS estimator, let's define the matrix $\mathbf{A}$ in terms of the OLS matrix plus some other matrix $\mathbf{D}$:
$\mathbf{A}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T+\mathbf{D}$
So, our alternative estimator is $\tilde{\mathbf{\beta }}=\left(({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T+\mathbf{D}\right)\mathbf{y}$.

Step 2: Apply the Unbiasedness Condition
We require $\tilde{\mathbf{\beta }}$ to be unbiased, i.e., $E[\tilde{\mathbf{\beta }}]=\mathbf{\beta }$.

For this to equal $\mathbf{\beta }$ for any possible $\mathbf{\beta }$, we must have $(\mathbf{I}+\mathbf{DX})=\mathbf{I}$, which implies:
$\mathbf{DX}=0$
This is a crucial result we will use shortly.

Step 3: Calculate the Covariance of the Alternative Estimator
Now we find the covariance matrix of $\tilde{\mathbf{\beta }}$.

Since $\mathbf{X}\mathbf{\beta }$ is a constant (non-random) term, it does not contribute to the variance. So, we have:

Using the rule $\text{Var}(\mathbf{M}\mathbf{ϵ})=\mathbf{M}\text{Var}(\mathbf{ϵ}){\mathbf{M}}^T$ and Assumption 2 ($\text{Var}(\mathbf{ϵ})={\sigma }^2\mathbf{I}$):

This is the starting point of the video. As shown, when we multiply this out, we get four terms:

Step 4: Simplify the Expression

• Term 1: Simplifies to $({\mathbf{X}}^T\mathbf{X}{)}^{-1}$. Note that ${\sigma }^2({\mathbf{X}}^T\mathbf{X}{)}^{-1}$ is precisely $\text{Var}(\hat{\mathbf{\beta }})$.
• Term 3: Since $\mathbf{DX}=0$ from our unbiasedness condition, this term is $0$.
• Term 2: We can rewrite this as $({\mathbf{X}}^T\mathbf{X}{)}^{-1}(\mathbf{DX}{)}^T$. Since $\mathbf{DX}=0$, this term is also $0$.
• Term 4: The term $\mathbf{D}{\mathbf{D}}^T$ remains.

Step 5: The Final Result
Putting it all together, the expression for the variance of our alternative estimator simplifies beautifully:

Therefore,
$\text{Var}(\tilde{\mathbf{\beta }})=\text{Var}(\hat{\mathbf{\beta }})+{\sigma }^2\mathbf{D}{\mathbf{D}}^T$

The matrix $\mathbf{D}{\mathbf{D}}^T$ is always positive semi-definite. This is because for any non-zero vector $\mathbf{v}$, the quadratic form ${\mathbf{v}}^T(\mathbf{D}{\mathbf{D}}^T)\mathbf{v}=({\mathbf{D}}^T\mathbf{v}{)}^T({\mathbf{D}}^T\mathbf{v})=||{\mathbf{D}}^T\mathbf{v}|{|}_2^2\ge 0$.

This means the variance of any other linear unbiased estimator is equal to the variance of the OLS estimator plus a "non-negative" matrix. This proves that OLS has the minimum variance. For a more detailed textual reference of this final step, you can consult the following resource.

Conclusion

Congratulations! You have now walked through the complete proof of the Gauss-Markov theorem. This is a significant result in statistics and econometrics, and understanding its derivation is a mark of a deep theoretical foundation in data science.

Key Takeaways:

• The Meaning of BLUE: The OLS estimator is the Best Linear Unbiased Estimator for the coefficients of a linear model.
  • Best: It has the minimum variance among all linear unbiased estimators. This is guaranteed by showing the difference in covariance matrices is positive semi-definite.
  • Linear: The estimator is a linear function of the observed data $\mathbf{y}$.
  • Unbiased: Its expected value is equal to the true parameter vector $\mathbf{\beta }$.
• The Required Assumptions: The theorem relies on the Gauss-Markov assumptions: zero mean errors, homoscedasticity, no error correlation, and no perfect multicollinearity. It does not require the errors to be normally distributed.
• Proof Strategy: The proof hinges on defining an arbitrary linear unbiased estimator and showing its variance equals the OLS variance plus a positive semi-definite matrix (${\sigma }^2\mathbf{D}{\mathbf{D}}^T$). The unbiasedness condition ($\mathbf{DX}=0$) is the key that makes the cross-product terms in the variance expansion disappear.

While the CRLB we studied previously gives an absolute bound on variance for any unbiased estimator, the Gauss-Markov theorem provides a powerful guarantee of optimality within the practical and widely-used class of linear estimators, and under less restrictive distributional assumptions.

Preview of the Next Lesson:

We have now explored two major frameworks for estimation: Maximum Likelihood (with its CRLB optimality) and Least Squares (with its Gauss-Markov optimality). Both are part of the "frequentist" school of statistics, which treats parameters as fixed, unknown constants. In the next lesson, we will pivot to a different philosophy: Bayesian Inference. You will learn how to treat parameters as random variables themselves and how to derive estimators like the Minimum Mean Square Error (MMSE) and Maximum A Posteriori (MAP) by combining prior beliefs with observed data.