Understanding the Gauss-Markov Theorem

Hello! Welcome to the next lesson in our journey through the statistical properties of linear regression.

In our previous lessons, we have established two fundamental properties of the Ordinary Least Squares (OLS) estimator:

1. It is unbiased, meaning on average, it hits the true parameter values ($E[\hat{\mathbf{\beta }}]=\mathbf{\beta }$).
2. We derived its variance-covariance matrix, $\text{Var}(\hat{\mathbf{\beta }})={\sigma }^2({\mathbf{X}}^T\mathbf{X}{)}^{-1}$, which quantifies its precision.

These are excellent properties, but they lead to a critical question: Could there be another linear and unbiased estimator that is even more precise (i.e., has a smaller variance)? Today, we will answer this question definitively.

Our learning outcome is to state and explain the Gauss-Markov theorem regarding the optimality of OLS estimators. This celebrated theorem provides the theoretical backbone for why OLS is so widely used. It establishes that, within a certain class of estimators, OLS is not just good—it's the best.

1. The Meaning of "BLUE"

The Gauss-Markov theorem states that under a specific set of assumptions, the OLS estimator is BLUE, which stands for Best Linear Unbiased Estimator. Let's break down this acronym, as it perfectly frames the theorem's content.

• Estimator: $\hat{\mathbf{\beta }}$ is a rule (an estimator) for calculating an estimate of the true parameter $\mathbf{\beta }$ from a sample of data.
• Linear: The estimator is a linear function of the observed outcomes, $\mathbf{y}$. We've already confirmed this property, as $\hat{\mathbf{\beta }}=[({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T]\mathbf{y}$, where the term in brackets is a matrix of constants once $\mathbf{X}$ is known.
• Unbiased: The expected value of the estimator is the true parameter, $E[\hat{\mathbf{\beta }}]=\mathbf{\beta }$. We formally proved this in a previous lesson.
• Best: This is the new and crucial piece. "Best" means that the OLS estimator has the minimum variance among all other linear unbiased estimators.

An estimator with lower variance is more precise, meaning its estimates are more tightly clustered around the true parameter value.

For a vector of estimates $\hat{\mathbf{\beta }}$, "minimum variance" has a specific meaning. An estimator $\hat{\mathbf{\beta }}$ is "best" if for any other linear unbiased estimator $\tilde{\mathbf{\beta }}$, the matrix $\text{Var}(\tilde{\mathbf{\beta }})-\text{Var}(\hat{\mathbf{\beta }})$ is a positive semi-definite matrix. This is a powerful condition that implies that the variance of any linear combination of the coefficients (e.g., $\text{Var}(a_1{\hat{\beta }}_1+a_2{\hat{\beta }}_2)$) is minimized by the OLS estimates.

2. The Gauss-Markov Theorem and Its Assumptions

To fully appreciate the proof, let's first formally state the theorem and the assumptions it relies upon. These are the Classical Linear Model (CLM) assumptions we have been building over the past lessons.

To summarize, the key assumptions are:

1. Linearity: The model is linear in parameters: $\mathbf{y}=\mathbf{X}\mathbf{\beta }+\mathbf{ϵ}$.
2. Full Rank: The matrix $\mathbf{X}$ has full column rank, meaning no perfect multicollinearity.
3. Zero Conditional Mean: The errors have a mean of zero conditional on $\mathbf{X}$: $E[\mathbf{ϵ}|\mathbf{X}]=0$.
4. Spherical Errors: The errors are homoscedastic ($\text{Var}({ϵ}_i)={\sigma }^2$ for all $i$) and not autocorrelated ($\text{Cov}({ϵ}_i,{ϵ}_j)=0$ for $i\ne j$). This is compactly written as $E[\mathbf{ϵ}{\mathbf{ϵ}}^T|\mathbf{X}]={\sigma }^2\mathbf{I}$.

With these in place, we can state the theorem.

The Gauss-Markov Theorem:

Under the classical linear model assumptions 1-4, the Ordinary Least Squares (OLS) estimator $\hat{\mathbf{\beta }}$ is the Best Linear Unbiased Estimator (BLUE) of $\mathbf{\beta }$.

This means that no other estimator that is both linear in $\mathbf{y}$ and unbiased can have a smaller variance-covariance matrix than OLS.

3. Proof of the Gauss-Markov Theorem

Your preference for mathematical rigor will be satisfied here. The proof is elegant and demonstrates the power of the matrix algebra framework we've developed. We will construct a generic linear unbiased estimator and show that its variance must be greater than or equal to the OLS variance.

For a comprehensive guide through the proof, the Statlect article is an excellent resource.

Let's work through the logic together.

Step 1: Define an arbitrary alternative linear estimator.
Let $\tilde{\mathbf{\beta }}$ be any other linear estimator of $\mathbf{\beta }$. By definition of linearity, it can be written as:

$$\tilde{\mathbf{\beta }}=\mathbf{Cy}$$

where $\mathbf{C}$ is some $k\times n$ matrix. We can express $\mathbf{C}$ in a clever way by relating it to the OLS estimator's matrix, $({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T$. Let:

$$\mathbf{C}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T+\mathbf{D}$$

where $\mathbf{D}$ is another $k\times n$ matrix that represents the difference between our new estimator and OLS.

Step 2: Impose the unbiasedness constraint.
We require $\tilde{\mathbf{\beta }}$ to be unbiased, meaning $E[\tilde{\mathbf{\beta }}]=\mathbf{\beta }$.

$$\begin{array}{rl}E[\tilde{\mathbf{\beta }}]& =E[\mathbf{Cy}]=E[\mathbf{C}(\mathbf{X}\mathbf{\beta }+\mathbf{ϵ})]\\ & =E[(\mathbf{CX})\mathbf{\beta }+\mathbf{C}\mathbf{ϵ}]\\ & =(\mathbf{CX})\mathbf{\beta }+\mathbf{C}E[\mathbf{ϵ}]\\ & =(\mathbf{CX})\mathbf{\beta }\end{array}$$

For this to equal $\mathbf{\beta }$ for any possible $\mathbf{\beta }$, we must have $\mathbf{CX}=\mathbf{I}$.

Now substitute our definition of $\mathbf{C}$:

$$\mathbf{CX}=\left(({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T+\mathbf{D}\right)\mathbf{X}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}({\mathbf{X}}^T\mathbf{X})+\mathbf{DX}=\mathbf{I}+\mathbf{DX}$$

For $\mathbf{CX}=\mathbf{I}$ to hold, we must have $\mathbf{I}+\mathbf{DX}=\mathbf{I}$, which implies the crucial condition:

$$\mathbf{DX}=0$$

Step 3: Derive the variance of $\tilde{\mathbf{\beta }}$.
The variance of our alternative estimator is:

$$\text{Var}(\tilde{\mathbf{\beta }})=\text{Var}(\mathbf{Cy})=\mathbf{C}\text{Var}(\mathbf{y}){\mathbf{C}}^T=\mathbf{C}({\sigma }^2\mathbf{I}){\mathbf{C}}^T={\sigma }^2\mathbf{C}{\mathbf{C}}^T$$

Now substitute $\mathbf{C}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T+\mathbf{D}$ and expand:

$$\begin{array}{rl}\text{Var}(\tilde{\mathbf{\beta }})& ={\sigma }^2\left(({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T+\mathbf{D}\right){\left(({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T+\mathbf{D}\right)}^T\\ & ={\sigma }^2\left(({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T+\mathbf{D}\right)\left(\mathbf{X}({\mathbf{X}}^T\mathbf{X}{)}^{-1}+{\mathbf{D}}^T\right)\end{array}$$

Expanding the brackets gives four terms:

1. $({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{X}({\mathbf{X}}^T\mathbf{X}{)}^{-1}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}$
2. $({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T{\mathbf{D}}^T=(\mathbf{DX}({\mathbf{X}}^T\mathbf{X}{)}^{-1}{)}^T$. Since $\mathbf{DX}=0$, this term is $0$.
3. $\mathbf{DX}({\mathbf{X}}^T\mathbf{X}{)}^{-1}$. Since $\mathbf{DX}=0$, this term is $0$.
4. $\mathbf{D}{\mathbf{D}}^T$

Step 4: Combine terms and conclude.
Putting it all together, the cross-product terms vanish, and we are left with:

$$\text{Var}(\tilde{\mathbf{\beta }})={\sigma }^2\left(({\mathbf{X}}^T\mathbf{X}{)}^{-1}+\mathbf{D}{\mathbf{D}}^T\right)$$

Recognizing that $\text{Var}({\hat{\mathbf{\beta }}}_{OLS})={\sigma }^2({\mathbf{X}}^T\mathbf{X}{)}^{-1}$, we arrive at the final result:

$$\text{Var}(\tilde{\mathbf{\beta }})=\text{Var}({\hat{\mathbf{\beta }}}_{OLS})+{\sigma }^2\mathbf{D}{\mathbf{D}}^T$$

This equation is the heart of the theorem. It shows that the variance of any linear unbiased estimator $\tilde{\mathbf{\beta }}$ is equal to the variance of the OLS estimator plus another matrix, ${\sigma }^2\mathbf{D}{\mathbf{D}}^T$.

The matrix $\mathbf{D}{\mathbf{D}}^T$ is positive semi-definite. For any vector $\mathbf{z}$, ${\mathbf{z}}^T(\mathbf{D}{\mathbf{D}}^T)\mathbf{z}=({\mathbf{D}}^T\mathbf{z}{)}^T({\mathbf{D}}^T\mathbf{z})=||{\mathbf{D}}^T\mathbf{z}|{|}^2\ge 0$. This confirms that $\text{Var}(\tilde{\mathbf{\beta }})-\text{Var}({\hat{\mathbf{\beta }}}_{OLS})$ is a positive semi-definite matrix. Therefore, the OLS estimator has the "smallest" variance. It is the Best Linear Unbiased Estimator.

This video provides a great visualization of the final algebraic steps of the proof.

Test your understanding!

According to our proof, the variance of any other linear unbiased estimator $\tilde{\mathbf{\beta }}$ is given by $\text{Var}(\tilde{\mathbf{\beta }})=\text{Var}({\hat{\mathbf{\beta }}}_{OLS})+{\sigma }^2\mathbf{D}{\mathbf{D}}^T$.

Under what condition is $\text{Var}(\tilde{\mathbf{\beta }})=\text{Var}({\hat{\mathbf{\beta }}}_{OLS})$? What does this imply about the estimator $\tilde{\mathbf{\beta }}$?

Show answer

The variances are equal if and only if the additional term ${\sigma }^2\mathbf{D}{\mathbf{D}}^T$ is a matrix of zeros. Since ${\sigma }^2>0$, this requires $\mathbf{D}{\mathbf{D}}^T=0$, which in turn implies that the matrix $\mathbf{D}$ must be the zero matrix, $\mathbf{D}=0$.

If $\mathbf{D}=0$, then our alternative estimator matrix $\mathbf{C}$ becomes $\mathbf{C}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T+0$. This means that $\tilde{\mathbf{\beta }}=\mathbf{Cy}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$, which is precisely the OLS estimator.

This demonstrates that the only linear unbiased estimator with the minimum possible variance is the OLS estimator itself. It is not just an optimal estimator; it is the unique optimal estimator in its class.

Conclusion

Today, we have reached a major milestone in our theoretical understanding of linear regression. We formally stated and proved the Gauss-Markov theorem, the cornerstone result that justifies the widespread use of OLS.

Key Takeaways:

• The Gauss-Markov theorem states that under the classical linear model assumptions, the OLS estimator is the Best Linear Unbiased Estimator (BLUE).
• Best signifies that OLS has the minimum variance among the class of all linear and unbiased estimators.
• The proof hinges on defining an arbitrary linear unbiased estimator and demonstrating that its variance matrix is equal to the OLS variance matrix plus a positive semi-definite matrix, proving the optimality of OLS.
• This theorem gives us strong theoretical confidence in OLS. When its assumptions hold, we are using the most precise linear unbiased estimator possible.

Preview of the Next Lesson:
Having established the optimality of the OLS estimator for the coefficients $\mathbf{\beta }$, we will shift our focus from the properties of the estimator to the properties of the overall model. In the next lesson, we will explore how to quantify how well our model explains the variation in the data by defining residuals and deriving the coefficient of determination (R-squared).