OLS Estimator Variance-Covariance Matrix

Hello! Welcome back to our exploration of the statistical properties of linear regression.

In the last lesson, we established that the Ordinary Least Squares (OLS) estimator is unbiased, meaning that on average, it correctly identifies the true population parameters ($E[\hat{\mathbf{\beta }}]=\mathbf{\beta }$). However, being correct on average is not enough. An estimator whose estimates are wildly scattered around the true value is not very useful. We also need our estimator to be precise.

Today, we will quantify this precision. Our learning outcome is to derive the variance-covariance matrix of the OLS estimators. This will give us a formula for the uncertainty associated with our coefficient estimates. This derivation is a cornerstone of econometrics and statistical learning, and your comfort with formal proofs will be key to mastering it.

This lesson directly builds on our unbiasedness proof and sets the stage for the famous Gauss-Markov theorem, which we will tackle next.

1. Defining the Variance-Covariance Matrix

For a single random variable, variance is a scalar that measures its spread. For a vector of random variables, like our estimator $\hat{\mathbf{\beta }}$, this concept extends to a variance-covariance matrix.

The variance-covariance matrix of a random vector $\mathbf{Z}$ is formally defined as:

$$\text{Var}(\mathbf{Z})=E[(\mathbf{Z}-E[\mathbf{Z}])(\mathbf{Z}-E[\mathbf{Z}]{)}^T]$$

Since we proved in the last lesson that $E[\hat{\mathbf{\beta }}]=\mathbf{\beta }$, the variance-covariance matrix for our OLS estimator is:

$$\text{Var}(\hat{\mathbf{\beta }})=E[(\hat{\mathbf{\beta }}-\mathbf{\beta })(\hat{\mathbf{\beta }}-\mathbf{\beta }{)}^T]$$

If $\hat{\mathbf{\beta }}$ is a $k\times 1$ vector, then $(\hat{\mathbf{\beta }}-\mathbf{\beta })$ is $k\times 1$ and $(\hat{\mathbf{\beta }}-\mathbf{\beta }{)}^T$ is $1\times k$. Their product is a $k\times k$ matrix:

$$\text{Var}(\hat{\mathbf{\beta }})=\left[\begin{array}{cccc}\text{Var}({\hat{\beta }}_1)& \text{Cov}({\hat{\beta }}_1,{\hat{\beta }}_2)& \cdots & \text{Cov}({\hat{\beta }}_1,{\hat{\beta }}_k)\\ \text{Cov}({\hat{\beta }}_2,{\hat{\beta }}_1)& \text{Var}({\hat{\beta }}_2)& \cdots & \text{Cov}({\hat{\beta }}_2,{\hat{\beta }}_k)\\ ⋮& ⋮& \ddots & ⋮\\ \text{Cov}({\hat{\beta }}_k,{\hat{\beta }}_1)& \text{Cov}({\hat{\beta }}_k,{\hat{\beta }}_2)& \cdots & \text{Var}({\hat{\beta }}_k)\end{array}\right]$$

The diagonal elements represent the variance of each individual coefficient estimate, telling us about its precision. The off-diagonal elements represent the covariance between pairs of coefficient estimates, telling us how they vary together.

2. Assumptions about the Error Term's Variance

In our proof of unbiasedness, the critical assumption was the zero conditional mean of the errors ($E[\mathbf{ϵ}|\mathbf{X}]=0$). For deriving the variance, we need an additional set of assumptions concerning the second moment of the errors.

This is Assumption 4 from the Classical Linear Model (CLM) assumptions, which we can revisit in the Stanford notes.

The assumption $E[\mathbf{ϵ}{\mathbf{ϵ}}^T|\mathbf{X}]={\sigma }^2\mathbf{I}$ can be broken down as follows:

1. Homoscedasticity (Constant Variance): The variance of each error term ${ϵ}_i$, conditional on $\mathbf{X}$, is a constant value ${\sigma }^2$.

   $$\text{Var}({ϵ}_i|\mathbf{X})=E[{ϵ}_i^2|\mathbf{X}]={\sigma }^2\text{for all }i=1,\dots ,n$$

   These are the diagonal elements of the $n\times n$ matrix $E[\mathbf{ϵ}{\mathbf{ϵ}}^T|\mathbf{X}]$.

2. No Autocorrelation (or Serial Correlation): Any two different error terms, ${ϵ}_i$ and ${ϵ}_j$, are uncorrelated.

   $$\text{Cov}({ϵ}_i,{ϵ}_j|\mathbf{X})=E[{ϵ}_i{ϵ}_j|\mathbf{X}]=0\text{for all }i\ne j$$

   These are the off-diagonal elements of the matrix, which are all zero.

Together, these assumptions mean the variance-covariance matrix of the error vector $\mathbf{ϵ}$ is a diagonal matrix with ${\sigma }^2$ on the diagonal, which is concisely written as ${\sigma }^2\mathbf{I}$. These errors are sometimes called spherical disturbances.

3. Derivation of the Variance-Covariance Matrix

With the definition and assumptions in place, we can proceed with the derivation. The process involves matrix algebra and applying the properties of the expectation operator.

To see the derivation unfold, let's start with a video that walks through the steps.

Now, let's formally construct the proof step-by-step. This derivation requires careful application of matrix transpose rules and the properties of expectation.

Step 1: Express the estimation error $\hat{\mathbf{\beta }}-\mathbf{\beta }$.
From our previous lesson on unbiasedness, we derived the crucial relationship:

$$\hat{\mathbf{\beta }}=\mathbf{\beta }+({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}$$

Therefore, the estimation error is:

$$\hat{\mathbf{\beta }}-\mathbf{\beta }=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}$$

Step 2: Substitute this into the definition of the variance-covariance matrix.

$$\text{Var}(\hat{\mathbf{\beta }})=E\left[\left(({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}\right){\left(({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}\right)}^T\right]$$

Step 3: Apply the transpose rule $(\mathbf{AB}{)}^T={\mathbf{B}}^T{\mathbf{A}}^T$ to the second term.

$${\left(({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}\right)}^T={\mathbf{ϵ}}^T({\mathbf{X}}^T{)}^T(({\mathbf{X}}^T\mathbf{X}{)}^{-1}{)}^T={\mathbf{ϵ}}^T\mathbf{X}(({\mathbf{X}}^T\mathbf{X}{)}^T{)}^{-1}$$

Since ${\mathbf{X}}^T\mathbf{X}$ is a symmetric matrix, $({\mathbf{X}}^T\mathbf{X}{)}^T={\mathbf{X}}^T\mathbf{X}$. The expression simplifies to:

$${\mathbf{ϵ}}^T\mathbf{X}({\mathbf{X}}^T\mathbf{X}{)}^{-1}$$

Step 4: Substitute back and rearrange.

$$\text{Var}(\hat{\mathbf{\beta }})=E\left[({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}{\mathbf{ϵ}}^T\mathbf{X}({\mathbf{X}}^T\mathbf{X}{)}^{-1}\right]$$

Step 5: Apply the expectation operator.
We take the expectation conditional on $\mathbf{X}$. The terms involving $\mathbf{X}$ can be treated as non-random and pulled outside the expectation.

$$\text{Var}(\hat{\mathbf{\beta }}|\mathbf{X})=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^{T}E[\mathbf{ϵ}{\mathbf{ϵ}}^T|\mathbf{X}]\mathbf{X}({\mathbf{X}}^T\mathbf{X}{)}^{-1}$$

Step 6: Apply the assumption of spherical errors.
Now we use our key assumption: $E[\mathbf{ϵ}{\mathbf{ϵ}}^T|\mathbf{X}]={\sigma }^2\mathbf{I}$.

$$\text{Var}(\hat{\mathbf{\beta }}|\mathbf{X})=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T({\sigma }^2\mathbf{I})\mathbf{X}({\mathbf{X}}^T\mathbf{X}{)}^{-1}$$

Step 7: Simplify the expression.
Since ${\sigma }^2$ is a scalar and $\mathbf{I}$ is the identity matrix, we can rearrange:

$$\text{Var}(\hat{\mathbf{\beta }}|\mathbf{X})={\sigma }^2({\mathbf{X}}^T\mathbf{X}{)}^{-1}({\mathbf{X}}^T\mathbf{X})({\mathbf{X}}^T\mathbf{X}{)}^{-1}$$

The middle terms cancel out: $({\mathbf{X}}^T\mathbf{X}{)}^{-1}({\mathbf{X}}^T\mathbf{X})=\mathbf{I}$.

$$\text{Var}(\hat{\mathbf{\beta }}|\mathbf{X})={\sigma }^2\mathbf{I}({\mathbf{X}}^T\mathbf{X}{)}^{-1}$$

Step 8: Final Result.
This gives us the celebrated formula for the variance-covariance matrix of the OLS estimators:

$$\text{Var}(\hat{\mathbf{\beta }})={\sigma }^2({\mathbf{X}}^T\mathbf{X}{)}^{-1}$$

The handwritten derivation below summarizes this fundamental result.

For a final review, you can see this complete derivation in the Stanford notes, which we've been using as our primary reference.

Test your understanding!

Suppose the assumption of no autocorrelation holds, but the errors are heteroskedastic. This means that $\text{Var}({ϵ}_i)={\sigma }_i^2$, where the variance is different for each observation. In this case, the error covariance matrix $\mathbf{\Omega }=E[\mathbf{ϵ}{\mathbf{ϵ}}^T]$ is a diagonal matrix but not ${\sigma }^2\mathbf{I}$.

What would the formula for $\text{Var}(\hat{\mathbf{\beta }})$ look like in this situation? At which step would our derivation change?

Show answer

The derivation would change at Step 6. We could no longer substitute $E[\mathbf{ϵ}{\mathbf{ϵ}}^T|\mathbf{X}]$ with ${\sigma }^2\mathbf{I}$. Instead, we would have to leave it as a general diagonal matrix $\mathbf{\Omega }$.

The formula from Step 5 would be:

$$\text{Var}(\hat{\mathbf{\beta }})=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{\Omega }\mathbf{X}({\mathbf{X}}^T\mathbf{X}{)}^{-1}$$

This more complex formula is known as the "sandwich estimator". Because the standard OLS formula ${\sigma }^2({\mathbf{X}}^T\mathbf{X}{)}^{-1}$ would be incorrect, the standard errors computed by basic regression software would be wrong, leading to invalid hypothesis tests. This is why testing for heteroscedasticity is a crucial part of applied regression analysis.

4. Interpretation and Estimation

The formula $\text{Var}(\hat{\mathbf{\beta }})={\sigma }^2({\mathbf{X}}^T\mathbf{X}{)}^{-1}$ is powerful. It tells us that the precision of our estimates depends on two things:

1. ${\sigma }^2$: The variance of the unobservable errors. The more "noisy" the underlying process is, the less precise our estimates will be.
2. $({\mathbf{X}}^T\mathbf{X}{)}^{-1}$: The data matrix of our regressors. A greater spread in the values of $\mathbf{X}$ and lower correlation between them (avoiding multicollinearity) leads to a "smaller" inverse matrix, and thus more precise estimates.

There is one practical problem: ${\sigma }^2$ is an unknown population parameter. We must estimate it from our sample data. The unbiased estimator for ${\sigma }^2$ is:

$$s^2={\hat{\sigma }}^2=\frac{{\mathbf{e}}^T\mathbf{e}}{n-k}$$

where $\mathbf{e}=\mathbf{y}-\mathbf{X}\hat{\mathbf{\beta }}$ is the vector of sample residuals, $n$ is the number of observations, and $k$ is the number of parameters (including the intercept). The denominator $n-k$ represents the degrees of freedom.

By plugging this in, we get the estimated variance-covariance matrix:

$$\widehat{\text{Var}}(\hat{\mathbf{\beta }})=s^2({\mathbf{X}}^T\mathbf{X}{)}^{-1}$$

The square roots of the diagonal elements of this matrix are the standard errors of the OLS coefficients, which you have likely seen in the output of statistical packages. These standard errors are the fundamental building blocks for constructing confidence intervals and conducting t-tests.

Conclusion

In this lesson, we completed a crucial step in understanding the properties of the OLS estimator. By adding the assumptions of homoscedasticity and no autocorrelation, we derived the formula for its variance-covariance matrix.

Key Takeaways:

• The variance of the OLS estimator $\hat{\mathbf{\beta }}$ is a $k\times k$ matrix containing the variances of each coefficient on its diagonal and the covariances in the off-diagonal positions.
• The derivation relies on the CLM assumption of spherical errors: $E[\mathbf{ϵ}{\mathbf{ϵ}}^T|\mathbf{X}]={\sigma }^2\mathbf{I}$.
• The final formula is $\text{Var}(\hat{\mathbf{\beta }})={\sigma }^2({\mathbf{X}}^T\mathbf{X}{)}^{-1}$.
• The precision of OLS estimates depends on the underlying error variance (${\sigma }^2$) and the structure of the data ($\mathbf{X}$).
• In practice, we use an estimate of the error variance, $s^2$, to compute the estimated variance-covariance matrix, from which we obtain the standard errors for inference.

Preview of the Next Lesson:
We now have the two main ingredients to evaluate the quality of the OLS estimator: it is unbiased, and we have an expression for its variance. The question remains: is there another linear and unbiased estimator that might have an even smaller variance? In our next lesson, we will prove the Gauss-Markov Theorem, which shows that OLS is, in fact, the Best Linear Unbiased Estimator (BLUE), meaning it has the minimum variance among all estimators in its class.