Unbiasedness of OLS Estimators

Hello! Welcome back to your study of linear regression.

In our last lesson, we established a powerful probabilistic justification for Ordinary Least Squares (OLS). We showed that by assuming the errors in our model are independent and normally distributed, the OLS solution is precisely the Maximum Likelihood Estimate (MLE) for the regression coefficients. This gave a deep statistical meaning to the "sum of squared errors" criterion.

Having derived the OLS estimator $\hat{\mathbf{\beta }}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$, our focus now shifts to analyzing its properties as a statistical estimator. How reliable is it? Does it, on average, give us the right answer? Today, we will answer this question by addressing a fundamental property: unbiasedness. Our learning outcome is to prove the unbiasedness of the OLS estimators under the classical linear model assumptions.

This lesson marks our transition from finding the estimator to evaluating it, a critical step in rigorous model-building.

1. What is an Unbiased Estimator?

Before diving into the proof, let's clarify what "unbiased" means. An estimator is a function of our sample data that gives us an estimate of an unknown population parameter. Since our sample is random, the estimate itself is a random variable—if we were to draw a different sample, we would get a different estimate.

An estimator is called unbiased if the mean of its sampling distribution is equal to the true, unknown population parameter. In other words, while any single estimate might be higher or lower than the true value, the estimation procedure does not systematically over- or underestimate the parameter. On average, it hits the target.

The image below provides a clear visual. The "Unbiased Estimator's Distribution" is centered on the true parameter $\theta$, while the "Biased Estimator's Distribution" is systematically off-center.

For our OLS estimator $\hat{\mathbf{\beta }}$, unbiasedness means that $E[\hat{\mathbf{\beta }}]=\mathbf{\beta }$.

2. The Classical Linear Model (CLM) Assumptions

To prove that the OLS estimator is unbiased, we must first lay out the set of assumptions under which this property holds. These are often called the Gauss-Markov assumptions or the Classical Linear Model (CLM) assumptions. Your familiarity with formal definitions from your previous coursework will be helpful here.

To summarize the key assumptions for our proof today:

1. Linearity in Parameters: The true relationship between $\mathbf{y}$ and $\mathbf{X}$ is linear: $\mathbf{y}=\mathbf{X}\mathbf{\beta }+\mathbf{ϵ}$.
2. Full Rank: The $n\times k$ matrix $\mathbf{X}$ has full column rank, meaning its columns are linearly independent (no perfect multicollinearity). This ensures that $({\mathbf{X}}^T\mathbf{X}{)}^{-1}$ exists.
3. Zero Conditional Mean: The expected value of the error term, conditional on the regressors, is zero. $$E[\mathbf{ϵ}|\mathbf{X}]=0$$ This is the most critical assumption for proving unbiasedness. It implies that the independent variables do not contain any information about the expected value of the error. In practical terms, it means there are no other systematic factors related to $\mathbf{X}$ that influence $\mathbf{y}$ and have been omitted from the model.

Note that the normality assumption (${ϵ}_i\sim \mathcal{N}(0,{\sigma }^2)$) we used for the MLE derivation in the last lesson is not required to prove unbiasedness. Unbiasedness is a more general property.

3. The Proof of Unbiasedness

We are now ready to prove that $E[\hat{\mathbf{\beta }}]=\mathbf{\beta }$. The proof is elegant and relies on matrix algebra, which should feel familiar given your work in previous modules.

To start, let's watch a short video that walks through the derivation step-by-step. This will help you visualize the flow of the proof before we formalize it.

Let's now formally write out the proof, following the same logic.

Step 1: Start with the definition of the OLS estimator.

$$\hat{\mathbf{\beta }}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$$

Step 2: Substitute the true linear model, $\mathbf{y}=\mathbf{X}\mathbf{\beta }+\mathbf{ϵ}$, into the equation.

$$\hat{\mathbf{\beta }}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T(\mathbf{X}\mathbf{\beta }+\mathbf{ϵ})$$

Step 3: Distribute the $({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T$ term.

$$\hat{\mathbf{\beta }}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{X}\mathbf{\beta }+({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}$$

Step 4: Simplify the first term.
Since $({\mathbf{X}}^T\mathbf{X}{)}^{-1}({\mathbf{X}}^T\mathbf{X})=\mathbf{I}$ (the identity matrix), the equation becomes:

$$\hat{\mathbf{\beta }}=\mathbf{I}\mathbf{\beta }+({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}=\mathbf{\beta }+({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}$$

This is a key result. It shows that our estimator, $\hat{\mathbf{\beta }}$, is equal to the true parameter, $\mathbf{\beta }$, plus a term that depends on the unobservable errors. The estimator is unbiased if and only if the expected value of this error-dependent term is zero.

Step 5: Take the expectation of both sides, conditional on $\mathbf{X}$.
Treating $\mathbf{X}$ as fixed (or conditioning upon it) is a standard technique.

$$E[\hat{\mathbf{\beta }}|\mathbf{X}]=E[\mathbf{\beta }|\mathbf{X}]+E[({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}|\mathbf{X}]$$

Step 6: Evaluate each term.

• The true parameter vector $\mathbf{\beta }$ is a fixed, non-random constant. Therefore, $E[\mathbf{\beta }|\mathbf{X}]=\mathbf{\beta }$.
• For the second term, since we are conditioning on $\mathbf{X}$, the matrix $({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T$ is treated as a fixed quantity and can be pulled out of the expectation. $$E[({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}|\mathbf{X}]=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^{T}E[\mathbf{ϵ}|\mathbf{X}]$$

Step 7: Apply the Zero Conditional Mean assumption.
Using Assumption 3, we know that $E[\mathbf{ϵ}|\mathbf{X}]=0$. Substituting this in:

$$({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^{T}0=0$$

Step 8: Combine the results.
We now have:

$$E[\hat{\mathbf{\beta }}|\mathbf{X}]=\mathbf{\beta }+0=\mathbf{\beta }$$

Since the expected value of $\hat{\mathbf{\beta }}$ is $\mathbf{\beta }$ for any given $\mathbf{X}$, its unconditional expectation must also be $\mathbf{\beta }$ (by the law of total expectation: $E[\hat{\mathbf{\beta }}]=E_{\mathbf{X}}[E[\hat{\mathbf{\beta }}|\mathbf{X}]]=E_{\mathbf{X}}[\mathbf{\beta }]=\mathbf{\beta }$).

Thus, we have proven that the OLS estimator $\hat{\mathbf{\beta }}$ is an unbiased estimator of $\mathbf{\beta }$. Q.E.D.

For a formal write-up that mirrors this logic, you can refer back to the Stanford notes.

Test your understanding!

In the proof, we showed that $E[({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}]=0$. Which specific CLM assumption was essential for this step, and what would happen to the proof if this assumption were violated? For example, what if $E[\mathbf{ϵ}|\mathbf{X}]=\mathbf{c}$, where $\mathbf{c}$ is some non-zero constant vector?

Show answer

The essential assumption is the Zero Conditional Mean assumption ($E[\mathbf{ϵ}|\mathbf{X}]=0$).

If this assumption were violated, and instead $E[\mathbf{ϵ}|\mathbf{X}]=\mathbf{c}\ne 0$, the proof would fail at Step 7. The expectation of the error-dependent term would be:

$$E[({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{ϵ}|\mathbf{X}]=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^{T}E[\mathbf{ϵ}|\mathbf{X}]=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{c}$$

This term is generally not zero. The expected value of our estimator would then be:

$$E[\hat{\mathbf{\beta }}|\mathbf{X}]=\mathbf{\beta }+({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{c}$$

Since $E[\hat{\mathbf{\beta }}]\ne \mathbf{\beta }$, the OLS estimator would be biased. The magnitude and direction of this bias would depend on the relationship between the regressors ($\mathbf{X}$) and the systematic, non-zero errors ($\mathbf{c}$). This situation is known as omitted variable bias.

Conclusion

In this lesson, we have formally proven a cornerstone property of the Ordinary Least Squares estimator. By leveraging a core set of assumptions about our model, we demonstrated that OLS provides an unbiased estimate of the true population parameters.

Key Takeaways:

• An estimator is unbiased if its expected value (the mean of its sampling distribution) is equal to the true population parameter it is intended to estimate.
• The proof of unbiasedness for the OLS estimator $\hat{\mathbf{\beta }}$ relies critically on the Classical Linear Model (CLM) assumptions, especially the Zero Conditional Mean assumption ($E[\mathbf{ϵ}|\mathbf{X}]=0$).
• The proof involves substituting the true linear model into the OLS formula and taking the expectation, which simplifies to $E[\hat{\mathbf{\beta }}]=\mathbf{\beta }$.
• This property assures us that the OLS estimation procedure is "correct on average," even if any single estimate will deviate from the true value due to random sampling error.

Preview of the Next Lesson:
Unbiasedness is a good start, but it's only half the story. An estimator could be unbiased but have an enormous variance, meaning its estimates are wildly spread out and unreliable. We also want an estimator that is precise. In our next lesson, we will derive the variance-covariance matrix of the OLS estimators. This will allow us to quantify the uncertainty in our estimates and will lead us directly to the famous Gauss-Markov theorem, which states that OLS is the "Best Linear Unbiased Estimator" (BLUE).