Deriving the Best Linear Unbiased Estimator (BLUE)

1. The General Linear Model (Aitken Model)

Let's begin by formally stating the model we'll be working with. The structure is the same as before, but the assumption on the error covariance is relaxed.

The general linear model, also known as the Aitken model, is given by:
$\mathbf{y}=\mathbf{X}\mathbf{\beta }+\mathbf{ϵ}$
subject to the following assumptions:

1. $E[\mathbf{ϵ}]=0$ (The errors have zero mean).
2. $\text{Cov}(\mathbf{ϵ})=\mathbf{C}$, where $\mathbf{C}$ is a known $n\times n$ positive definite matrix.
3. $\mathbf{X}$ is a fixed $n\times p$ matrix with full column rank.

Notice that if $\mathbf{C}={\sigma }^2\mathbf{I}$, we recover the standard Gauss-Markov model. The matrix $\mathbf{C}$ being non-diagonal allows for correlation between errors, and its diagonal elements not being equal allows for heteroscedasticity.

Our objective is to find an estimator $\hat{\mathbf{\beta }}=\mathbf{Ay}$ that is:

• Linear in $\mathbf{y}$.
• Unbiased, meaning $E[\hat{\mathbf{\beta }}]=\mathbf{\beta }$.
• Best, meaning it has the minimum variance among all linear unbiased estimators.

We will explore two powerful methods to derive this estimator.

2. Derivation via Whitening Transformation

The first method is a clever strategy: transform the general linear model into an equivalent one that satisfies the Gauss-Markov assumptions. We can then use the OLS formula we already know on the transformed model. This process is often called "whitening" because it transforms correlated errors into "white noise" (uncorrelated errors with constant variance).

The following reading introduces the Aitken model and outlines this exact strategy.

Let's break down the key steps of this "whitening" derivation:

1. Find the Transformation Matrix: Since $\mathbf{C}$ is a positive definite symmetric matrix, it is invertible, and we can find a non-singular matrix $\mathbf{P}$ such that ${\mathbf{PCP}}^T=\mathbf{I}$. A standard choice for this is the matrix square root inverse, $\mathbf{P}={\mathbf{C}}^{-1/2}$.

2. Transform the Model: We pre-multiply our model equation by $\mathbf{P}$:
   $\mathbf{Py}=\mathbf{PX}\mathbf{\beta }+\mathbf{P}\mathbf{ϵ}$
   Let's define new, transformed variables:

   • ${\mathbf{y}}^{\ast }=\mathbf{Py}$
   • ${\mathbf{X}}^{\ast }=\mathbf{PX}$
   • ${\mathbf{ϵ}}^{\ast }=\mathbf{P}\mathbf{ϵ}$

   The model now becomes ${\mathbf{y}}^{\ast }={\mathbf{X}}^{\ast }\mathbf{\beta }+{\mathbf{ϵ}}^{\ast }$.

3. Check the New Error Term: Let's find the mean and covariance of ${\mathbf{ϵ}}^{\ast }$:

   • Mean: $E[{\mathbf{ϵ}}^{\ast }]=E[\mathbf{P}\mathbf{ϵ}]=\mathbf{P}E[\mathbf{ϵ}]=0$.
   • Covariance: $\text{Cov}({\mathbf{ϵ}}^{\ast })=\text{Cov}(\mathbf{P}\mathbf{ϵ})=\mathbf{P}\text{Cov}(\mathbf{ϵ}){\mathbf{P}}^T={\mathbf{PCP}}^T=\mathbf{I}$.

   The transformed model ${\mathbf{y}}^{\ast }={\mathbf{X}}^{\ast }\mathbf{\beta }+{\mathbf{ϵ}}^{\ast }$ satisfies the Gauss-Markov assumptions!

4. Apply OLS to the Transformed Model: From our previous lesson, we know the BLUE for $\mathbf{\beta }$ in this transformed model is given by the OLS formula:
   ${\hat{\mathbf{\beta }}}_{GLS}=({\mathbf{X}}^{\ast T}{\mathbf{X}}^{\ast }{)}^{-1}{\mathbf{X}}^{\ast T}{\mathbf{y}}^{\ast }$

5. Substitute Back: Now, we substitute the original variables back into this expression:

   $$\begin{array}{rl}{\hat{\mathbf{\beta }}}_{GLS}& ={\left((\mathbf{PX}{)}^T(\mathbf{PX})\right)}^{-1}(\mathbf{PX}{)}^T(\mathbf{Py})\\ & ={\left({\mathbf{X}}^T{\mathbf{P}}^T\mathbf{PX}\right)}^{-1}{\mathbf{X}}^T{\mathbf{P}}^T\mathbf{Py}\end{array}$$

   Since $\mathbf{P}={\mathbf{C}}^{-1/2}$, we have ${\mathbf{P}}^T\mathbf{P}=({\mathbf{C}}^{-1/2}{)}^T({\mathbf{C}}^{-1/2})={\mathbf{C}}^{-1}$ (as ${\mathbf{C}}^{-1/2}$ is symmetric). Substituting this gives us the final form of the Generalized Least Squares (GLS) estimator:
   ${\hat{\mathbf{\beta }}}_{GLS}=({\mathbf{X}}^T{\mathbf{C}}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^T{\mathbf{C}}^{-1}\mathbf{y}$

This elegant result shows that the BLUE in the general case is a "generalized" form of the OLS estimator, where the data ($\mathbf{X}$ and $\mathbf{y}$) are weighted by the inverse of the error covariance matrix.

3. Derivation via Constrained Optimization

The whitening approach is intuitive, but a more direct and fundamental way to derive the BLUE is by formulating it as a constrained optimization problem. This method aligns perfectly with your interest in rigorous mathematical proofs.

The problem is:
Minimize the variance of the estimator, trace(Var(β_hat)).
Subject to the constraint that the estimator is unbiased, E[β_hat] = β.

The following lecture notes walk through this derivation using the method of Lagrange multipliers.

Let's recap the essential logic from the notes:

1. Estimator and Constraint: We define a linear estimator $\hat{\mathbf{\theta }}=\mathbf{Ay}$ (using the notation from the slides, with $\mathbf{\theta }$ instead of $\mathbf{\beta }$ and $\mathbf{H}$ instead of $\mathbf{X}$). The unbiasedness constraint $E[\mathbf{Ay}]=\mathbf{\theta }$ leads to the condition $\mathbf{AX}=\mathbf{I}$.

2. Objective Function: The variance of the estimator is $\text{Var}(\hat{\mathbf{\theta }})=\mathbf{A}\text{Cov}(\mathbf{y}){\mathbf{A}}^T={\mathbf{ACA}}^T$. We want to minimize its trace.

3. Lagrange Multipliers: The problem is to minimize $\text{trace}({\mathbf{ACA}}^T)$ subject to $\mathbf{AX}=\mathbf{I}$. Using a matrix of Lagrange multipliers $\mathbf{\Lambda }$, the Lagrangian is:
   $\mathcal{L}(\mathbf{A},\mathbf{\Lambda })=\text{trace}({\mathbf{ACA}}^T)+\text{trace}\left({\mathbf{\Lambda }}^T(\mathbf{AX}-\mathbf{I})\right)$

4. Solving the System: By taking the matrix derivative with respect to $\mathbf{A}$ and setting it to zero, and then using the constraint, the notes derive the solution for the optimal $\mathbf{A}$:
   $\mathbf{A}=({\mathbf{X}}^T{\mathbf{C}}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^T{\mathbf{C}}^{-1}$
   This gives us exactly the same GLS estimator as the whitening method.

Having derived the same result from two different perspectives strengthens our understanding. The whitening method provides the "why" (it's just OLS in disguise), while the optimization method provides a direct, rigorous proof of optimality.

4. Properties of the GLS Estimator

Now that we have the form of the BLUE, let's determine its covariance matrix. This tells us the precision of our new estimator.

The calculation is a straightforward application of the variance formula for linear transformations.

Let's write it out. Let $\mathbf{A}=({\mathbf{X}}^T{\mathbf{C}}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^T{\mathbf{C}}^{-1}$.

So, the covariance of the GLS estimator is $\text{Cov}({\hat{\mathbf{\beta }}}_{GLS})=({\mathbf{X}}^T{\mathbf{C}}^{-1}\mathbf{X}{)}^{-1}$. This is a beautifully compact result.

Special Case: Weighted Least Squares (WLS)

A very intuitive special case of GLS is when the errors are uncorrelated but heteroscedastic. In this scenario, the covariance matrix $\mathbf{C}$ is diagonal:
$\mathbf{C}=\left(\begin{array}{cccc}{\sigma }_1^2& 0& \dots & 0\\ 0& {\sigma }_2^2& \dots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \dots & {\sigma }_n^2\end{array}\right)$
Then, its inverse ${\mathbf{C}}^{-1}$ is also diagonal, with entries $1/{\sigma }_i^2$. The GLS estimator minimizes the quadratic form $(\mathbf{y}-\mathbf{X}\mathbf{\beta }{)}^T{\mathbf{C}}^{-1}(\mathbf{y}-\mathbf{X}\mathbf{\beta })$, which in this case becomes:
${\sum }_{i=1}^n\frac{(y_i-{\mathbf{x}}_i^T\mathbf{\beta }{)}^2}{{\sigma }_i^2}$
This is known as Weighted Least Squares (WLS). It minimizes the sum of squared errors, where each error is weighted by the inverse of its variance. This makes perfect intuitive sense: observations with smaller variance (i.e., more precision) are given more weight in the estimation of $\mathbf{\beta }$.

Conclusion

In this lesson, we have moved beyond the standard Gauss-Markov assumptions to derive the Best Linear Unbiased Estimator for the general linear model, where errors can be correlated and heteroscedastic.

Key Takeaways:

• The General Linear Model (Aitken Model): This model allows for a general error covariance matrix, $\text{Cov}(\mathbf{ϵ})=\mathbf{C}$, making it more flexible than the standard model.
• The BLUE is the GLS Estimator: We derived, through two different methods, that the BLUE for this general model is the Generalized Least Squares (GLS) estimator:
  ${\hat{\mathbf{\beta }}}_{GLS}=({\mathbf{X}}^T{\mathbf{C}}^{-1}\mathbf{X}{)}^{-1}{\mathbf{X}}^T{\mathbf{C}}^{-1}\mathbf{y}$
• GLS Generalizes OLS: If the errors are spherical ($\mathbf{C}={\sigma }^2\mathbf{I}$), the GLS formula simplifies to the familiar OLS formula.
• Intuition of Whitening and Weighting: The GLS estimator can be understood as either applying OLS to a "whitened" version of the data or as a "weighted" least squares problem where data points are weighted by the inverse of the error covariance.
• Covariance of the GLS Estimator: The performance of the estimator is given by its covariance matrix, $\text{Cov}({\hat{\mathbf{\beta }}}_{GLS})=({\mathbf{X}}^T{\mathbf{C}}^{-1}\mathbf{X}{)}^{-1}$.

You have now mastered the frequentist approach to optimal linear estimation. We've seen how to find the "best" estimator under different assumptions about the data-generating process.

Preview of the Next Lesson:

We now pivot from the frequentist paradigm to the Bayesian school of thought. In the upcoming lessons, we will no longer treat parameters like $\mathbf{\beta }$ as fixed, unknown constants. Instead, we will treat them as random variables with their own probability distributions. You will learn how to combine prior beliefs about a parameter with observed data to form an updated posterior belief. This will lead us to new types of estimators, such as the Minimum Mean Square Error (MMSE) and Maximum A Posteriori (MAP) estimators.