Properties of Estimators: Bias, Variance, and Consistency

Bias: The Accuracy of an Estimator

The first property we care about is bias. Intuitively, we want an estimator that is, on average, centered on the true parameter value.

An estimator $\hat{\theta }$ for a parameter $\theta$ is itself a random variable, because its value depends on the random sample we draw. The bias of the estimator is the difference between its expected value and the true value of the parameter.

Definition: Bias
The bias of an estimator $\hat{\theta }$ is given by:
$B(\hat{\theta })=E[\hat{\theta }]-\theta$
An estimator is said to be unbiased if its bias is zero for all possible values of $\theta$. This means:
$E[\hat{\theta }]=\theta$
An unbiased estimator is "correct on average." If we were to draw an infinite number of samples and compute our estimate for each one, the average of all those estimates would be equal to the true parameter.

Example 1: The Sample Mean is an Unbiased Estimator

Let's prove that the sample mean, $\bar{X}=\frac{1}{n}{\sum }_{i=1}^n{X}_i$, is an unbiased estimator for the population mean, $\mu$. The following video provides a concise introduction to estimators and a clear proof.

As the video demonstrated, the proof relies on the linearity of expectation:
$E[\bar{X}]=E\left[\frac{1}{n}{\sum }_{i=1}^n{X}_i\right]=\frac{1}{n}{\sum }_{i=1}^{n}E[X_i]$
Since each $X_i$ is drawn from the same population, $E[X_i]=\mu$ for all $i$.
$E[\bar{X}]=\frac{1}{n}{\sum }_{i=1}^n\mu =\frac{1}{n}(n\mu )=\mu$
Thus, the sample mean $\bar{X}$ is an unbiased estimator of $\mu$.

Example 2: Sample Variance and Bessel's Correction

In the last lesson, I pointed out the curious $n-1$ in the denominator of the sample variance formula. The reason for this is to ensure the estimator is unbiased. Let's define two potential estimators for the population variance ${\sigma }^2$:

1. The "naive" estimator: ${\hat{\sigma }}_n^2=\frac{1}{n}{\sum }_{i=1}^n(X_i-\bar{X}{)}^2$
2. The standard sample variance: $S^2=\frac{1}{n-1}{\sum }_{i=1}^n(X_i-\bar{X}{)}^2$

It turns out that $E[{\hat{\sigma }}_n^2]\ne {\sigma }^2$, making it a biased estimator. The division by $n-1$ in $S^2$ is called Bessel's correction, and it's precisely what's needed to make the estimator unbiased. The next segment of the same video provides a full derivation.

The key takeaway from that detailed proof is:
$E\left[\frac{1}{n}\sum (X_i-\bar{X}{)}^2\right]=\frac{n-1}{n}{\sigma }^2$
This shows that the naive estimator, on average, underestimates the true variance. By multiplying by the correction factor $\frac{n}{n-1}$, we get our unbiased estimator:
$E[S^2]=E\left[\frac{1}{n-1}\sum (X_i-\bar{X}{)}^2\right]=\frac{n}{n-1}E\left[\frac{1}{n}\sum (X_i-\bar{X}{)}^2\right]=\frac{n}{n-1}\left(\frac{n-1}{n}{\sigma }^2\right)={\sigma }^2$

Variance: The Precision of an Estimator

Knowing an estimator is unbiased is good, but it's not the whole story. We also need it to be precise. The variance of an estimator measures how much the estimate varies from one sample to the next.

Definition: Variance of an Estimator
The variance of an estimator $\hat{\theta }$ is given by:
$Var(\hat{\theta })=E\left[(\hat{\theta }-E[\hat{\theta }]{)}^2\right]$
A smaller variance is better. A low-variance estimator is reliable and will give you similar results each time you draw a sample.

Referring back to the target diagram, variance is about the spread of the shots. The left column shows low-variance (tightly clustered) estimators, while the right column shows high-variance (spread out) estimators.

For example, the variance of the sample mean is:
$Var(\bar{X})=Var\left(\frac{1}{n}{\sum }_{i=1}^n{X}_i\right)=\frac{1}{n^2}{\sum }_{i=1}^{n}Var(X_i)=\frac{1}{n^2}(n{\sigma }^2)=\frac{{\sigma }^2}{n}$
This confirms our intuition: as the sample size $n$ increases, the variance of the sample mean decreases, meaning our estimate becomes more precise.

The Bias-Variance Tradeoff: Mean Squared Error

We want low bias and low variance. But what if we have to choose between an unbiased estimator with high variance and a slightly biased estimator with low variance? We need a single metric that combines both. This metric is the Mean Squared Error (MSE).

Definition: Mean Squared Error (MSE)
The MSE of an estimator $\hat{\theta }$ is the expected squared difference between the estimator and the true parameter:
$MSE(\hat{\theta })=E\left[(\hat{\theta }-\theta {)}^2\right]$
The MSE is arguably the most important measure of an estimator's overall quality. A remarkable result is that the MSE can be decomposed into two components: the variance and the square of the bias.

Theorem: Bias-Variance Decomposition
For any estimator $\hat{\theta }$, the following holds:
$MSE(\hat{\theta })=Var(\hat{\theta })+[B(\hat{\theta }){]}^2$

Let's watch a short video that walks through the proof of this fundamental identity.

The proof involves a simple but powerful trick: adding and subtracting $E[\hat{\theta }]$.
\begin{align*}
MSE(\hat{\theta}) &= E[(\hat{\theta} - \theta)^2] \
&= E[(\hat{\theta} - E[\hat{\theta}] + E[\hat{\theta}] - \theta)^2] \
&= E[(\underbrace{\hat{\theta} - E[\hat{\theta}]}{A})^2 + 2(\underbrace{\hat{\theta} - E[\hat{\theta}]}{A})(\underbrace{E[\hat{\theta}] - \theta}{B}) + (\underbrace{E[\hat{\theta}] - \theta}{B})^2]
\end{align*}
By linearity of expectation, we can split this up. The middle term becomes zero, and the first and last terms become the definitions of variance and squared bias, respectively.

This decomposition is central to machine learning and statistics. It tells us that the total error of an estimator comes from two sources: error from being off-target on average (bias) and error from being imprecise (variance).

Consistency: Asymptotic Correctness

Bias and variance are properties for a fixed sample size $n$. Consistency is an asymptotic property: it describes what happens as our sample size grows infinitely large.

Definition: Consistency
An estimator ${\hat{\theta }}_n$ (where the subscript $n$ emphasizes its dependence on sample size) is consistent if it converges in probability to the true parameter $\theta$ as $n\to \infty$. Formally, for any small number $ϵ>0$:
${\lim }_{n\to \infty }P(|{\hat{\theta }}_n-\theta |>ϵ)=0$
In simple terms, a consistent estimator is one that is guaranteed to get arbitrarily close to the true value if you collect enough data.

How can we prove an estimator is consistent? A straightforward sufficient (but not necessary) condition is to show that both its bias and its variance go to zero as $n\to \infty$.

For example, the sample mean $\bar{X}$ is consistent for $\mu$ because:

1. Bias: $B(\bar{X})=0$ for all $n$, so ${\lim }_{n\to \infty }B(\bar{X})=0$.
2. Variance: $Var(\bar{X})=\frac{{\sigma }^2}{n}$, so ${\lim }_{n\to \infty }Var(\bar{X})=0$.

Since both conditions are met, $\bar{X}$ is a consistent estimator.

Consistency vs. Unbiasedness

It's crucial not to confuse these two properties.

• Unbiasedness is a finite-sample property. It says that for a given $n$, the estimator is correct on average.
• Consistency is a large-sample property. It says the estimator converges to the right answer as $n$ grows.

Can an estimator be one but not the other? Absolutely. The following reading gives excellent examples.

Here's a summary of the key ideas from that text:

1. Unbiased and Consistent: The sample mean $\bar{X}$.
2. Unbiased but Not Consistent: The estimator $\hat{\theta }=X_1$. Its expectation is $\mu$, so it's unbiased. But its variance is ${\sigma }^2$ and doesn't decrease as $n$ grows. It never improves, so it's not consistent.
3. Biased but Consistent: The MLE for the variance of a Normal distribution, ${\hat{\sigma }}_n^2$, is biased by a factor of $\frac{n-1}{n}$. However, as $n\to \infty$, this bias disappears ($\frac{n-1}{n}\to 1$). Since its variance also goes to zero, it is consistent.

Self-Assessment

Let's apply these concepts. Consider a random sample $X_1,...,X_n$ from a Uniform distribution $U(0,\theta )$. The population mean is $E[X]=\theta /2$.

An estimator for $\theta$ derived from the Method of Moments (which we will study soon) is ${\hat{\theta }}_{MoM}=2\bar{X}$.

1. Is ${\hat{\theta }}_{MoM}$ an unbiased estimator of $\theta$?
2. What is the variance of ${\hat{\theta }}_{MoM}$? (Hint: The variance of a $U(a,b)$ distribution is $(b-a{)}^2/12$).
3. Is ${\hat{\theta }}_{MoM}$ a consistent estimator of $\theta$?

Conclusion

In this lesson, we have built a formal framework for evaluating estimators. These properties are not just abstract definitions; they are the tools we use to decide which statistical methods to trust.

Key Takeaways:

• Bias measures the accuracy of an estimator on average ($E[\hat{\theta }]-\theta$). Unbiased estimators are correct on average.
• Variance measures the precision or reliability of an estimator ($E[(\hat{\theta }-E[\hat{\theta }]{)}^2]$). Low-variance estimators give consistent results across different samples.
• Mean Squared Error (MSE) provides a complete picture of an estimator's quality by combining both sources of error: $MSE=Var+Bia{s}^2$.
• Consistency is an asymptotic property ensuring that an estimator converges to the true parameter value as the amount of data increases.

Preview of the Next Lesson:

Now that we know how to judge an estimator, how do we find one in the first place? Is there a systematic way to derive good estimators from a given probability model?

In the next lesson, we will explore two of the most important principles for constructing estimators: the Method of Moments and Maximum Likelihood Estimation (MLE). These powerful techniques provide recipes for creating estimators that often have desirable properties like consistency and efficiency.