The Weak Law of Large Numbers

The Weak Law of Large Numbers

At its heart, the WLLN states that the sample mean of a large number of independent and identically distributed (i.i.d.) random variables is very likely to be close to the true mean of the distribution. It formalizes the idea that averaging reduces uncertainty and stabilizes around a true value.

To see a full derivation and explanation of the most common form of the WLLN, please watch the following video. It is a concise and self-contained presentation that walks through the setup, the proof, and the interpretation.

Unpacking the Proof

As you saw in the video, the proof is an elegant application of concepts you already know. Let's formalize the statement and the steps.

Statement of the WLLN (with finite variance)

Let $X_1,X_2,\dots$ be a sequence of independent and identically distributed (i.i.d.) random variables with a finite mean $E[X_i]=\mu$ and a finite variance $\text{Var}(X_i)={\sigma }^2<\infty$. Let the sample mean be denoted by ${\bar{X}}_n=\frac{1}{n}{\sum }_{i=1}^n{X}_i$.

Then, for any positive number $ϵ$, the sample mean converges in probability to the true mean $\mu$:

Or, using the notation from our last lesson, ${\bar{X}}_n\stackrel{p}{\to }\mu$.

The Proof, Step-by-Step

The proof rests on three key calculations:

1. Expectation of the Sample Mean: By linearity of expectation, the sample mean is an unbiased estimator of $\mu$.

   $$E[{\bar{X}}_n]=E\left[\frac{1}{n}\sum _{i=1}^n{X}_i\right]=\frac{1}{n}\sum _{i=1}^{n}E[X_i]=\frac{1}{n}(n\mu )=\mu$$

2. Variance of the Sample Mean: This is the crucial step. Because the variables are independent, the variance of the sum is the sum of the variances.

   $$\text{Var}({\bar{X}}_n)=\text{Var}\left(\frac{1}{n}\sum _{i=1}^n{X}_i\right)=\frac{1}{n^2}\text{Var}\left(\sum _{i=1}^n{X}_i\right)=\frac{1}{n^2}\sum _{i=1}^n\text{Var}(X_i)=\frac{1}{n^2}(n{\sigma }^2)=\frac{{\sigma }^2}{n}$$

   This result shows that the variance of the sample mean shrinks as the sample size $n$ increases. This inverse relationship should be very familiar from analyzing statistical errors in experimental physics, where the standard error of the mean decreases with the square root of the number of measurements.

3. Application of Chebyshev's Inequality: In the previous lesson, we discussed Chebyshev's inequality, which bounds the probability of a random variable deviating from its mean. For any random variable $Y$ with mean ${\mu }_Y$ and variance ${\sigma }_Y^2$, and for any $ϵ>0$:

   $$P(|Y-{\mu }_Y|\ge ϵ)\le \frac{{\sigma }_Y^2}{{ϵ}^2}$$

   Applying this to our sample mean ${\bar{X}}_n$, where $Y={\bar{X}}_n$, ${\mu }_Y=\mu$, and ${\sigma }_Y^2={\sigma }^2/n$:

   $$P(|{\bar{X}}_n-\mu |\ge ϵ)\le \frac{\text{Var}({\bar{X}}_n)}{{ϵ}^2}=\frac{{\sigma }^2}{n{ϵ}^2}$$

   As we take the limit $n\to \infty$, the right-hand side goes to zero for any fixed $ϵ>0$, which completes the proof.

   $$\underset{n\to \infty }{\lim }P(|{\bar{X}}_n-\mu |\ge ϵ)\le \underset{n\to \infty }{\lim }\frac{{\sigma }^2}{n{ϵ}^2}=0$$

A More General and Powerful Result

The proof above is straightforward, but it relies on the condition of finite variance (${\sigma }^2<\infty$). This excludes many "heavy-tailed" distributions that appear in various physical and economic models, such as certain Pareto distributions which may have a finite mean but infinite variance.

A more powerful version of the WLLN, often called Khinchin's Weak Law of Large Numbers, relaxes this requirement. It states that the law holds as long as the mean is finite ($E|X_1|<\infty$), even if the variance is infinite.

The proof is more involved and typically uses one of two advanced techniques:

1. Truncation: The random variables are split into a "well-behaved" part (by cutting off the tails) and a "tail" part. Chebyshev's inequality is applied to the well-behaved part, and the probability of the tail part is shown to be negligible.
2. Characteristic Functions: This method uses the fact that convergence in distribution is equivalent to the pointwise convergence of characteristic functions (a result from Lévy's continuity theorem). Since convergence in probability to a constant is equivalent to convergence in distribution, one can prove the WLLN by showing that the characteristic function of ${\bar{X}}_n$ converges to the characteristic function of the constant $\mu$. This approach leverages tools analogous to Fourier analysis.

To see a more formal statement of this general law and the truncation technique, please review the following notes.

This more general result is a testament to the robustness of the law of large numbers. It asserts that the stabilizing effect of averaging is a very fundamental property, not just an artifact of distributions with well-behaved variances.

Conclusion

In this lesson, we have formally defined and proven the Weak Law of Large Numbers, connecting it to the concept of convergence in probability.

Key Takeaways:

• The Weak Law of Large Numbers states that the sample mean ${\bar{X}}_n$ of i.i.d. random variables converges in probability to the true population mean $\mu$.
• It provides the rigorous mathematical foundation for the common practice of using sample averages to estimate unknown parameters.
• A simple proof exists if the variance is finite, relying on the fact that $\text{Var}({\bar{X}}_n)={\sigma }^2/n\to 0$.
• A more general version of the WLLN holds even if the variance is infinite, as long as the mean is finite. This underscores the broad applicability of the principle.

The WLLN is a statement about probabilities for a fixed, large $n$. It tells us that the probability of a large error is small, but it doesn't preclude the possibility that large errors, however improbable, might occur infinitely often as $n$ increases. To make a stronger claim about the limiting behavior of the entire sequence of sample means, we need a stronger mode of convergence.

Preview of the Next Lesson:

We will move on to the Strong Law of Large Numbers (SLLN). This theorem strengthens the WLLN by using almost sure convergence, stating that for almost every sequence of outcomes, the sample mean will eventually converge to and stay at the true mean. This will complete our study of the two fundamental Laws of Large Numbers.