The Strong Law of Large Numbers

The "Strong" in the Strong Law: Almost Sure Convergence

The difference between the WLLN and SLLN lies entirely in the mode of convergence. The WLLN uses convergence in probability, while the SLLN uses almost sure convergence. Let's make this distinction clear.

Let ${\bar{X}}_n$ be the sample mean and $\mu$ be the true mean.

• Weak Law (Convergence in Probability): For any $ϵ>0$, ${\lim }_{n\to \infty }P(|{\bar{X}}_n-\mu |>ϵ)=0$.
  This looks at each $n$ individually and says that for a large enough $n$, the chance of being far from $\mu$ is small.

• Strong Law (Almost Sure Convergence): $P({\lim }_{n\to \infty }{\bar{X}}_n=\mu )=1$.
  This is a statement about the entire sequence of random variables $\{{\bar{X}}_1,{\bar{X}}_2,\dots \}$. It says that the set of all possible outcomes (or sample paths) for which the sequence of sample means does not converge to $\mu$ has a total probability of zero.

To build your intuition, the following short video provides an excellent high-level introduction to the SLLN and what "convergence with probability 1" means.

Almost sure convergence implies convergence in probability, which is why the SLLN is "stronger" than the WLLN. An example can make the distinction concrete.

Imagine a sequence of random variables $Y_n$ that mostly takes the value 0, but occasionally spikes.

• If the spikes become less and less probable as $n\to \infty$, the sequence might converge in probability to 0.
• However, if these spikes are guaranteed to happen infinitely often (even if they become very rare), the sequence of values will never settle down to 0 for any given sample path. Thus, it would not converge almost surely.

The following video clip illustrates exactly such a case.

The SLLN guarantees that for the sequence of sample means ${\bar{X}}_n$, this kind of pathological spiking behavior does not happen. The average truly settles down.

Statement and Proof of the Strong Law

Now, let's look at the formal statement of the SLLN, known as Kolmogorov's Strong Law.

Theorem (Kolmogorov's Strong Law of Large Numbers):
Let $X_1,X_2,\dots$ be a sequence of independent and identically distributed (i.i.d.) random variables. The sample mean ${\bar{X}}_n=\frac{1}{n}{\sum }_{i=1}^n{X}_i$ converges almost surely to a constant $\mu$ if and only if $E|X_1|<\infty$. If this condition holds, then $\mu =E[X_1]$.

Proving this theorem in its full generality is quite involved. However, we can prove it under a stronger assumption—finiteness of the fourth moment—using tools we've already discussed. This illustrates the core logic connecting moments to almost sure convergence.

Proof of SLLN with a Finite Fourth Moment

The key tool that links statements about sums of probabilities to statements about almost sure convergence is the Borel-Cantelli Lemma. The first part states that if the sum of probabilities of a sequence of events $\{A_n\}$ is finite, ${\sum }_{n=1}^{\infty }P(A_n)<\infty$, then the probability that infinitely many of these events occur is 0.

Our strategy is to define the "bad" event $A_n$ as the sample mean being far from the true mean, $A_n=\{|{\bar{X}}_n-\mu |>ϵ\}$, and show that $\sum P(A_n)<\infty$.

The following video provides a detailed, step-by-step proof of the SLLN under the assumption that $E[X^4]<\infty$. Given your background, you should be able to follow the derivation.

The General Proof Strategy

The proof for the general case, which only assumes a finite first moment ($E|X|<\infty$), is a masterpiece of analysis. It combines several clever ideas. The renowned mathematician Terence Tao wrote an excellent blog post outlining this strategy. His explanation is aimed at a sophisticated audience and perfectly suited for your background.

This strategy beautifully demonstrates how different analytical tools can be combined to prove a powerful result with minimal assumptions.

Applications of the Strong Law

The SLLN is not just a theoretical curiosity; it's the engine behind many practical statistical techniques.

1. Monte Carlo Integration

One of the most direct applications is Monte Carlo integration. Suppose you want to compute a definite integral $I={\int }_a^{b}g(x)dx$. We can rewrite this as:
$I=(b-a){\int }_a^{b}g(x)\frac{1}{b-a}dx$
The term $\frac{1}{b-a}$ is the probability density function of a Uniform($a,b$) random variable. Therefore, the integral is simply $(b-a)E[g(U)]$, where $U\sim \text{Uniform}(a,b)$.

By the SLLN, if we draw $n$ i.i.d. samples $u_1,u_2,\dots ,u_n$ from Uniform($a,b$), the sample mean of $g(u_i)$ will converge almost surely to $E[g(U)]$.
$\frac{1}{n}{\sum }_{i=1}^{n}g(u_i)\stackrel{a.s.}{\to }E[g(U)]$
This gives us a practical way to approximate the integral:
$I\approx (b-a)\frac{1}{n}{\sum }_{i=1}^{n}g(u_i)$
The SLLN guarantees that this approximation converges to the true value as $n\to \infty$. This method is especially powerful for high-dimensional integrals where traditional numerical quadrature fails.

2. Consistency of Estimators

In statistics, an estimator is called strongly consistent if it converges almost surely to the true parameter it is meant to estimate. The SLLN is the primary tool for proving the consistency of estimators based on sample moments.

A prime example is the sample variance. Let $X_1,\dots ,X_n$ be i.i.d. with mean $\mu$ and variance ${\sigma }^2$. The sample variance is $s_n^2=\frac{1}{n-1}{\sum }_{i=1}^n(X_i-{\bar{X}}_n{)}^2$. We can show it's a strongly consistent estimator for ${\sigma }^2$.

The proof, which you can find in Example 7.14 of the DasGupta PDF (resource LINK), follows this logic:

1. Rewrite $\frac{1}{n}\sum (X_i-{\bar{X}}_n{)}^2$ as $\left(\frac{1}{n}\sum X_i^2\right)-({\bar{X}}_n{)}^2$.
2. By the SLLN, ${\bar{X}}_n\stackrel{a.s.}{\to }E[X]=\mu$.
3. By the SLLN again (applied to the sequence of squared variables $X_i^2$), $\frac{1}{n}\sum X_i^2\stackrel{a.s.}{\to }E[X^2]={\sigma }^2+{\mu }^2$.
4. Since convergence is almost sure, we can apply the continuous mapping theorem. The function $f(a,b)=a-b^2$ is continuous.
5. Therefore, $\left(\frac{1}{n}\sum X_i^2\right)-({\bar{X}}_n{)}^2\stackrel{a.s.}{\to }({\sigma }^2+{\mu }^2)-{\mu }^2={\sigma }^2$.
6. Finally, since $\frac{n}{n-1}\to 1$, we have $s_n^2\stackrel{a.s.}{\to }{\sigma }^2$.

Conclusion

This lesson has elevated our understanding from the Weak to the Strong Law of Large Numbers, providing a much more powerful guarantee about the long-run behavior of sample averages.

Key Takeaways:

• The Strong Law of Large Numbers (SLLN) states that the sample mean converges almost surely to the true mean, provided the mean is finite.
• Almost sure convergence—$P({\lim }_{n\to \infty }{\bar{X}}_n=\mu )=1$—is a statement about the convergence of the entire sample path, which is a stronger guarantee than convergence in probability.
• The proof of the SLLN relies on the Borel-Cantelli lemma to connect summable probabilities of deviations to an almost sure event.
• For the general case (finite first moment), the proof involves a combination of truncation, sparsification, and moment methods.
• The SLLN is the theoretical foundation for many critical statistical applications, including Monte Carlo methods and proving the consistency of estimators.

Preview of the Next Lesson:

The Laws of Large Numbers tell us that the sample mean ${\bar{X}}_n$ converges to $\mu$. But they don't tell us about the fluctuations of ${\bar{X}}_n$ around $\mu$ for a large but finite $n$. What is the distribution of the error ${\bar{X}}_n-\mu$? Answering this question leads us to the other pillar of classical probability theory: the Central Limit Theorem (CLT). In the next lesson, we will state and apply the CLT, which describes the approximately Normal distribution of the standardized sample mean.