The Central Limit Theorem for i.i.d. Random Variables

Stating the Central Limit Theorem

The most common version of the CLT, often called the Lindeberg-Lévy CLT, applies to independent and identically distributed (i.i.d.) random variables.

Theorem (Central Limit Theorem):
Let $X_1,X_2,\dots$ be a sequence of i.i.d. random variables with a finite mean $E[X_i]=\mu$ and a finite, positive variance $Var(X_i)={\sigma }^2$.

Let ${\bar{X}}_n=\frac{1}{n}{\sum }_{i=1}^n{X}_i$ be the sample mean. Then the standardized sample mean,
$Z_n=\frac{{\bar{X}}_n-\mu }{\sigma /\sqrt{n}}=\frac{{\sum }_{i=1}^n{X}_i-n\mu }{\sqrt{n}\sigma }$
converges in distribution to a standard normal random variable $Z\sim N(0,1)$ as $n\to \infty$.

This is written as:
$Z_n\stackrel{d}{\to }N(0,1)$
Formally, this means the cumulative distribution function (CDF) of $Z_n$ converges to the CDF of the standard normal distribution, $\Phi (z)$:
${\lim }_{n\to \infty }P(Z_n\le z)=\Phi (z)\text{for all }z\in \mathbb{R}$

Key Insights from the Theorem

1. Universality: The striking feature is that the limiting distribution is always normal. The original distribution of the $X_i$ can be discrete (like Bernoulli or Poisson) or continuous (like Uniform or Exponential), symmetric or skewed. As long as the variance is finite, their sum, when appropriately scaled, will be approximately normal.

2. Rate of Convergence: The factor $\sqrt{n}$ is crucial. The Laws of Large Numbers tell us that ${\bar{X}}_n-\mu$ goes to zero. The CLT gives a more precise statement: the typical magnitude of the error ${\bar{X}}_n-\mu$ is on the order of $1/\sqrt{n}$. Multiplying by $\sqrt{n}$ "zooms in" on the fluctuations at the correct scale to reveal a stable, non-degenerate distribution.

3. Convergence in Distribution: The CLT guarantees convergence of the CDFs, which is a weaker mode of convergence than the almost sure convergence of the SLLN. The random variable $Z_n$ itself does not converge to a single value, but its probabilistic character becomes indistinguishable from a standard normal.

The following video provides an excellent introduction, distinguishing the CLT from the LLN and highlighting its significance.

The Proof of the Central Limit Theorem

The standard proof of the Lindeberg-Lévy CLT is a beautiful application of transform methods. The strategy hinges on a fundamental result called Lévy's Continuity Theorem, which states that convergence in distribution of a sequence of random variables is equivalent to the pointwise convergence of their characteristic functions (or moment generating functions, if they exist in a neighborhood of zero).

The proof proceeds as follows:

1. Goal: Show that the MGF of the standardized mean, $M_{Z_n}(t)$, converges to $e^{t^2/2}$, which is the MGF of a $N(0,1)$ random variable.
2. Simplify: Without loss of generality, assume the original variables are already standardized, i.e., $E[X_i]=0$ and $Var(X_i)=1$. If not, we can just define $Y_i=(X_i-\mu )/\sigma$ and proceed. In this case, $Z_n=\frac{1}{\sqrt{n}}{\sum }_{i=1}^n{X}_i$.
3. Use MGF Properties: The MGF of a sum of independent variables is the product of their MGFs. The MGF of a scaled variable $aX$ is $M_X(at)$. Combining these gives $M_{Z_n}(t)={\left[M_X(t/\sqrt{n})\right]}^n$.
4. Taylor Expansion: Expand $M_X(u)$ around $u=0$: $M_X(u)=M_X(0)+u{M}_X^{\prime }(0)+\frac{u^2}{2}{M}_X^{\prime \prime }(0)+O(u^3)$.
5. Use Moments: By definition, $M_X(0)=1$, $M_X^{\prime }(0)=E[X]=0$, and $M_X^{\prime \prime }(0)=E[X^2]=Var(X)+(E[X]{)}^2=1$. So, $M_X(u)=1+\frac{u^2}{2}+O(u^3)$.
6. Substitute and Take the Limit: Substitute $u=t/\sqrt{n}$ into the expansion and take the $n$-th power. This results in an expression of the form ${\left(1+\frac{t^2/2}{n}+\text{smaller terms}\right)}^n$. As $n\to \infty$, this converges to $e^{t^2/2}$ by the definition of the exponential function, ${\lim }_{n\to \infty }(1+a/n{)}^n=e^a$.

The following two videos walk through this proof with excellent clarity. They use slightly different techniques for the final limit (L'Hôpital's Rule vs. Taylor series), offering two robust perspectives on the same core logic.

For completeness, it's worth noting that the most general proofs use characteristic functions (CFs), which are related to the Fourier transform and exist for all probability distributions, unlike MGFs. The logic is identical. The document below provides a concise proof using CFs.

Applying the Central Limit Theorem

The primary application of the CLT is to approximate the distribution of a sample mean or a sum. For a large $n$, we can treat ${\bar{X}}_n$ as if it were drawn from a normal distribution:
${\bar{X}}_n\approx N(\mu ,{\sigma }^2/n)$
Similarly, the sum $S_n=\sum X_i$ is approximately normal:
$S_n\approx N(n\mu ,n{\sigma }^2)$

Example: Normal Approximation to the Binomial

A classic application is approximating binomial probabilities. A random variable $X\sim Bin(n,p)$ represents the number of successes in $n$ trials and can be viewed as the sum of $n$ i.i.d. $Bernoulli(p)$ random variables.
For a Bernoulli variable, $\mu =p$ and ${\sigma }^2=p(1-p)$.
Therefore, for the sum $X$, we have $E[X]=np$ and $Var(X)=np(1-p)$.

By the CLT, for large $n$, $X$ is approximately normally distributed with this mean and variance. This is invaluable because calculating exact binomial probabilities, $P(X=k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k}$, can be computationally intensive for large $n$.

Continuity Correction: When approximating a discrete distribution (like the Binomial) with a continuous one (like the Normal), accuracy is improved by using a continuity correction. The probability mass at a single integer value $k$ in the discrete distribution is represented by the area over the interval $[k-0.5,k+0.5]$ in the continuous approximation.

• To approximate $P(X\le k)$, we compute $P(\text{Normal}\le k+0.5)$.
• To approximate $P(X\ge k)$, we compute $P(\text{Normal}\ge k-0.5)$.
• To approximate $P(X=k)$, we compute $P(k-0.5\le \text{Normal}\le k+0.5)$.

The following video demonstrates this application, including the continuity correction.

Conclusion

The Central Limit Theorem is a cornerstone of statistical theory, providing the justification for why the normal distribution appears so frequently in applied problems.

Key Takeaways:

• The Central Limit Theorem states that the distribution of the standardized sample mean of i.i.d. variables with finite variance converges to a standard normal distribution.
• This result is universal, holding regardless of the shape of the parent distribution.
• The correct scaling factor is $1/\sqrt{n}$, which describes the rate at which the sample mean approaches the true mean.
• The standard proof uses moment generating functions or characteristic functions, relying on Taylor expansions and the properties of limits.
• A key application is the normal approximation of other distributions, like the Binomial, which is made more accurate by using a continuity correction.

Preview of the Next Lesson:

The CLT provides the asymptotic distribution for the sample mean ${\bar{X}}_n$. But what if we are interested in a parameter that is a function of the mean, such as $1/{\bar{X}}_n$ or $\log ({\bar{X}}_n)$? How do we find the distribution of these transformed estimators? In our next lesson, we will introduce the delta method, a powerful tool that combines the CLT with a Taylor series approximation to derive the asymptotic distributions of functions of random variables.