Proving the Central Limit Theorem with Characteristic Functions

1. The Power of Characteristic Functions

The characteristic function is the probabilistic cousin of the Fourier transform, a concept you are likely familiar with from your background in mechanical engineering and signal processing.

For a random variable $X$, its characteristic function ${\varphi }_X(t)$ is a complex-valued function defined as:
${\varphi }_X(t)=E[e^{itX}]={\int }_{-\infty }^{\infty }{e}^{itx}{f}_X(x)dx$
where $i=\sqrt{-1}$ and $t$ is a real number. Unlike the moment-generating function, the characteristic function exists for any random variable because $|e^{itX}|=1$, ensuring the expectation is always finite.

For our proof of the CLT, we need two critical properties:

1. Uniqueness: A distribution is uniquely defined by its characteristic function (Lévy's Inversion Theorem). If two random variables have the same characteristic function, they have the same distribution. The characteristic function for a standard normal random variable $Z\sim N(0,1)$ is ${\varphi }_Z(t)=e^{-t^2/2}$.
2. Sums of Independent Variables: If $X_1,X_2,\dots ,X_n$ are independent random variables, the characteristic function of their sum $S_n=X_1+\cdots +X_n$ is the product of their individual characteristic functions:
   ${\varphi }_{S_n}(t)=E[e^{it(X_1+\cdots +X_n)}]=E[e^{it{X}_1}\cdots e^{it{X}_n}]={\prod }_{i=1}^{n}E[e^{it{X}_i}]={\prod }_{i=1}^n{\varphi }_{X_i}(t)$

To build a solid foundation, please read the following sections from the provided lecture notes.

2. The Bridge: Lévy's Continuity Theorem

The central pillar connecting characteristic functions to the CLT is Lévy's Continuity Theorem. It provides the rigorous justification for our entire approach.

Theorem (Lévy's Continuity): A sequence of random variables $X_n$ converges in distribution to a random variable $X$ if and only if the characteristic function ${\varphi }_{X_n}(t)$ converges pointwise to the characteristic function ${\varphi }_X(t)$ for every $t\in \mathbb{R}$.

In simple terms, to prove that our standardized sum converges to a normal distribution, we "only" need to show that its characteristic function converges to $e^{-t^2/2}$. This transforms a problem about probability distributions into a problem of analyzing the limit of a sequence of functions.

For a formal statement of this powerful theorem, refer to the same notes.

3. The Engine: Taylor Expansion of a Characteristic Function

Our goal is to analyze the limit of $[{\varphi }_Y(t/\sqrt{n}){]}^n$, where $Y$ is a standardized random variable. Since the argument $t/\sqrt{n}$ becomes very small as $n\to \infty$, a Taylor series expansion around $t=0$ is the natural tool.

If a random variable $Y$ has finite moments, we can relate them to the derivatives of its characteristic function at zero. Specifically, if $E[Y]=0$ and $\text{Var}(Y)=E[Y^2]={\sigma }^2$, then the Taylor expansion of ${\varphi }_Y(t)$ around $t=0$ is:
${\varphi }_Y(t)={\varphi }_Y(0)+{\varphi }_Y^{\prime }(0)t+\frac{{\varphi }_Y^{\prime \prime }(0)}{2!}{t}^2+o(t^2)$
Using the property that $E[Y^k]=i^{-k}{\varphi }_Y^{(k)}(0)$, we get:

• ${\varphi }_Y(0)=E[e^0]=1$
• ${\varphi }_Y^{\prime }(0)=iE[Y]=0$
• ${\varphi }_Y^{\prime \prime }(0)=i^{2}E[Y^2]=-{\sigma }^2$

Substituting these in gives the crucial approximation:
${\varphi }_Y(t)=1-\frac{{\sigma }^2{t}^2}{2}+o(t^2)$
where $o(t^2)$ is a term that goes to zero faster than $t^2$.

The following video provides a detailed, step-by-step derivation of this expansion, which is essential for understanding the mechanics of the CLT proof.

4. Assembling the Proof of the Central Limit Theorem

We now have all the necessary components. Let's assemble them to prove the CLT.

Statement: Let $X_1,X_2,\dots$ be i.i.d. random variables with mean $E[X_i]=\mu$ and variance $\text{Var}(X_i)={\sigma }^2<\infty$. Let $S_n={\sum }_{i=1}^n{X}_i$. Then the standardized sum
$Z_n=\frac{S_n-n\mu }{\sigma \sqrt{n}}$
converges in distribution to a standard normal random variable $Z\sim N(0,1)$.

Proof Outline:

1. Standardize: Define $Y_i=\frac{X_i-\mu }{\sigma }$. These are i.i.d. with $E[Y_i]=0$ and $\text{Var}(Y_i)=1$. Our standardized sum becomes $Z_n=\frac{1}{\sqrt{n}}{\sum }_{i=1}^n{Y}_i$.

2. Find the Characteristic Function of $Z_n$: Using the properties we learned:
   ${\varphi }_{Z_n}(t)={\varphi }_{\sum Y_i}(t/\sqrt{n})={\left[{\varphi }_{Y_1}(t/\sqrt{n})\right]}^n$

3. Apply the Taylor Expansion: Since $E[Y_1]=0$ and $\text{Var}(Y_1)=1$, we use the expansion from the previous section with ${\sigma }^2=1$:
   ${\varphi }_{Y_1}(u)=1-\frac{u^2}{2}+o(u^2)$
   Substitute $u=t/\sqrt{n}$:
   ${\varphi }_{Y_1}(t/\sqrt{n})=1-\frac{(t/\sqrt{n}{)}^2}{2}+o((t/\sqrt{n}{)}^2)=1-\frac{t^2}{2n}+o(1/n)$

4. Take the Limit: We now need to find the limit of ${\varphi }_{Z_n}(t)$ as $n\to \infty$:
   ${\lim }_{n\to \infty }{\varphi }_{Z_n}(t)={\lim }_{n\to \infty }{\left[1-\frac{t^2}{2n}+o(1/n)\right]}^n$
   Using the well-known limit identity ${\lim }_{n\to \infty }(1+x/n{)}^n=e^x$, we find:
   ${\lim }_{n\to \infty }{\varphi }_{Z_n}(t)=e^{-t^2/2}$

5. Conclude with Lévy's Theorem: The limit $e^{-t^2/2}$ is the characteristic function of a standard normal distribution $Z\sim N(0,1)$. By Lévy's Continuity Theorem, since ${\varphi }_{Z_n}(t)\to {\varphi }_Z(t)$, we conclude that $Z_n\stackrel{d}{\to }Z$. This completes the proof.

The following resource provides a very clean write-up of this exact sequence of steps.

5. From Theory to Practice: A Visual Demonstration

To connect this abstract proof to a practical result, let's visualize the CLT. The theorem's magic is that it works for any initial distribution. Let's pick a very non-Gaussian one, like the exponential distribution. If we take many samples from it, calculate their mean, and repeat this process thousands of times, the distribution of these means will be Gaussian.

Here is a Python script you can run to reproduce this yourself. This directly addresses your preference for hands-on coding exercises.

import numpy as np
import matplotlib.pyplot as plt




# 1. Define the initial (non-normal) distribution
# Let's use an exponential distribution.
# 'scale' is 1/lambda, 'size' is the number of points.
original_pop = np.random.exponential(scale=2.0, size=100000)




# 2. Set up the simulation parameters
sample_size = 50  # How many data points in each sample
num_samples = 10000 # How many times we draw a sample




# 3. Run the simulation
sample_means = []
for _ in range(num_samples):



    # Draw a random sample from the original population
    sample = np.random.choice(original_pop, size=sample_size)



    # Calculate the mean of that sample and store it
    sample_means.append(np.mean(sample))




# 4. Visualize the results
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(8, 8))




# Plot the original population's distribution
ax1.hist(original_pop, bins=100, density=True, color='skyblue')
ax1.set_title("Original Distribution (Exponential)")
ax1.set_xlim(0, 20)




# Plot the distribution of sample means
ax2.hist(sample_means, bins=50, density=True, color='salmon')
ax2.set_title(f"Distribution of {num_samples} Sample Means (Sample Size = {sample_size})")
ax2.set_xlabel("Value")

plt.tight_layout()
plt.show()




# Theoretical check
pop_mean = np.mean(original_pop)
pop_std = np.std(original_pop)
print(f"Population Mean: {pop_mean:.2f}")
print(f"Mean of Sample Means: {np.mean(sample_means):.2f}")
print("-" * 20)



# According to CLT, the std of sample means should be sigma/sqrt(n)
print(f"Population Std Dev: {pop_std:.2f}")
print(f"Std Dev of Sample Means: {np.std(sample_means):.2f}")
print(f"Theoretical Std Dev of Sample Means (sigma/sqrt(n)): {pop_std/np.sqrt(sample_size):.2f}")

I encourage you to experiment with this code. Try changing the sample_size (e.g., to 2, 10, or 100) and see how the bottom histogram becomes more or less "bell-shaped."

Conclusion

In this lesson, we have constructed a complete and rigorous outline of the proof for the Central Limit Theorem.

Key Takeaways:

• The CLT states that the standardized sum of i.i.d. random variables converges in distribution to a standard normal distribution.
• Characteristic functions are the key analytical tool, turning problems of distributional convergence into problems of functional convergence.
• Lévy's Continuity Theorem is the bridge that formalizes this connection.
• The core of the proof relies on a Taylor expansion of the characteristic function around zero, which leads to the famous limit: ${\lim }_{n\to \infty }(1-t^2/(2n){)}^n=e^{-t^2/2}$.

Preview of the Next Lesson:

We have just used a powerful technique—characteristic functions—to prove a major theorem. In our next lesson, "Gaussian Combinations and Limiting Distributions," we will continue to leverage this tool. Our first objective will be to prove that a linear combination of independent Gaussian random variables is itself a Gaussian. This is a property that can be demonstrated with remarkable elegance using characteristic functions, providing another excellent application of the theory you've learned today.