Moments of the Gaussian Distribution

Hello! Welcome to your next lesson on the mathematical foundations of AI.

In our last session, we formally derived the mean and variance for two fundamental discrete distributions, the Bernoulli and Binomial. We saw how algebraic manipulation and understanding the relationships between distributions were key to these proofs.

Today, we shift our focus from the discrete to the continuous realm to study the single most important probability distribution in statistics and machine learning: the Gaussian (or Normal) distribution. Your learning outcome is to derive the mean and variance of the univariate Gaussian distribution from its Probability Density Function (PDF). This will require us to apply the definitions of expectation and variance in a continuous setting, leveraging your calculus skills for integration over infinite domains.

Given your goal of achieving a deep theoretical understanding, we will follow the standard, rigorous approach: first, we will prove the properties for the simpler standard normal distribution, and then we will generalize our findings.

1. The Gaussian Distribution and the Standardization Strategy

The Probability Density Function (PDF) for a general univariate Gaussian random variable $X$ with parameters $\mu$ and ${\sigma }^2$, denoted $X\sim N(\mu ,{\sigma }^2)$, is given by:

$f_X(x)=\frac{1}{\sigma \sqrt{2\pi }}\exp \left\{-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right\}$

As the notation suggests, our goal is to prove that the parameter $\mu$ is the mean and ${\sigma }^2$ is the variance.

Attempting to derive these directly from this general formula involves some heavy algebra. A more elegant and standard approach in mathematics is to first analyze the simplest case: the standard normal distribution, $Z\sim N(0,1)$, where $\mu =0$ and $\sigma =1$. Its PDF is:

$f_Z(z)=\frac{1}{\sqrt{2\pi }}\exp \left\{-\frac{z^2}{2}\right\}$

We will prove that for $Z$, the mean is indeed 0 and the variance is 1. Then, we will show how any normal variable $X$ can be seen as a simple transformation of $Z$, which will allow us to find the mean and variance of $X$ effortlessly.

2. The Normalizing Constant and the Polar Coordinates Trick

Before we can compute the mean or variance, we must first trust that $f_Z(z)$ is a valid PDF. This means its integral over the entire real line must be 1.
${\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}\exp \left\{-\frac{z^2}{2}\right\}dz=1$
This is equivalent to showing that the integral of $\exp (-z^2/2)$ is $\sqrt{2\pi }$. This integral, known as the Gaussian integral, has no elementary antiderivative. Its solution is a classic and beautiful proof that you should know.

The following video from a Harvard Statistics course provides a rigorous derivation of this normalizing constant.

This proof is a fantastic example of a non-obvious mathematical strategy solving a seemingly intractable problem. The appearance of $\pi$ from this integration is what connects the Gaussian distribution to circles, a surprising and deep result.

3. Deriving the Mean and Variance of the Standard Normal Distribution

Now that we have confidence in the PDF, let's derive its mean and variance. The following resource provides a clear walkthrough of both derivations.

Let's summarize the steps from the video, as they are crucial.

Mean of $Z\sim N(0,1)$

By definition, the expected value is:
$E[Z]={\int }_{-\infty }^{\infty }z\cdot f_Z(z)dz={\int }_{-\infty }^{\infty }z\frac{1}{\sqrt{2\pi }}\exp \left\{-\frac{z^2}{2}\right\}dz$
The integrand, let's call it $g(z)=z\exp (-z^2/2)$, is an odd function because $g(-z)=(-z)\exp (-(-z{)}^2/2)=-z\exp (-z^2/2)=-g(z)$. The integral of any odd function over a symmetric interval [-a, a] is always zero, as the negative area cancels the positive area. Therefore:
$E[Z]=0$

Variance of $Z\sim N(0,1)$

We use the formula $\text{Var}(Z)=E[Z^2]-(E[Z]{)}^2$. Since $E[Z]=0$, this simplifies to $\text{Var}(Z)=E[Z^2]$.

$E[Z^2]={\int }_{-\infty }^{\infty }{z}^2\cdot f_Z(z)dz=\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }{z}^2\exp \left\{-\frac{z^2}{2}\right\}dz$

This is solved using integration by parts, $\int udv=uv-\int vdu$. The key is the choice of $u$ and $dv$:

• Let $u=z$, so $du=dz$.
• Let $dv=z\exp (-z^2/2)dz$. We can integrate this to get $v=-\exp (-z^2/2)$.

Substituting into the formula:

Evaluating this over $(-\infty ,\infty )$:

The first term goes to zero because the exponential term decays much faster than $z$ grows. The second term is the Gaussian integral we evaluated earlier, which equals $\sqrt{2\pi }$.

Putting it all together for the variance:
$\text{Var}(Z)=E[Z^2]=\frac{1}{\sqrt{2\pi }}\cdot \sqrt{2\pi }=1$

For a concise, written version of these proofs, the following resource is excellent.

4. Generalizing to an Arbitrary Gaussian Distribution

We have now rigorously proven that for $Z\sim N(0,1)$, $E[Z]=0$ and $\text{Var}(Z)=1$. The final step is to generalize this to $X\sim N(\mu ,{\sigma }^2)$.

Any general normal random variable $X$ can be expressed as a linear transformation of a standard normal variable $Z$:
$X=\mu +\sigma Z$
Using the properties of expectation and variance that we have already established, the proofs become trivial.

As shown in the resource, the derivation is beautifully simple:

Mean of $X\sim N(\mu ,{\sigma }^2)$
Using the linearity of expectation:
$E[X]=E[\mu +\sigma Z]=E[\mu ]+E[\sigma Z]=\mu +\sigma E[Z]=\mu +\sigma (0)=\mu$

Variance of $X\sim N(\mu ,{\sigma }^2)$
Using the properties of variance:
$\text{Var}(X)=\text{Var}(\mu +\sigma Z)=\text{Var}(\sigma Z)={\sigma }^2\text{Var}(Z)={\sigma }^2(1)={\sigma }^2$

And with that, we have proven that the parameters $\mu$ and ${\sigma }^2$ in the Gaussian PDF are indeed its mean and variance. This standardization approach ($Z=(X-\mu )/\sigma$) is a fundamental technique used throughout statistics.

Conclusion

In this lesson, you have performed one of the most classic derivations in probability theory, solidifying your understanding of both the Gaussian distribution and advanced calculus techniques.

Key Takeaways:

• The mean and variance of a general Gaussian distribution $N(\mu ,{\sigma }^2)$ are $\mu$ and ${\sigma }^2$, respectively.
• The rigorous proof follows a standard strategy:
  1. Prove the properties for the standard normal $N(0,1)$ first.
  2. Generalize to $N(\mu ,{\sigma }^2)$ using the linear transformation $X=\mu +\sigma Z$.
• Key mathematical tools used were:
  • The Gaussian integral (solved with polar coordinates) to find the normalizing constant $\frac{1}{\sqrt{2\pi }}$.
  • The property of odd functions integrating to zero to find the mean.
  • Integration by parts to find the variance.

Preview of the Next Lesson:
So far, we have focused on single random variables. However, most real-world AI problems, from image recognition to natural language processing, involve relationships between many variables. In the next lesson, we will begin our exploration of the multivariate world by defining joint, marginal, and conditional distributions for multiple random variables.