Multivariate Gaussian Density from Affine Transformations

Hello! In our last lesson, we explored the many roles of the Gaussian distribution in deep learning, from justifying the mean squared error loss to enabling the reparameterization trick in VAEs and driving the noise process in diffusion models. A recurring theme was the "shift and scale" or affine property: any Gaussian can be generated by applying a linear transformation and a shift to a standard normal variable.

Today, we will formalize this idea. Your objective is to derive the multivariate Gaussian density via an affine transformation of standard normal variables. We will start with the simplest possible building block—a vector of independent, standard normal variables—and use the change of variables formula from calculus to see how its density function transforms under an affine map. This will demystify the somewhat intimidating formula for the multivariate Gaussian PDF, showing you precisely where each component comes from.

Let's begin with a high-level roadmap of this derivation. The image below provides a fantastic, hand-drawn sketch of the entire process, which we will follow step-by-step.

This image sketches the derivation of the multivariate normal distribution. It starts with a standard univariate normal, extends it to a multivariate product of independent normals, and then applies an affine transformation (shift and scale) to arrive at the general multivariate normal density.

1. The Geometric Intuition: A Constructive Approach

Before we dive into the calculus, let's build a strong geometric intuition for the transformation. If we start with samples from a simple distribution, what sequence of geometric operations would produce the characteristic ellipsoidal shape of a general multivariate Gaussian?

The following video provides an excellent constructive visualization. It demonstrates how to generate samples from an arbitrary 2D Gaussian by starting with independent, standard normal samples (a spherical cloud of points) and then sequentially applying scaling, rotation, and shifting.

Multivariate Gaussian distributions

This video from Alexander Ihler's course provides a clear, step-by-step visualization of how an affine transformation shapes a distribution of points.

Please watch the section on sampling from a Gaussian. Observe how the initial, spherically symmetric data is first scaled, then rotated, and finally shifted. This sequence of operations—scaling, rotation, and translation—is precisely what an affine transformation x = Az + \mu accomplishes. The matrix A handles the scaling and rotation, while the vector \mu handles the shift.

This constructive process shows that any multivariate Gaussian can be seen as a "distorted" version of a standard, spherical Gaussian. Our goal now is to derive the mathematical formula that describes the probability density of this distorted distribution.

2. The Mathematical Derivation

To derive the probability density function (PDF) of the transformed variable $x$ , we need three ingredients:

The PDF of the original standard normal variable $z$ .
The affine transformation relating $x$ and $z$ .
The change of variables formula, which tells us how a PDF changes under a transformation.

The following resource walks through this derivation clearly. It methodically introduces each component before putting them together to arrive at the final result.

Derivations of the Univariate and Multivariate Normal Density

This document by Noah Golmant provides a very clear, step-by-step derivation of the multivariate normal PDF. It is more pedagogical than a compact textbook proof.

First, read the section The Basic Case (Section 2.1) to see the simple joint PDF for our starting point: a vector Z of independent, standard normal variables. Next, read about the tool we will use in PDF of a Transformed Random Vector (Section 2.3). This section introduces the change of variables formula and the crucial role of the Jacobian determinant. Finally, read The Multivariate Normal PDF (Section 2.4), which puts all the pieces together. Pay close attention to how the simple PDF of Z is transformed into the general PDF of X.

Let's break down the key steps from that reading, as this is the core of our lesson.

Step 1: The Base Distribution

We start with a $d$ -dimensional random vector $z = [z_{1}, \dots, z_{d}]^{T}$ , where each component $z_{i}$ is an independent standard normal variable, $z_{i} \sim N (0, 1)$ . Because they are independent, the joint probability density is the product of their individual densities:

p (z) = i = 1 \prod d \frac{1}{2 π} exp (- \frac{1}{2} z_{i}^{2}) = \frac{1}{( 2 π ) ^{d /2}} exp (- \frac{1}{2} i = 1 \sum d z_{i}^{2}) = \frac{1}{( 2 π ) ^{d /2}} exp (- \frac{1}{2} z^{T} z)

This describes a "spherical" probability cloud centered at the origin.

Step 2: The Affine Transformation

We now create a new random variable $x$ by applying a general affine transformation to $z$ :

x = A z + μ

where $A$ is an invertible $d \times d$ matrix and $μ$ is a $d$ -dimensional vector. As we saw in the video, $A$ applies rotation and scaling, while $μ$ applies a shift.

Step 3: Applying the Change of Variables Formula

To find the PDF of $x$ , $p (x)$ , we use the change of variables formula. First, we express $z$ in terms of $x$ : $z = A^{- 1} (x - μ)$ . The formula is:

p (x) = p (z (x)) det (\frac{\partial z}{\partial x})

The term $det (\frac{\partial z}{\partial x})$ is the absolute value of the determinant of the Jacobian matrix of the transformation from $x$ to $z$ . Since $z = A^{- 1} x - A^{- 1} μ$ , the Jacobian is simply $\frac{\partial z}{\partial x} = A^{- 1}$ .

Substituting the components into the formula, we get:

p (x) = \frac{1}{( 2 π ) ^{d /2}} exp (- \frac{1}{2} [A^{- 1} (x - μ)]^{T} [A^{- 1} (x - μ)]) \cdot ∣ det (A^{- 1}) ∣

Step 4: Re-parameterizing in terms of the Covariance Matrix

The expression above is correct, but it's parameterized by $A$ and $μ$ . It's more conventional to parameterize it by the mean and covariance of $x$ .
The mean of $x$ is $E [x] = E [A z + μ] = A E [z] + μ = A \cdot 0 + μ = μ$ .
The covariance of $x$ is:

Σ = Cov (x) = E [(x - μ) (x - μ)^{T}] = E [(A z) (A z)^{T}] = E [A z z^{T} A^{T}] = A E [z z^{T}] A^{T}

Since the components of $z$ are independent with unit variance, $E [z z^{T}] = I$ (the identity matrix). Thus, we arrive at the crucial relationship:

Σ = A A^{T}

Now we can rewrite the PDF in terms of $μ$ and $Σ$ :

The exponent term: $- \frac{1}{2} (x - μ)^{T} (A^{- 1})^{T} A^{- 1} (x - μ) = - \frac{1}{2} (x - μ)^{T} (A A^{T})^{- 1} (x - μ) = - \frac{1}{2} (x - μ)^{T} Σ^{- 1} (x - μ)$ .
The normalization constant: $∣ det (A^{- 1}) ∣ = \frac{1}{∣ d e t ( A ) ∣}$ . Since $det (Σ) = det (A A^{T}) = det (A) det (A^{T}) = (det (A))^{2}$ , we have $∣ det (A) ∣ = det (Σ) = ∣Σ ∣^{1/2}$ . Therefore, the normalization term is $∣Σ ∣^{- 1/2}$ .

Putting it all together, we arrive at the familiar PDF for the multivariate Gaussian distribution:

p (x; μ, Σ) = \frac{1}{( 2 π ) ^{d /2} ∣Σ ∣ ^{1/2}} exp (- \frac{1}{2} (x - μ)^{T} Σ^{- 1} (x - μ))

This derivation confirms that the multivariate Gaussian is not just an arbitrary formula, but the natural result of applying a linear transformation to independent normal variables.

For an alternative, more condensed presentation of the same derivation, you can refer to the bonus slides of the UBC lecture notes.

Advanced Machine Learning - Multivariate Gaussian

This is a bonus section from a different resource that presents the same derivation in a more compact format, which you may appreciate.

Review the two slides titled Multivariate Gaussian from Univariate Gaussians. This condenses the entire derivation we just walked through into a few lines, reinforcing the core logic.

Conclusion

In this lesson, we have demystified the multivariate Gaussian distribution by deriving its probability density function from first principles. We started with the simple, intuitive case of independent standard normal variables and showed how an affine transformation warps their distribution, introducing a mean, and more importantly, a covariance structure.

Key Takeaways:

The multivariate Gaussian distribution $N (μ, Σ)$ can be generated by applying an affine transformation $x = A z + μ$ to a standard normal vector $z \sim N (0, I)$ .
The covariance matrix $Σ$ is directly related to the linear part of the transformation via $Σ = A A^{T}$ . It captures the scaling and rotation that distorts the initial spherical distribution into an ellipsoid.
The change of variables formula from calculus provides the mathematical machinery to derive the PDF of the transformed variable, connecting the Jacobian determinant of the transformation to the $∣Σ ∣^{- 1/2}$ normalization constant.

We now have a robust understanding of not just what the multivariate Gaussian is, but why it has the form it does. In the next lesson, we will leverage this structural understanding to compute the marginal and conditional distributions of a multivariate Gaussian. These operations are fundamental building blocks for many generative models, where we often need to reason about parts of a variable vector while holding others fixed or integrating them out.

Recommended resources