PCA: Rotating for Maximal Variance

1. The Big Picture: A Change of Perspective

Imagine a scatter plot of data where the variables are correlated. For instance, in a dataset of people, height and weight are often correlated, forming a data cloud that is tilted and elongated. The standard axes (height and weight) might not be the most "natural" way to describe the variation in the data.

PCA provides a new set of axes to describe the data. These new axes, the principal components, are chosen to align with the directions of maximum variance. The first principal component (PC1) points in the direction of the greatest spread, and the second (PC2) points in the direction of the next greatest spread, while being orthogonal to PC1.

This process is geometrically equivalent to rotating the entire coordinate system. The goal is to orient the axes so that they capture the data's structure more efficiently. In this new system, the transformed features are uncorrelated.

The following blog post provides an excellent visual and conceptual introduction to this idea.

2. How PCA Finds the "Best" Rotation

We've established that PCA performs a rotation. But how does it determine the best rotation? This brings us back to our goal from the previous lesson: maximizing variance. The "best" new axes are those that capture the most variance.

The StatQuest video below offers a step-by-step geometric construction of this process. It intuitively explains how finding the line that maximizes the spread of projected data points leads us to the first principal component.

As the video demonstrates, PCA finds:

1. PC1: The line (direction) that maximizes the variance of the projected data.
2. PC2: The line orthogonal to PC1 that maximizes the remaining variance.
3. ...and so on for all dimensions.

The final transformation is a rigid rotation of the original axes to align with these newly found principal components.

3. The Mathematics of Rotation: Eigendecomposition

Now we bridge the gap between the geometric rotation we just saw and the algebraic solution from our last lesson. We know the principal components are the eigenvectors of the covariance matrix $\mathbf{S}$. How does this relate to rotation?

Let the original, centered data be represented by the matrix $\mathbf{X}$ (where rows are observations and columns are features). Let $\mathbf{P}$ be the matrix whose columns are the eigenvectors of the covariance matrix $\mathbf{S}=\frac{1}{n-1}{\mathbf{X}}^T\mathbf{X}$. These eigenvectors form an orthonormal basis.

The act of rotating the data into the new coordinate system is accomplished by projecting the original data onto the new axes. This is done with a matrix multiplication:

Here, $\mathbf{Z}$ is the matrix of our transformed data, also known as the principal component scores. Each column of $\mathbf{Z}$ gives the coordinates of our data points along the corresponding principal component axis.

The matrix $\mathbf{P}$ is a change-of-basis matrix. Because its columns are orthonormal, it represents a rigid rotation (and possibly a reflection) of the space.

The Punchline: Why this rotation works

The magic of this transformation is that it decorrelates the data. We can prove this by calculating the covariance matrix of our new data, $\mathbf{Z}$.
The new covariance matrix, ${\mathbf{C}}_{\mathbf{Z}}$, is:

Recognizing that $\mathbf{S}=\frac{1}{n-1}{\mathbf{X}}^T\mathbf{X}$, we have:

From the previous lesson, we know that any symmetric matrix $\mathbf{S}$ can be diagonalized by its matrix of eigenvectors $\mathbf{P}$ according to the spectral theorem:

where $\mathbf{D}$ is the diagonal matrix of eigenvalues. Substituting this into our equation for ${\mathbf{C}}_{\mathbf{Z}}$:

Since the eigenvectors are orthonormal, ${\mathbf{P}}^T\mathbf{P}=\mathbf{I}$ (the identity matrix). This gives us:

This is a profound result. The covariance matrix of the transformed data, $\mathbf{Z}$, is a diagonal matrix $\mathbf{D}$.

• The off-diagonal elements are all zero, which means the new variables (the principal components) are uncorrelated.
• The diagonal elements are the eigenvalues of the original covariance matrix $\mathbf{S}$. This confirms that the variance of the data along each principal component axis is equal to the corresponding eigenvalue.

This derivation formally proves that finding the eigenvectors of the covariance matrix gives you the exact rotation needed to align the axes with maximal variance and decorrelate your data.

To see this derivation again from a slightly different angle, review the following resource.

Conclusion

In this lesson, we connected the algebraic definition of PCA to its geometric meaning. We saw that PCA is not just an abstract algorithm but a concrete transformation of our data's coordinate system.

Key Takeaways:

• Geometrically, PCA performs a rotation of the coordinate system.
• The new axes are the principal components, which are aligned with the directions of maximum variance in the data.
• This optimal rotation is defined by the eigenvectors of the data's covariance matrix. The matrix of eigenvectors acts as a change-of-basis matrix.
• The result of this rotation is a new representation of the data where the features are uncorrelated. The covariance matrix of the transformed data is a diagonal matrix of the original eigenvalues.
• The variance explained by each principal component is equal to its corresponding eigenvalue.

Preview of the Next Lesson

We have now derived PCA algebraically and interpreted it geometrically. In the next lesson, we will formalize the relationship between the PCA solution and the eigendecomposition of the covariance matrix. We will explore how this decomposition provides a complete "recipe" for PCA and serves as the foundation for its practical implementation. We will also introduce the Singular Value Decomposition (SVD) as an alternative, numerically stable method for performing PCA.