PCA Explained Variance for Dimensionality Selection

Introduction

In our previous lesson, we established the theoretical foundation of Principal Component Analysis, deriving it as the eigenvalue decomposition of a dataset's covariance matrix. We concluded that the eigenvectors of this matrix form the new orthogonal basis (the principal components), and the corresponding eigenvalues represent the variance of the data projected onto these components. This gives us a ranked list of components, from most to least important, in terms of captured variance.

This lesson addresses the natural next question: now that we have these ranked components, how many should we actually keep? This is the central challenge in using PCA for dimensionality reduction. Simply choosing an arbitrary number would be unprincipled. Instead, we will explore systematic methods for making this decision. Our goal is to interpret the eigenvalues, which we know represent variance, to guide the selection of an appropriate number of dimensions for a given dataset, balancing information retention with complexity reduction.

From Eigenvalues to Explained Variance

The key to selecting the right number of principal components lies in understanding how much of the original data's total variability each component accounts for. The eigenvalues give us this information directly.

Recall that for a data matrix $X$ with $p$ features, the covariance matrix $C_X$ is a $p\times p$ symmetric matrix. Its eigenvalue decomposition yields $p$ eigenvalues, ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_p$, where by convention we sort them in descending order: ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_p\ge 0$.

The total variance of the dataset is the sum of the variances of all its original features. A convenient property of the covariance matrix is that its trace (the sum of its diagonal elements) is equal to this total variance. Furthermore, a fundamental theorem of linear algebra states that the trace of a matrix is also equal to the sum of its eigenvalues. Therefore:

This demonstrates that PCA is a variance-preserving transformation; the total variance of the data projected onto all principal components is the same as the total variance of the original data.

With this, we can define the explained variance ratio for the $i$-th principal component as the proportion of the total variance that it captures:

This value, often expressed as a percentage, tells us the "importance" of each principal component.

Methods for Selecting the Number of Components

We now have a principled way to quantify the contribution of each component. This leads to two primary methods for deciding how many components, $k$, to retain.

1. The Cumulative Explained Variance Threshold

Perhaps the most common approach is to set a target for the amount of variance you wish to preserve and select the minimum number of components needed to meet that target. We do this by calculating the cumulative explained variance:

A common practice is to choose $k$ such that the cumulative explained variance is above a certain threshold, such as 95%, 99%, or even 80%, depending on the application. For tasks where information loss is critical (e.g., preprocessing for a supervised model), a higher threshold like 99% might be used. For data visualization, where the goal is to get a general sense of the data structure, a lower threshold might suffice.

The following video by Steve Brunton provides a concise explanation of this exact method.

2. The Scree Plot and the "Elbow" Method

A more visual and exploratory approach is to use a scree plot. This is simply a plot of the eigenvalues (or explained variance ratios) sorted in descending order.

The name comes from geology, where "scree" refers to the pile of broken rock fragments at the base of a cliff. In a typical scree plot for real-world data, you will see a sharp "cliff" formed by the first few large eigenvalues, followed by a flatter "scree" of many small eigenvalues. The idea is to identify the "elbow" or "knee" of this plot—the point where the drop in eigenvalue magnitude slows down considerably.

The principal components before the elbow are considered the significant ones that capture the main structure (the "signal") of the data. The components after the elbow are often interpreted as representing noise or minor variations.

Let's examine some excellent resources that explain and illustrate this concept.

The next video provides a dynamic illustration of interpreting the eigenvalue spectrum, which is precisely what a scree plot visualizes.

A Visual Synthesis

The two plots below show both methods in action for the same dataset.

As you can see, the two methods might suggest different numbers of components. The elbow method might suggest k=4, where the curve starts to level off, while a 95% variance threshold requires about k=10. There is no single correct answer; the choice depends on your goal.

• The Elbow Method is excellent for discovering the "natural" or "latent" dimensionality of your data.
• The Variance Threshold Method is better when you have a specific goal for information preservation, often in the context of feeding the reduced data into a subsequent algorithm.

A Note on SVD

In practice, PCA is often implemented using Singular Value Decomposition (SVD) rather than by explicitly forming the covariance matrix and solving for its eigenvalues. This is for reasons of numerical stability. The relationship is direct: the eigenvalues ${\lambda }_i$ of the covariance matrix $C_X=\frac{1}{n-1}X{X}^T$ are related to the singular values ${\sigma }_i$ of the mean-centered data matrix $X$ by ${\lambda }_i\propto {\sigma }_i^2$.

This means all the interpretations remain the same. The explained variance ratio can be calculated directly from the squared singular values.

Conclusion

In this lesson, we bridged the gap between the theory of PCA and its practical application for dimensionality reduction. You now have a principled framework for deciding how many principal components to retain, a critical step in using PCA effectively.

The key takeaways are:

• Explained Variance Ratio: The eigenvalues of the covariance matrix, when normalized by their sum, tell us the proportion of total data variance each principal component captures.
• Dimensionality Selection Methods: We have two primary tools for this task:
  1. Cumulative Explained Variance: Choose the number of components $k$ needed to exceed a predefined variance threshold (e.g., 95%).
  2. Scree Plot (Elbow Method): Visually identify the "elbow" point where the eigenvalues level off, suggesting the data's intrinsic dimensionality.
• Context Matters: The choice between these methods, and the specific parameters used (like the variance threshold), depends on the ultimate goal of your analysis, whether it's data visualization, noise reduction, or preprocessing for another model.

In our next lesson, we will explore a different pillar of unsupervised learning: probabilistic modeling. We will introduce Gaussian Mixture Models (GMMs), a powerful technique for density estimation and a more flexible alternative to K-means for clustering.