Applying PCA with Scikit-learn

Part 1: The PCA Workflow in scikit-learn

The beauty of a library like scikit-learn is that it abstracts away the complex linear algebra we discussed (like eigendecomposition) into a few simple commands. The standard workflow for applying PCA involves three main steps:

1. Standardize the Data: As we discussed, PCA is sensitive to the scale of your features. We must first standardize the data so each feature has a mean of 0 and a standard deviation of 1. We'll use scikit-learn's StandardScaler for this.
2. Instantiate the PCA Model: We create a PCA object from sklearn.decomposition.PCA. This is where we'll specify how many components we want to keep.
3. Fit and Transform: We apply the PCA object to our data using the .fit_transform() method.
   • The .fit() part is where scikit-learn does the heavy lifting: it calculates the mean, variance, covariance matrix, and the eigenvectors (principal components) and eigenvalues (explained variance) from your data.
   • The .transform() part then projects your original data onto the new principal component axes that were found during the fit.

Let's watch a video that demonstrates this entire process on a real-world dataset.

Part 2: Configuring and Interpreting PCA

The video demonstrated a powerful and common way to use PCA: specifying the desired percentage of variance to retain. Let's break down the two main ways you can control the dimensionality reduction.

Method 1: Specifying Variance to Keep (e.g., n_components=0.95)

This is the method you just saw in the video. When you set n_components to a float between 0.0 and 1.0, you are not telling PCA how many components to return. Instead, you are giving it a target: "Find the minimum number of principal components required to capture at least this much of the total variance in the original dataset."

In the video's example, it took 29 principal components to explain 95% of the variance in the 64-feature dataset. This is a great way to reduce dimensionality while ensuring a minimal loss of information.

Method 2: Specifying the Number of Components (e.g., n_components=2)

Often, your goal is to reduce data to a specific number of dimensions, most commonly 2 or 3 for visualization. In this case, you pass an integer to n_components.

from sklearn.decomposition import PCA




# We want to reduce the data to exactly 2 dimensions
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)

print(f"Original shape: {X_scaled.shape}")
print(f"Transformed shape: {X_pca.shape}")

Original shape: (1797, 64)
Transformed shape: (1797, 2)

By forcing the data into just two dimensions, you will likely lose more information, but you gain the ability to plot it on a 2D scatter plot. This trade-off is often worthwhile for gaining visual insights.

Checking the Explained Variance

How do you know how much information was captured by your chosen components? The fitted PCA object has an attribute called explained_variance_ratio_. This is the practical equivalent of the eigenvalues we discussed in the last lesson. It returns an array where each value is the percentage of total variance captured by the corresponding principal component.

As the article shows, the explained_variance_ratio_ for a PCA with n_components=2 on the breast cancer data is [0.4427, 0.1897]. This tells us:

• The first principal component accounts for about 44.3% of the total variance.
• The second principal component accounts for about 19.0% of the total variance.

Together, these two components capture approximately $44.3\%+19.0\%=63.3\%$ of the total information from the original 30-dimensional dataset. The remaining ~37% of the variance is lost in this transformation.

Part 3: Visualizing the Result

The primary reason to reduce data to 2 or 3 components is for visualization. By plotting the transformed data points on a 2D scatter plot, we can often see patterns, clusters, and relationships that were impossible to discern in high-dimensional space.

Let's look at the result of plotting the breast cancer data after reducing it to two principal components.

This is the power of PCA for data analysis. Even though we lost some information, the 2D projection reveals the underlying structure of the data. Here is another example using the famous Iris dataset, which has 4 features. PCA reduces it to 2 principal components, allowing the three different flower species to be visualized and distinguished.

This visual separation confirms that our new principal component axes are meaningful and have successfully captured the most important information that distinguishes the different classes in the dataset.

Conclusion

In this lesson, you bridged the gap between the theory of PCA and its practical application. You've learned the standard scikit-learn workflow to perform dimensionality reduction and how to interpret the results.

Key Takeaways:

• The scikit-learn workflow for PCA is: Standardize (StandardScaler), Instantiate (PCA), and Apply (fit_transform).
• You can control PCA by specifying the percentage of variance to retain (e.g., n_components=0.95) or a fixed number of components (e.g., n_components=2).
• The explained_variance_ratio_ attribute of a fitted PCA object tells you the proportion of total variance captured by each principal component, allowing you to quantify information loss.
• Applying PCA is a crucial step for visualizing high-dimensional data, often revealing hidden structures and clusters.

Preview of the Next Lesson:
You now have the ability to create these powerful 2D projections. In our next lesson, "Visualize the results of clustering on a 2D projection of data created with PCA," we will put this skill to direct use. You will first perform clustering on high-dimensional data and then use the PCA projection you learned to create today to visualize and validate your clustering results.