Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 52

Principal Component Analysis (PCA) for Dimensionality Reduction

Recommended resources

Hello! Welcome to your next lesson.

In our previous lessons, we explored clustering algorithms like K-Means and hierarchical clustering, which are fantastic tools for finding hidden groups in your data. Today, we're shifting gears to another powerful unsupervised learning technique: dimensionality reduction.

Imagine having a dataset with hundreds or even thousands of features. Working with such high-dimensional data can be computationally expensive, difficult to visualize, and can sometimes even hurt the performance of your machine learning models due to the "curse of dimensionality."

This lesson introduces Principal Component Analysis (PCA), the most widely used technique for dimensionality reduction. Our goal is to learn how to perform dimensionality reduction using PCA. We'll cover the core intuition behind how it works and, most importantly, how to apply it in Python as part of a practical machine learning workflow.

Let's begin!

1. What Is Principal Component Analysis (PCA)?

At its core, PCA is a technique that transforms your data into a new, lower-dimensional space. It does this by creating a new set of features, called principal components, which are linear combinations of the original features. The key idea is that these new components are ordered by the amount of variance in the data they capture.

  • PC1 (Principal Component 1): The first component is the new axis that captures the most variance in the data.
  • PC2 (Principal Component 2): The second component is perpendicular to the first and captures the second-most variance.
  • ...and so on.

By keeping only the first few principal components, we can reduce the number of dimensions in our dataset while retaining the majority of the information (variance).

Dimensionality Reduction with Principal Component Analysis (PCA) Visualization
This image illustrates the core idea of PCA. Data points in a 3D space are projected onto a 2D plane defined by the two most important new axes, Principal Component 1 (PC1) and Principal Component 2 (PC2). This reduces the dimensionality from 3 to 2 while preserving the main structure of the data.

Why is this useful?

  • Visualization: It allows you to visualize high-dimensional data in 2D or 3D.
  • Performance: Training models on fewer features is much faster.
  • Noise Reduction: By discarding components with low variance, you often filter out noise in the data, which can sometimes improve model accuracy.

To build a solid intuition for how PCA finds these components, the following video provides a brilliant step-by-step explanation.

StatQuest: Principal Component Analysis (PCA), Step-by-Step

This video from StatQuest with Josh Starmer breaks down the geometric intuition behind PCA. It explains how PCA finds the lines of best fit (the principal components) that maximize the variance of the projected data.

Please watch from 03:32 to 15:06. Focus on these key ideas: How the data is first centered at the origin. The concept of finding the best-fitting line (PC1) by maximizing the sum of squared distances of the projected points from the origin. How PC2 is simply the line perpendicular to PC1 in a 2D space. How the final PCA plot is created by rotating the axes so that PC1 is horizontal.

2. Performing PCA with Scikit-Learn: A Practical Walkthrough

Now that we have the conceptual foundation, let's put it into practice. We'll use the popular scikit-learn library to perform PCA on a real dataset.

A critical prerequisite for PCA is feature scaling. Because PCA is trying to find directions of maximum variance, if your features are on vastly different scales (e.g., one feature ranges from 0-1 and another from 0-1,000,000), the feature with the larger scale will dominate the analysis. Using StandardScaler to give all features a mean of 0 and a standard deviation of 1 is a standard and highly recommended practice.

The following video will guide you through a complete example using the handwritten digits dataset, which has 64 features (one for each pixel in an 8x8 image). This is a perfect use case for PCA.

Machine Learning Tutorial Python - 19: Principal Component Analysis (PCA) with Python Code

This practical tutorial from the 'codebasics' channel demonstrates the end-to-end process of applying PCA in Python.

Watch the segment from 09:05 to 21:14. As you watch, pay close attention to the following steps in the code: Loading and Preparing Data (09:05 - 14:17): See how the digits dataset is loaded and structured into features (X) and a target (y). Scaling the Features (14:17 - 15:03): This is a crucial step! Notice the use of StandardScaler to scale the data before applying PCA. Applying PCA (16:36 - 19:35): See how PCA is instantiated. Notice the use of n_components=0.95. This is a powerful feature that tells PCA to automatically select the number of components needed to retain 95% of the variance. Evaluating the Result (19:35 - 21:14): The video trains a Logistic Regression model on both the original and the PCA-transformed data to compare accuracy. This shows the practical trade-off: a significant reduction in dimensionality with only a very small drop in model performance.

This hands-on example demonstrates the power of PCA: we reduced the number of features from 64 down to 29 (a >50% reduction!) while keeping the model's accuracy almost identical.

3. Integrating PCA into a Machine Learning Pipeline

In a real-world project, you wouldn't perform scaling and PCA as separate, manual steps. You would integrate them into a seamless workflow using scikit-learn's Pipeline. A Pipeline chains together multiple steps (like scaling, dimensionality reduction, and a model) into a single object that behaves like a regular estimator.

This is extremely useful because it prevents data leakage (e.g., fitting the scaler on your test data) and simplifies your code.

The following article provides an excellent guide on how to do this.

Principal Component Analysis for Dimensionality Reduction in Python

This article from Machine Learning Mastery gives a clear, code-driven example of how to use PCA for dimensionality reduction within a Pipeline and how to find the optimal number of components.

Please read the sections 'PCA Scikit-Learn API' and 'Worked Example of PCA for Dimensionality Reduction'. I encourage you to run the code provided. Focus on: The syntax for creating a Pipeline that combines PCA and a LogisticRegression model. How the cross_val_score function is used to evaluate the entire pipeline. The experiment where the code loops through different values for n_components to see how model accuracy changes. This is a great example of the diagnostic, hands-on approach you're interested in.

This approach of systematically testing different numbers of components and measuring the impact on model performance is a core skill in practical machine learning. The results from the article show a clear trade-off: as you increase the number of components, accuracy rises, but so does the computational complexity. The sweet spot is often where the accuracy curve begins to plateau.

4. An Important Clarification: Transformation vs. Selection

A common point of confusion is whether PCA is a "feature selection" method. It is not.

  • Feature Selection methods (like RFE) select a subset of your original features. The remaining features are unchanged.
  • Feature Transformation methods, like PCA, create a set of entirely new features (the principal components) by combining the original ones. The new features are not directly interpretable as your original columns.

The pca.components_ attribute in scikit-learn can show you the "recipe" for each principal component (i.e., the weight of each original feature in its creation), but you cannot say that PCA "selected" certain original features and discarded others. It transformed all of them. The "Machine Learning Mastery" article you just read has a great discussion on this in the comments section (response #19 and onwards) if you're curious to read more.

Conclusion

Great work! You've just added a fundamental dimensionality reduction technique to your toolkit. PCA is not just a theoretical concept; it's a practical tool used across data science for visualization, noise filtering, and making models more efficient.

Key Takeaways:

  • PCA reduces dimensionality by creating new features, called principal components, that capture the maximum variance in the data.
  • It is essential to scale your data (e.g., using StandardScaler) before applying PCA to prevent features with large scales from dominating the analysis.
  • You can apply PCA in scikit-learn by specifying the exact number of components (n_components=k) or the desired percentage of variance to retain (n_components=0.95).
  • For robust machine learning workflows, PCA should be integrated into a Pipeline along with scaling and the final model.
  • PCA is a feature transformation technique, not a feature selection technique. It creates new, synthetic features rather than picking a subset of the original ones.

Preview of the next lesson:
In this lesson, we saw how to use n_components=0.95 to let PCA choose the number of components for us. But how does it know what "95% of the variance" means? In the next lesson, we will dive deeper into the concept of explained variance. You'll learn how to plot and interpret the cumulative explained variance to make a more informed decision about how many principal components to keep for your specific problem.