Data Dimensionality Reduction with PCA

1. Why Do We Need Fewer Dimensions? The Curse of Dimensionality

In machine learning, it's natural to think that "more data is better," and this often extends to "more features are better." However, having a very large number of features (i.e., high-dimensional data) can introduce significant problems.

As the article explains, a high number of dimensions leads to several issues:

• Visualization is Impossible: We can intuitively understand data in 2D or 3D, but we cannot visualize it in 10, 50, or 100 dimensions.
• Data Becomes "Sparse": As you add more dimensions, the amount of data needed to maintain the same density of coverage grows exponentially. Your data points become very far apart from each other, making it difficult for algorithms to find local patterns.
• Increased Risk of Overfitting: With more features, a model has more opportunities to learn noise and spurious correlations from the training data, hurting its ability to generalize to new data.
• Multicollinearity: It becomes more likely that some features are correlated with others, which can make certain models (like linear regression) unstable.

Dimensionality reduction techniques, like PCA, aim to solve these problems by creating a compact, lower-dimensional representation of the data while preserving as much of the original information as possible.

2. The Core Idea of PCA: Finding New, More Informative Axes

Imagine you have a 2D dataset where the two features are highly correlated, like a city's population and its average cost of living. The data points would form an elongated cloud on a scatter plot.

Instead of describing each point with its original "population" and "cost" values, could we find a new, single axis that runs along the direction of this elongated cloud? This new axis would capture most of the variation in the data. This is the central idea of PCA.

PCA finds a new set of axes, called Principal Components (PCs), that are ordered by how much of the data's variance they capture.

• Principal Component 1 (PC1): The axis that captures the most variance in the dataset. It's the "line of best fit" that passes through the centered data.
• Principal Component 2 (PC2): The axis that is perpendicular (orthogonal) to PC1 and captures the most remaining variance.
• And so on, for as many dimensions as you have.

The following video provides an excellent visual and conceptual explanation of this idea.

The key insight is that these new principal components are linear combinations of the original features. For example, in our city data, PC1 might be something like 0.7 * population + 0.7 * cost_of_living, representing an overall "urbanization" score.

3. The PCA Algorithm: Step-by-Step

Now that we have the intuition, let's walk through the steps required to perform PCA. Your background in engineering provides a solid foundation for understanding this process, even without deep expertise in linear algebra.

Step 1: Standardize the Data

PCA is sensitive to the scale of the features. A feature with a large variance (e.g., house prices in dollars) will dominate the PCA process over a feature with a small variance (e.g., number of bedrooms). To prevent this, we must first standardize each feature to have a mean of 0 and a standard deviation of 1.

Step 2: Compute the Covariance Matrix

Once the data is standardized, we compute the covariance matrix. In Module 2, we learned that covariance measures how two variables change together. The covariance matrix summarizes the pairwise covariance between all features. This matrix effectively describes the shape and orientation of our data cloud.

Step 3: Calculate Eigenvectors and Eigenvalues

This is the core mathematical step of PCA. Don't worry about the complex calculations; it's the what and why that matter here. For the covariance matrix:

• Eigenvectors represent the directions of the new axes (the principal components).
• Eigenvalues represent the magnitude or amount of variance captured by each eigenvector.

The eigenvector with the highest eigenvalue is our PC1. The eigenvector with the second-highest eigenvalue is PC2, and so on.

The following video provides a helpful conceptual bridge from the covariance matrix to eigenvectors and eigenvalues.

In practice, most libraries use a more numerically stable method called Singular Value Decomposition (SVD) to find the principal components, but the final result is the same.

Step 4: Transform the Data

After identifying the principal components, the final step is to project our original, standardized data onto the new axes. We do this by taking the dot product of our data with the matrix of chosen eigenvectors. If we want to reduce our data from 10 dimensions to 2, we would select the first two eigenvectors (PC1 and PC2) and project the data onto them.

The result is a new dataset with the desired number of dimensions, where each new feature is one of the principal components.

4. Applying PCA with scikit-learn

Now for the practical part. scikit-learn makes applying PCA incredibly straightforward. The process encapsulates all the steps we just discussed.

Let's see how to apply PCA to a dataset and reduce its dimensionality.

The key steps in the code are:

1. Standardize the data: x_scaled = StandardScaler().fit_transform(x)
2. Initialize and fit PCA: pca = PCA(n_components=2) and pca.fit(x_scaled)
3. Transform the data: x_pca = pca.transform(x_scaled)

The resulting x_pca is your new, lower-dimensional dataset.

Conclusion

In this lesson, we introduced the "Curse of Dimensionality" and motivated the need for dimensionality reduction. We then dove deep into the concepts and application of Principal Component Analysis, one of the most widely used techniques for this purpose.

Key Takeaways:

• PCA is an unsupervised learning technique used to reduce the dimensionality of a dataset.
• It works by finding a new set of orthogonal axes, called principal components, that are ordered by the amount of variance they explain.
• The PCA algorithm involves standardizing the data, computing the covariance matrix, and finding its eigenvectors (the PCs) and eigenvalues (the explained variance).
• scikit-learn provides a simple PCA object that handles these steps, allowing you to easily fit a model and transform your data into a lower-dimensional space.

Preview of the Next Lesson:
We've successfully applied PCA to reduce our data's dimensions. But a crucial question remains: how do we decide how many principal components to keep? Is it 2, 5, or 10? In the next lesson, we will learn how to interpret the results of PCA, using tools like the explained variance ratio and scree plots to make an informed decision and find the right balance between dimensionality reduction and information preservation.