Hello! Welcome back to our journey through the mathematical foundations of machine learning.
In our last lesson, we explored how multiplying a vector by a matrix applies a linear transformation—essentially rotating and scaling the vector. This is a core operation for processing data in many models.
Today, we'll focus on special vectors that have a uniquely simple relationship with these transformations. Our learning outcome is to explain the role of eigenvalues and eigenvectors in representing the principal axes of data variation. We'll first build an intuition for what eigenvalues and eigenvectors are, and then connect this powerful concept directly to how we can understand the fundamental structure and "shape" of a dataset. This is a cornerstone of dimensionality reduction techniques like Principal Component Analysis (PCA).
What are Eigenvectors and Eigenvalues?
When a matrix transforms a space, most vectors are knocked off their original line (their "span"). However, some special vectors are not rotated at all; they are simply stretched, squished, or flipped. These are the eigenvectors. The factor by which they are scaled is their corresponding eigenvalue.
To get a strong visual intuition for this, the following video is one of the best explanations available.
Eigenvectors and eigenvalues | Chapter 14, Essence of linear algebra
Watch this segment from the 'Essence of linear algebra' series by 3Blue1Brown. It provides a fantastic visual introduction to the core idea of eigenvectors and eigenvalues.
Watch from 01:19 to 05:15. Focus on how the narrator distinguishes between vectors that get rotated off their span and the special 'eigenvectors' that stay on their span, only changing in length. The 3D rotation example provides a great intuition for why finding these special vectors is useful.
As the video illustrates, this relationship can be summarized by the following iconic equation:
Let's break this down:
- is the matrix representing the linear transformation.
- is the eigenvector, a non-zero vector.
- (lambda) is the corresponding eigenvalue, which is a scalar number.
This equation elegantly states that applying the transformation to the eigenvector produces the exact same result as simply scaling by its eigenvalue . The eigenvector's direction remains unchanged.
Finding the "Shape" of Data
Now, let's connect this abstract idea to machine learning. Imagine you have a dataset with two features, plotted as a cloud of points.
How can we mathematically identify the main axes along which this data is spread? This is where eigenvalues and eigenvectors come into play, but first we need to introduce a tool that summarizes the "shape" of our data: the covariance matrix.
The Covariance Matrix: A Summary of Variation
For a dataset, the covariance matrix tells us how its features vary.
- The diagonal elements are the variances of each feature (how spread out it is).
- The off-diagonal elements are the covariances between pairs of features (how they change together). A positive covariance means that as one feature increases, the other tends to increase as well.
Let's watch a video that explains variance, covariance, and how they form the covariance matrix.
Principal Component Analysis (PCA)
This video from Serrano.Academy gives a clear, geometric explanation of variance and covariance, and then shows how they are assembled into the covariance matrix.
Watch from 05:03 to 13:38. Focus on understanding what variance measures (spread) and what covariance measures (how two variables move together). Then, see how these values are placed into a 2x2 matrix to summarize the data's properties.
Eigenvectors of the Covariance Matrix = Principal Axes of Data
Here is the central idea of today's lesson:
When we treat the covariance matrix of our data as a linear transformation, its eigenvectors and eigenvalues tell us about the data's variation.
- The eigenvectors of the covariance matrix point in the directions of maximum variance in the data. These are the principal axes. The eigenvector with the largest eigenvalue points along the direction of the most variance.
- The eigenvalues quantify the amount of variance along these axes. A large eigenvalue means the data is very spread out along the corresponding eigenvector's direction.
This is the mathematical engine behind Principal Component Analysis (PCA). By finding these principal axes, we can transform our data to a new coordinate system where the axes are aligned with the directions of greatest variance.
The following resource provides an excellent animated view of this concept.
Eigenvalues and eigenvectors in PCA
This article from Towards Data Science provides some great animated visuals to connect the concepts.
Read the sections 'Eigenvectors', 'Eigenvalues', and 'Applications of eigenvectors and eigenvalues values in Data Science' up to the part about PCA. The GIFs in these sections are particularly helpful for visualizing how eigenvectors of the covariance matrix align with the data's spread.
Now, let's see how this all comes together in the context of PCA.
Principal Component Analysis (PCA)
Let's return to the Serrano.Academy video to see the final connection. It demonstrates how finding the eigenvectors and eigenvalues of the covariance matrix reveals the principal components of the data.
Watch from 20:30 to 23:26. This part is the culmination of the lesson. Notice how the eigenvector with the larger eigenvalue corresponds to the red vector, which aligns perfectly with the main direction of spread in the data cloud. This is the first principal component.
By identifying these principal axes, we can perform dimensionality reduction. We can project the data onto the first few principal components (the eigenvectors with the largest eigenvalues) and discard the rest, thereby reducing the number of features while retaining most of the information (variance) in the data.
Test your understanding!
Imagine a 2D dataset where the data points form a perfect circle centered at the origin. What can you say about the eigenvalues of its covariance matrix?
Show answer
If the data forms a perfect circle, it means the variance (spread) is the same in every direction. There is no single "principal" axis of variation.
In this case, the two eigenvalues of the covariance matrix would be equal. Any vector could serve as an eigenvector because the transformation (defined by the covariance matrix) scales the space uniformly in all directions.
Finally, for a concise summary of how these concepts fit together, let's read the conclusion of this helpful article.
Understanding the Role of Eigenvectors and Eigenvalues ...
This article from Medium provides a very clear, final summary of why this process works.
Read the section titled 'But then, why does PCA work?'. This neatly summarizes the roles of the covariance matrix, eigenvectors, and eigenvalues in PCA.
Conclusion
In this lesson, we demystified eigenvalues and eigenvectors and connected them to a crucial task in machine learning: understanding data structure.
Here are the key takeaways:
- Eigenvectors are special vectors that are only scaled by a linear transformation, not rotated. Eigenvalues are their corresponding scaling factors.
- The covariance matrix is a compact representation of the variance and inter-dependencies of features in a dataset.
- The eigenvectors of the covariance matrix are the principal axes of variation in the data. The eigenvector with the largest eigenvalue points in the direction where the data is most spread out.
- The eigenvalues quantify the amount of variance along these principal axes.
- This principle is the foundation of Principal Component Analysis (PCA), a widely used technique for dimensionality reduction.
In our next lesson, we will shift gears from linear algebra back to calculus. We'll explore the chain rule and see how it enables the process of backpropagation, the algorithm used to train most neural networks.