Eigenvalues and Eigenvectors Explained

1. The Core Idea: Vectors That Don't Change Direction

When a matrix transforms space, it can feel chaotic—vectors are rotated, sheared, and stretched all at once. However, for any given transformation, there are almost always a few special vectors that don't get rotated off their original line. The only thing the transformation does to them is stretch, shrink, or flip them. These special vectors are called eigenvectors.

The word "eigen" is German for "own" or "characteristic," so an eigenvector is a characteristic vector of a matrix. It reveals the natural axes of the transformation.

To build a strong visual intuition for this, let's watch a segment from the excellent "Essence of linear algebra" series by 3Blue1Brown.

As the video showed, eigenvectors are the vectors that stay on their own span after the transformation.

Here's another great visual using the Mona Lisa. Imagine the transformation is shearing the image.

2. Eigenvalues: The Scaling Factor

So, an eigenvector keeps its direction. But by how much is it stretched or shrunk? This scaling factor is called the eigenvalue, usually denoted by the Greek letter lambda ($\lambda$).

Every eigenvector has a corresponding eigenvalue.

• If $\lambda >1$, the eigenvector is stretched.
• If $0<\lambda <1$, the eigenvector is shrunk.
• If $\lambda =1$, the eigenvector is unchanged. This is a "steady state."
• If $\lambda$ is negative, the eigenvector is flipped and then scaled.
• If $\lambda =0$, the eigenvector is squashed to the origin. This means the eigenvector is in the nullspace of the matrix.

Let's continue with the Jon Krohn video, where he explains eigenvalues in the context of the same Mona Lisa example.

To get a hands-on feel for this, the following interactive visualization is excellent. You can move the matrix columns (the green and red arrows) to define a transformation and then drag the blue vector around to find the directions where it becomes an eigenvector.

3. The Formal Definition and Characteristic Equation

Now that we have the intuition, let's formalize it. We said that multiplying an eigenvector $v$ by a matrix $A$ has the same effect as multiplying $v$ by a scalar eigenvalue $\lambda$. This gives us the fundamental eigenvalue equation:

Where:

• $A$ is an $n\times n$ matrix (the transformation).
• $v$ is an $n\times 1$ column vector (the eigenvector).
• $\lambda$ is a scalar (the eigenvalue).

This single equation elegantly captures the entire concept.

How do we use this to find the eigenvalues and eigenvectors? We can rearrange the equation. Your experience with algebra and programming will make this manipulation straightforward.

1. Move $\lambda v$ to the left side:

   $$Av-\lambda v=0$$

2. To factor out the vector $v$, we need both terms to be matrix-vector products. We can't subtract a scalar $\lambda$ from a matrix $A$. So, we cleverly multiply $\lambda$ by the identity matrix $I$. Since $Iv=v$, this doesn't change the equation:

   $$Av-\lambda Iv=0$$

3. Now we can factor out $v$:

   $$(A-\lambda I)v=0$$

Let's pause and think about this equation. It states that the (non-zero) eigenvector $v$ is in the nullspace of the matrix $(A-\lambda I)$.

In our previous lesson, we learned a crucial fact: for a matrix to have a non-trivial nullspace (i.e., for it to collapse space and have non-zero solutions to $Mx=0$), its determinant must be zero.

Therefore, for a non-zero eigenvector $v$ to exist, the matrix $(A-\lambda I)$ must be singular. This leads us to the characteristic equation:

This is the key to finding the eigenvalues of a matrix. It's a polynomial equation in $\lambda$, and its roots are the eigenvalues of $A$.

The following video segment walks through this exact derivation visually.

For a more formal, text-based explanation of this process, the following resource from MIT OpenCourseWare is a standard reference.

Conclusion

In this lesson, we've built a strong conceptual foundation for eigenvalues and eigenvectors. You now understand them not just as abstract mathematical objects, but as fundamental properties of linear transformations with clear geometric meaning.

Key Takeaways:

• Eigenvectors are the special, non-zero vectors that do not change their direction (they stay on their span) when a linear transformation is applied.
• Eigenvalues ($\lambda$) are the scalar factors by which the eigenvectors are stretched, shrunk, or flipped.
• The relationship is defined by the core equation: $Av=\lambda v$.
• To find the eigenvalues, we solve the characteristic equation $\det (A-\lambda I)=0$, which forces the matrix $(A-\lambda I)$ to be singular.

Preview of the Next Lesson:

We have focused on the "what" and "why" of eigenvalues. In the next lesson, we will shift to the "how." While solving the characteristic equation by hand is instructive, in practice, we always use numerical libraries. You will learn how to calculate eigenvalues and eigenvectors for any given matrix using NumPy's numpy.linalg module, a practical skill you'll use throughout your machine learning journey.