Eigenvalues and Eigenvectors: Geometric Interpretation

Welcome to the second module of our course, where we delve into the fascinating world of Eigenanalysis and Matrix Decompositions.

Introduction

In the previous module, we established a strong foundation in linear algebra, culminating in our study of the determinant and matrix inverse. We learned that the determinant, a single number, reveals whether a matrix is invertible. You might recall the preview from our last lesson: this concept of the determinant is central to the topic we are beginning today.

This lesson introduces eigenvalues and eigenvectors, arguably one of the most important concepts in linear algebra, with profound applications across AI and data science, from Principal Component Analysis (PCA) to the theory behind Google's PageRank algorithm.

Our goal is to define eigenvalues and eigenvectors and explain their geometric interpretation in relation to linear transformations. We will explore the fundamental question: for a given linear transformation represented by a matrix $A$, are there any vectors whose direction is unchanged by the transformation? These "special" vectors, called eigenvectors, reveal the true "axes" of a transformation, and the corresponding scaling factors, the eigenvalues, tell us how the transformation acts along these axes.

1. The Eigenvalue-Eigenvector Equation

At its core, the concept is captured by a single, elegant equation. Let's begin with a formal definition.

Formally, for a given $n\times n$ matrix $A$, a non-zero vector $\mathbf{x}$ is an eigenvector of $A$ if it satisfies the equation:

for some scalar $\lambda$. The scalar $\lambda$ is called the eigenvalue corresponding to the eigenvector $\mathbf{x}$.

Let's break this down:

• Matrix $A$: Represents a linear transformation.
• Vector $\mathbf{x}$: This is the eigenvector. It must be non-zero. Why? Because if $\mathbf{x}$ were the zero vector, the equation $A0=\lambda 0$ would hold for any matrix $A$ and any scalar $\lambda$, making it a trivial and uninformative case.
• Scalar $\lambda$: This is the eigenvalue. It's a number that tells us how the eigenvector $\mathbf{x}$ is scaled by the transformation.
  • If $\lambda >1$, the vector is stretched.
  • If $0<\lambda <1$, the vector is shrunk.
  • If $\lambda <0$, the vector's direction is reversed.
  • If $\lambda =1$, the vector is unchanged.
  • If $\lambda =0$, the vector is mapped to the zero vector, meaning it lies in the null space of $A$.

The word "eigen" is German for "proper" or "characteristic". Thus, eigenvectors are the characteristic vectors of a linear transformation.

2. The Geometric Meaning: Axes of Transformation

The algebraic definition $A\mathbf{x}=\lambda \mathbf{x}$ has a powerful geometric interpretation. Most vectors, when multiplied by a matrix $A$, are both scaled and rotated. Eigenvectors are special because they are only scaled. Their direction, or more precisely the line they span, is invariant under the transformation.

This is best understood visually.

As the video explains, finding the eigenvectors and eigenvalues of a transformation helps us understand its "heart" in a way that is independent of our chosen coordinate system. They represent the fundamental directions along which the transformation acts most simply.

3. Examples of Eigenvectors in Action

Let's solidify this geometric intuition with a few concrete examples of matrices whose transformations are easy to visualize.

Let's summarize the key insights from these examples:

• Projection Matrix (onto a plane):

  • Any vector $\mathbf{x}$ already lying in the plane of projection is an eigenvector. The projection doesn't change it, so $P\mathbf{x}=\mathbf{x}$. The corresponding eigenvalue is $\lambda =1$.
  • Any vector $\mathbf{x}$ perpendicular to the plane is also an eigenvector. The projection maps it to the zero vector, so $P\mathbf{x}=0=0\mathbf{x}$. The corresponding eigenvalue is $\lambda =0$. This shows that eigenvectors for $\lambda =0$ are precisely the non-zero vectors in the null space.

• Permutation Matrix ($A=\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)$):

  • The vector ${\mathbf{x}}_1=\left(\begin{array}{c}1\\ 1\end{array}\right)$ is an eigenvector. The transformation swaps its components, but since they are equal, the vector is unchanged: $A{\mathbf{x}}_1={\mathbf{x}}_1$. The eigenvalue is ${\lambda }_1=1$.
  • The vector ${\mathbf{x}}_2=\left(\begin{array}{c}1\\ -1\end{array}\right)$ is also an eigenvector. The transformation gives $\left(\begin{array}{c}-1\\ 1\end{array}\right)$, which is exactly $-{\mathbf{x}}_2$. The eigenvalue is ${\lambda }_2=-1$.

4. From Geometry to Algebra: Finding Eigenvalues

Visual inspection works for simple matrices, but we need a systematic method for finding eigenvalues and eigenvectors for any $n\times n$ matrix. This is where your knowledge of determinants becomes crucial.

Let's revisit the defining equation:
$A\mathbf{x}=\lambda \mathbf{x}$

To solve this, we bring all terms to one side. Since $\lambda$ is a scalar, we must introduce the identity matrix $I$ to perform matrix subtraction.
$A\mathbf{x}-\lambda I\mathbf{x}=0$
$(A-\lambda I)\mathbf{x}=0$

This equation is fundamental. It states that an eigenvector $\mathbf{x}$ is a non-zero vector in the null space of the matrix $(A-\lambda I)$.

From our previous work, we know that a matrix has a non-trivial null space (i.e., its null space contains more than just the zero vector) if and only if the matrix is singular. And the defining property of a singular matrix is that its determinant is zero.

This gives us our method for finding eigenvalues:
The number $\lambda$ is an eigenvalue of $A$ if and only if the matrix $(A-\lambda I)$ is singular. This means:

This equation is called the characteristic equation of the matrix $A$.

The expression $\det (A-\lambda I)$ is a polynomial in $\lambda$ of degree $n$, called the characteristic polynomial. The roots of this polynomial are the eigenvalues of $A$. We will dedicate our next lesson to mastering the computation and solution of this equation.

Conclusion

In this lesson, we introduced the foundational concepts of eigenvalues and eigenvectors. You learned that they are not just an algebraic curiosity but the key to understanding the geometric essence of a linear transformation.

Key Takeaways:

• Eigenvectors are non-zero vectors whose direction is invariant under a linear transformation; they are only scaled.
• The defining equation is $A\mathbf{x}=\lambda \mathbf{x}$, where $\mathbf{x}$ is the eigenvector and $\lambda$ is the corresponding eigenvalue.
• The eigenvalue $\lambda$ represents the scaling factor along the direction of the eigenvector.
• Geometrically, eigenvectors represent the "principal axes" of a transformation.
• The search for eigenvalues leads to solving the characteristic equation: $\det (A-\lambda I)=0$.

Preview of the Next Lesson:
We have established the what and the why of eigenvalues. In the next lesson, we will focus on the how. We will delve into the characteristic polynomial, derive it for general matrices, and use it to systematically compute the eigenvalues of a matrix. This will be our first major step into the computational machinery of eigenanalysis.