Eigenvalues and Eigenvectors

Introduction

In our last lesson, we saw how matrices act as operators that transform vectors and, by extension, entire geometric objects in space. We built matrices to perform specific transformations like rotation, scaling, and translation. This was about construction: starting with a desired geometric outcome and engineering the matrix to achieve it.

Today, we begin the process of deconstruction. Given an arbitrary matrix, can we uncover its fundamental properties? Can we find its most natural "axes" of action? This leads us to one of the most important concepts in linear algebra: eigenvalues and eigenvectors. These are special vectors that, when transformed by a matrix, do not change their direction, only their magnitude. The eigenvalue is the factor by which the eigenvector is scaled.

This lesson directly addresses the first outcome of our second module: to compute eigenvalues and eigenvectors of a matrix. We will start with the geometric intuition, derive the algebraic method for finding these values, and work through a concrete example. Mastering this computation is the first step toward understanding how to decompose complex transformations into their simplest, most essential components, a technique with profound applications in graphics, physics, and machine learning.

The Core Idea: Vectors that Don't Change Direction

When a matrix $A$ multiplies a vector $\vec{v}$, the resulting vector $A\vec{v}$ can point in any direction. However, for any given square matrix, there exist certain non-zero vectors that are special. When the matrix acts on them, the resulting vector is simply a scaled version of the original. It points along the exact same line.

These special vectors are the eigenvectors of the matrix, and the scaling factor is the corresponding eigenvalue, typically denoted by $\lambda$ (lambda).

This relationship is captured by the defining equation of the eigenvalue problem:

$$A\vec{v}=\lambda \vec{v}$$

where $A$ is an $n\times n$ matrix, $\vec{v}$ is a non-zero $n\times 1$ eigenvector, and $\lambda$ is its associated scalar eigenvalue.

To build a strong visual intuition for this concept, the following video from 3Blue1Brown is unparalleled.

Deriving the Characteristic Equation

The definition $A\vec{v}=\lambda \vec{v}$ is conceptually clear, but it's not immediately obvious how to solve for $\vec{v}$ and $\lambda$ since they are both unknown. We need an equation that allows us to find the eigenvalues first.

Let's rearrange the defining equation:

$$A\vec{v}-\lambda \vec{v}=\vec{0}$$

We want to factor out the vector $\vec{v}$. However, we cannot simply write $(A-\lambda )\vec{v}$ because $A$ is a matrix and $\lambda$ is a scalar.

Question: What mathematical object can we introduce to make this subtraction valid, such that multiplying by $\lambda$ is equivalent to scaling by the scalar $\lambda$?

Answer

We can use the **identity matrix**, $I$. The term $\lambda\vec{v}$ is identical to $(\lambda I)\vec{v}$. This allows us to write the equation in a form where both terms are matrix-vector products.

By introducing the identity matrix, our equation becomes:

$$A\vec{v}-(\lambda I)\vec{v}=\vec{0}$$

Now we can factor out $\vec{v}$:

$$(A-\lambda I)\vec{v}=\vec{0}$$

This equation is profound. It states that an eigenvector $\vec{v}$ is a non-zero vector that lies in the null space of the matrix $(A-\lambda I)$. In our first module, we learned a crucial fact about matrices that have a non-trivial (non-zero) null space: they must be singular. And the defining property of a singular matrix is that its determinant is zero.

This gives us the tool we need to find the eigenvalues. For the matrix $(A-\lambda I)$ to be singular, its determinant must be zero.

The Characteristic Equation:

$$\det (A-\lambda I)=0$$

This equation, known as the characteristic equation, depends only on $\lambda$. Solving it will give us the eigenvalues of the matrix $A$.

Both the 3Blue1Brown video you just watched and the following lecture by Professor Gilbert Strang from MIT walk through this exact derivation.

The Algorithm for Finding Eigenvalues and Eigenvectors

We can now formalize the computation into a two-step process.

1. Find the Eigenvalues ($\lambda$):

   • Construct the matrix $A-\lambda I$ by subtracting $\lambda$ from each diagonal element of $A$.
   • Compute the determinant of this matrix, $\det (A-\lambda I)$. This will result in a polynomial in $\lambda$ of degree $n$, called the characteristic polynomial.
   • Solve the characteristic equation $\det (A-\lambda I)=0$ for its roots. These roots are the $n$ eigenvalues of $A$.

2. Find the Eigenvectors ($\vec{v}$):

   • For each distinct eigenvalue ${\lambda }_i$ you found, substitute it back into the equation $(A-{\lambda }_{i}I)\vec{v}=\vec{0}$.
   • Solve this homogeneous system of linear equations for $\vec{v}$. This is equivalent to finding the basis for the null space of the matrix $(A-{\lambda }_{i}I)$. The non-zero vectors in this null space are the eigenvectors corresponding to ${\lambda }_i$.

Let's solidify this with a step-by-step example.

Worked Example

Let's find the eigenvalues and eigenvectors for the matrix:

$$A=\left(\begin{array}{cc}3& 1\\ 1& 3\end{array}\right)$$

Step 1: Find the Eigenvalues

First, we set up the characteristic equation $\det (A-\lambda I)=0$.

$$\det \left(\left(\begin{array}{cc}3& 1\\ 1& 3\end{array}\right)-\lambda \left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)\right)=0$$ $$\det \left(\begin{array}{cc}3-\lambda & 1\\ 1& 3-\lambda \end{array}\right)=0$$

Now, we compute the determinant of the 2x2 matrix:

$$(3-\lambda )(3-\lambda )-(1)(1)=0$$ $$9-6\lambda +{\lambda }^2-1=0$$ $${\lambda }^2-6\lambda +8=0$$

We solve this quadratic equation by factoring:

$$(\lambda -4)(\lambda -2)=0$$

The roots, and therefore our eigenvalues, are ${\lambda }_1=4$ and ${\lambda }_2=2$.

A Quick Sanity Check:
Two useful properties link eigenvalues to the trace and determinant of a matrix:

• The sum of the eigenvalues equals the trace of the matrix (the sum of the diagonal elements).
  • ${\lambda }_1+{\lambda }_2=4+2=6$.
  • Trace(A) = $3+3=6$. They match.
• The product of the eigenvalues equals the determinant of the matrix.
  • ${\lambda }_1{\lambda }_2=4\times 2=8$.
  • det(A) = $(3)(3)-(1)(1)=8$. They match.

This is an excellent way to verify your eigenvalues, especially in a performance-critical or research setting.

Step 2: Find the Eigenvectors

Now we find the eigenvector for each eigenvalue.

Case 1: ${\lambda }_1=4$
We solve $(A-4I)\vec{v}=\vec{0}$.

$$\left(\begin{array}{cc}3-4& 1\\ 1& 3-4\end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right)$$ $$\left(\begin{array}{cc}-1& 1\\ 1& -1\end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right)$$

This gives us the system of equations:
$-x+y=0$
$x-y=0$
Both equations tell us that $x=y$. Any non-zero vector where the components are equal is a valid eigenvector. For simplicity, we can choose:

$${\vec{v}}_1=\left(\begin{array}{c}1\\ 1\end{array}\right)$$

Case 2: ${\lambda }_2=2$
We solve $(A-2I)\vec{v}=\vec{0}$.

$$\left(\begin{array}{cc}3-2& 1\\ 1& 3-2\end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right)$$ $$\left(\begin{array}{cc}1& 1\\ 1& 1\end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right)$$

This gives us the equation $x+y=0$, or $y=-x$. Any non-zero vector satisfying this is a valid eigenvector. We can choose:

$${\vec{v}}_2=\left(\begin{array}{c}1\\ -1\end{array}\right)$$

We have successfully computed the eigenvalues and a corresponding set of eigenvectors for matrix $A$. Notice that any scalar multiple of an eigenvector is also an eigenvector. For example, $\left(\begin{array}{c}2\\ 2\end{array}\right)$ is also an eigenvector for ${\lambda }_1=4$. The eigenvectors for a given eigenvalue form a subspace called the eigenspace.

Conclusion

In this lesson, we established the fundamental theory and practice of computing eigenvalues and eigenvectors. You learned that these pairs represent the intrinsic scaling behavior of a linear transformation—the directions that are preserved under the transformation.

Key Takeaways:

• Defining Equation: The eigenvalue problem is defined by $A\vec{v}=\lambda \vec{v}$, where $\vec{v}$ is a non-zero eigenvector and $\lambda$ is its corresponding scalar eigenvalue.
• Characteristic Equation: To find the eigenvalues, we solve the characteristic equation $\det (A-\lambda I)=0$, which follows from the fact that $(A-\lambda I)$ must be a singular matrix to have a non-trivial null space.
• Computational Process: Finding eigenvalues requires solving a polynomial equation. For each eigenvalue found, the corresponding eigenvectors are found by solving the homogeneous system $(A-\lambda I)\vec{v}=\vec{0}$, which is equivalent to finding the null space of $(A-\lambda I)$.
• Verification: The sum of eigenvalues equals the matrix trace, and their product equals the determinant, providing a valuable check on your calculations.

We have successfully built the computational machinery. In our next lesson, we will put it to use by exploring its powerful geometric interpretation. We'll see how finding the eigenvalues and eigenvectors of a matrix is equivalent to discovering a new basis in which the transformation acts as a simple scaling—a process known as eigendecomposition. This will reveal the "principal axes" of the transformation, a cornerstone for understanding techniques from Principal Component Analysis (PCA) to Singular Value Decomposition (SVD).