Eigenvalues and Eigenvectors with NumPy

1. From Manual Calculation to Computational Tools

Remember the process for finding eigenvectors and eigenvalues by hand?

1. Set up the characteristic equation: $\det (A-\lambda I)=0$.
2. Solve this polynomial equation for the eigenvalues ($\lambda$).
3. For each eigenvalue, plug it back into $(A-\lambda I)v=0$ and solve the system of linear equations to find the corresponding eigenvector(s) $v$.

Let's watch a brief segment from the 3Blue1Brown video you saw last lesson, which quickly demonstrates steps 2 and 3.

As you can imagine, for a 3x3 matrix, you'd be solving a cubic equation. For a 100x100 matrix, this becomes computationally infeasible to do by hand. This is where NumPy comes in.

2. Calculating Eigenvalues and Eigenvectors with numpy.linalg.eig()

The NumPy library provides a direct and efficient function for this task: numpy.linalg.eig(). This function takes a square matrix as input and returns its eigenvalues and eigenvectors.

Let's see how this works in practice.

As the article explains, np.linalg.eig(A) returns two things:

1. eigenvalues: A 1D NumPy array containing the eigenvalues of matrix A.
2. eigenvectors: A 2D NumPy array (a matrix) where the columns are the eigenvectors corresponding to the eigenvalues. The eigenvector eigenvectors[:, i] corresponds to the eigenvalue eigenvalues[i].

A crucial point is that NumPy returns normalized eigenvectors, meaning their length (L2 norm) is 1. This is a standard convention that makes them easier to work with.

Here's a visual summary of the process:

3. A Complete Example: Calculation and Verification

Let's work through a full example with a 3x3 matrix to solidify your understanding. We'll not only calculate the values but also verify that they are correct.

This process of verification is a great habit in programming and machine learning—always check that your functions are producing the expected output. How do we verify? We check if the fundamental equation $Av=\lambda v$ holds true for the values we found.

The following resource provides a perfect template for this.

Now, let's put it all together in code.

Step 1: Define the matrix and calculate

Let's use the same 3x3 matrix from the article.

import numpy as np

# Define our matrix
A = np.array([[1, 2, 3], 
              [4, 5, 6], 
              [7, 8, 9]])

# Calculate eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(A)

print("Eigenvalues:")
print(eigenvalues)
print("\nEigenvectors (as columns):")
print(eigenvectors)

Running this gives us:

• eigenvalues: [ 1.61168440e+01 -1.11684397e+00 -1.30367773e-15]
• eigenvectors: A 3x3 matrix where the first column corresponds to the first eigenvalue, and so on.

Note that the third eigenvalue is very close to zero (-1.3e-15). This is due to floating-point precision limitations and can be treated as zero. This also implies the matrix A is singular (or very close to it), which we know is true because its third row is a linear combination of the first two (row3 = 2*row2 - row1).

Step 2: Verify the first eigenvector/eigenvalue pair

Let's confirm the relationship $Av=\lambda v$ for the first pair.

# Get the first eigenvalue
lambda_1 = eigenvalues[0]

# Get the first eigenvector (the first column of the eigenvectors matrix)
v_1 = eigenvectors[:, 0]

# Calculate Av on the left side of the equation
Av_1 = A.dot(v_1)

# Calculate λv on the right side of the equation
lambda_v_1 = lambda_1 * v_1

print("A * v_1:")
print(Av_1)
print("\nλ_1 * v_1:")
print(lambda_v_1)

# Check if they are close (using np.allclose for floating point numbers)
print("\nAre they equal?", np.allclose(Av_1, lambda_v_1))

The output will show two almost identical vectors, and np.allclose() will return True, confirming our result!

# Get the second eigenvalue
lambda_2 = eigenvalues[1]

# Get the second eigenvector (the second column)
v_2 = eigenvectors[:, 1]

# Calculate Av on the left side
Av_2 = A.dot(v_2)

# Calculate λv on the right side
lambda_v_2 = lambda_2 * v_2

print("A * v_2:")
print(Av_2)
print("\nλ_2 * v_2:")
print(lambda_v_2)

# Check for equality
print("\nAre they equal?", np.allclose(Av_2, lambda_v_2))

Conclusion

Congratulations on completing the linear algebra module! You've moved from basic vector and matrix operations to understanding the deep, geometric properties of matrices through determinants, inverses, and now, eigenvalues.

Key Takeaways from this lesson:

• Solving for eigenvalues and eigenvectors by hand is instructive but not practical. We use computational libraries in real-world applications.
• numpy.linalg.eig() is the standard function in Python for this task.
• It returns a tuple: a 1D array of eigenvalues and a 2D array where the columns are the corresponding eigenvectors.
• The eigenvectors are normalized (unit length).
• You can and should verify your results by confirming that $Av=\lambda v$ for your calculated values, using np.allclose() to handle floating-point comparisons.

Preview of the Next Module:

We have now established the "language" of machine learning—linear algebra. We can represent data as vectors and operations as matrices. The next big question is: how do models learn? Learning is fundamentally a process of optimization—finding the best model parameters to minimize error. To navigate the landscape of error functions and find these minimums, we need the tools of calculus. In our next lesson, we will begin our journey into calculus by learning how to compute derivatives of single-variable functions.