Eigendecomposition and Orthonormal Eigenvectors of Symmetric Matrices

Recommended resources

The freely available resources this lesson will direct you to.

Welcome back. Last time, you computed distances between points and used the fact that an orthogonal matrix preserves lengths. This lesson asks a complementary question: can a matrix have orthogonal directions that it stretches without turning? For every real symmetric matrix, the answer is yes. You will compute those directions and stretch factors by hand, then verify the result in PyTorch. This tool will later help describe preferred directions in 3D data.

Allow about 40 minutes, including the video, reading, and code.

What symmetry guarantees

A real square matrix $A$ is symmetric when $A^{⊤} = A$ . An eigenvector is a nonzero vector $q$ satisfying

A q = λ q .

The number $λ$ is its eigenvalue. Along the direction $q$ , applying $A$ changes only the vector’s scale or sign, not its line of action.

The spectral theorem says that a real symmetric $n \times n$ matrix has $n$ orthonormal eigenvectors that form a basis of $R^{n}$ . Put these vectors in the columns of $Q$ , and their corresponding eigenvalues on the diagonal of $Λ$ . Then

Q^{⊤} Q = I, A Q = Q Λ, A = Q Λ Q^{⊤} .

Watch the opening of Linear Algebra - Spectral Decomposition from Arnold Yim for the theorem and an interpretation of the final equation.

Linear Algebra - Spectral Decomposition

Arnold Yim’s Linear Algebra - Spectral Decomposition introduces the spectral theorem and shows how an orthonormal eigenbasis reconstructs a symmetric matrix.

Watch the theorem for the guarantees about real eigenvalues and orthogonal eigenspaces. Continue through the decomposition; focus on why the inverse of the eigenvector matrix is its transpose, and how the matrix product becomes a sum of eigenvector outer products.

The distinction between orthogonal and orthonormal matters. Orthogonal vectors have zero dot product with one another; orthonormal vectors also each have length one. Thus $Q^{⊤} Q = I$ checks both properties at once: its off-diagonal entries are dot products between different columns, and its diagonal entries are their squared lengths. Because there are $n$ orthonormal columns in $R^{n}$ , they are an orthonormal basis, not merely an orthonormal set.

Compute a decomposition by hand

Consider

A = [2112] .

First find the eigenvalues by solving $det (A - λ I) = 0$ :

det [2 - λ 1 1 2 - λ] = (2 - λ)^{2} - 1 = (λ - 3) (λ - 1) .

The eigenvalues are $3$ and $1$ . For each one, solve $(A - λ I) q = 0$ . When $λ = 3$ , the equations give two equal components, so $(1, 1)^{⊤}$ is an eigenvector. When $λ = 1$ , they give opposite components, so $(1, - 1)^{⊤}$ is an eigenvector.

These vectors have dot product zero, but each has length $2$ . Normalize them and place them in columns, keeping each eigenvalue in the matching diagonal position:

q_{1} = \frac{1}{2} [11], q_{2} = \frac{1}{2} [1 - 1],

Q = \frac{1}{2} [11 1 - 1], Λ = [3001] .

Directly checking the dot products gives $q_{1}^{⊤} q_{1} = 1$ , $q_{2}^{⊤} q_{2} = 1$ , and $q_{1}^{⊤} q_{2} = 0$ . Hence $Q^{⊤} Q = I$ . Checking the eigenvector equations gives $A Q = Q Λ$ ; multiplying on the right by $Q^{⊤}$ recovers $A = Q Λ Q^{⊤}$ .

That last equation describes the action of $A$ on any vector. In the $Q$ -basis, its two coordinates are scaled independently—by $3$ and $1$ —before the result is expressed in the original coordinates.

Why the orthogonality check works

The hand calculation showed orthogonality for one matrix. Symmetry explains why it happens whenever eigenvalues are distinct. If $A u = λ u$ and $A v = μ v$ , then $A^{⊤} = A$ lets us calculate the same scalar in two ways:

(A u)^{⊤} v = u^{⊤} A v .

Substituting the eigenvector equations yields

λ u^{⊤} v = μ u^{⊤} v .

If $λ \neq = μ$ , their eigenvectors must therefore have zero dot product. The following reading develops this proof alongside a worked $3 \times 3$ decomposition.

8.2: Orthogonal Diagonalization

Read 8.2: Orthogonal Diagonalization on LibreTexts to see a three-dimensional calculation and the reason symmetry makes eigenvectors for distinct eigenvalues orthogonal.

In Section 8.2, read Example 4, beginning “Find an orthogonal matrix,” through its final diagonalization statement: the worked example. Track the order of its eigenvalues against the columns of its normalized eigenvector matrix. Then read the discussion immediately after Example 4, including Theorems 3 and 4, from “Actually, the fact that the eigenvectors in Example 8.2.4 are orthogonal is no coincidence” up to the introduction of Example 5: the orthogonality proof. Focus on where the proof uses A^\top=A.

There is an important qualification: two arbitrary eigenvectors with the same eigenvalue need not be orthogonal. For example, if $A$ has the value $2$ in both of its first two diagonal positions, then both $(1, 0, 0)^{⊤}$ and $(1, 1, 0)^{⊤}$ are eigenvectors with eigenvalue $2$ , but their dot product is $1$ . Within that shared eigenspace, you can choose an orthonormal basis instead. The spectral theorem guarantees a choice of orthonormal eigenvectors, not that every possible choice is already orthogonal.

Verify a 3D decomposition in PyTorch

For larger matrices, calculating a characteristic polynomial by hand quickly becomes cumbersome. For a real symmetric matrix, torch.linalg.eigh computes real eigenvalues and orthonormal eigenvectors directly. It returns eigenvalues in ascending order and places the corresponding eigenvectors in the columns of its second output.

import torch

A = torch.tensor(
    [[2., 1., 0.],
     [1., 2., 0.],
     [0., 0., 4.]],
    dtype=torch.float64,
)

eigenvalues, Q = torch.linalg.eigh(A)
Lambda = torch.diag(eigenvalues)
I = torch.eye(3, dtype=A.dtype)

print("eigenvalues:", eigenvalues)  # approximately [1., 3., 4.]
print("eigenvectors in columns:\n", Q)

assert torch.allclose(A, A.T)
assert eigenvalues.shape == (3,)
assert Q.shape == (3, 3)

# Orthonormality: all pairwise column dot products
assert torch.allclose(Q.T @ Q, I, atol=1e-12)

# Eigenvector equations, with each column matched to its eigenvalue
assert torch.allclose(A @ Q, Q @ Lambda, atol=1e-12)

# Reconstruction of the original matrix
assert torch.allclose(Q @ Lambda @ Q.T, A, atol=1e-12)

The upper-left $2 \times 2$ block is the matrix you decomposed by hand. Its eigenvalues $1$ and $3$ appear again, while the third coordinate direction has eigenvalue $4$ . Your printed eigenvectors may have signs opposite to those you wrote by hand: if $q$ is an eigenvector, so is $- q$ . A sign change does not invalidate any of the three checks.

The checks catch different mistakes. $Q^{⊤} Q \approx I$ verifies an orthonormal basis; $A Q \approx Q Λ$ verifies that each column is paired with the right eigenvalue; and $Q Λ Q^{⊤} \approx A$ verifies the full decomposition. Use a tolerance because floating-point outputs are approximate, even when the mathematical identities are exact.

For a 3D point cloud, a symmetric matrix constructed from its centered coordinates can describe how strongly the points vary along different directions. Its orthonormal eigenvectors provide directions in 3D space, while its eigenvalues quantify the associated variation. The computation you have learned here is the linear-algebra core of that geometric interpretation.

A real symmetric matrix can be decomposed as $A = Q Λ Q^{⊤}$ , with real eigenvalues in $Λ$ and an orthonormal eigenbasis in the columns of $Q$ . To compute one by hand, find each eigenvalue, solve for its eigenvectors, and choose normalized, mutually orthogonal vectors within any repeated-eigenvalue eigenspace. In code, verify orthonormality, eigenvector equations, and reconstruction rather than trusting a printed result alone.

Next, you will move from matrix operations to calculus: computing the gradient of a scalar field with respect to spatial coordinates and checking it with PyTorch autograd.