Orthogonal Projection onto a Subspace

Hello! Welcome to your next lesson on linear algebra.

In our last lesson, we learned the Gram-Schmidt process, an algorithm that uses projection onto a single vector as a key building block. As we concluded, today we'll generalize that idea significantly. This lesson directly addresses your goal of building a foundation for deep learning and quantum mechanics, as projections are fundamental to both fields—from data compression and model fitting to the measurement process in quantum systems.

Our learning outcome is to compute the orthogonal projection of a vector onto a subspace. We'll move from projecting onto a simple line to projecting onto higher-dimensional subspaces like planes. This concept answers a crucial question: What is the "best approximation" of a vector within a given subspace?

1. The Geometry of Projection

Imagine you're in a 3D room and you shine a flashlight directly down onto the floor. The shadow cast by an object is its projection onto the 2D subspace of the floor. The key idea is that the "projection" is the point on the floor closest to the object.

Mathematically, the orthogonal projection of a vector $b$ onto a subspace $W$ is the vector $p$ in $W$ that is closest to $b$ . This "closest vector" $p$ has a unique and critical property: the vector representing the "error" or "difference," $e = b - p$ , is orthogonal to the entire subspace $W$ .

2. From a Line to a Subspace: Building the General Formula

Let's start by formalizing the projection onto a line using matrix notation, and then we'll expand from there. Professor Gilbert Strang provides a fantastic walkthrough of this entire logical progression, from a 1D line to a general n-D subspace.

15. Projections onto Subspaces

This lecture by Professor Strang from MIT OpenCourseWare will be our guide. We'll watch it in two parts. First, he'll recap projection onto a line, introducing the idea of a projection matrix. Then, he'll generalize this to any subspace.

First, watch from 00:09 to 14:55. This part covers projection onto a 1D subspace (a line) and introduces the projection matrix for this case. Pay attention to how he uses the orthogonality condition a^T * e = 0 to derive the formula. Then, watch from 18:39 to 41:43. This is the core of our lesson. He extends the same logic to a subspace spanned by multiple vectors, which form the columns of a matrix A. The key takeaway is the derivation of the 'normal equations' A^T A x_hat = A^T B and the final formula for the projection matrix.

Let's break down and solidify the crucial formulas from that lecture.

The Core Logic

Define the Subspace: Any subspace $W$ can be defined as the column space of a matrix $A$ , where the columns of $A$ form a basis for $W$ . Let this basis be ${a_{1}, a_{2}, \dots, a_{n}}$ .
$A = ∣ a_{1} ∣ ∣ a_{2} ∣ \dots ∣ a_{n} ∣$
Define the Projection: The projection vector $p$ must be in the subspace $W$ . This means it can be written as a linear combination of the basis vectors:
$p = c_{1} a_{1} + c_{2} a_{2} + \dots + c_{n} a_{n} = A x^$
where $x^= (c_{1}, c_{2}, \dots, c_{n})^{T}$ is the vector of unknown coefficients. Our goal is to find $x^$ .
Apply the Orthogonality Condition: The error vector, $b - p$ , must be orthogonal to every vector in $W$ . It's sufficient to make it orthogonal to all the basis vectors of $W$ .
$a_{i}^{T} (b - A x^) = 0 for all i = 1, \dots, n$
As Professor Strang showed, this entire system of equations can be written in one elegant matrix equation:
$A^{T} (b - A x^) = 0$
Solve for the Coefficients: Rearranging this gives us the famous normal equations:
$A^{T} A x^= A^{T} b$
This is a standard system of linear equations for $x^$ . If the columns of $A$ are linearly independent (which they are for a basis), the matrix $A^{T} A$ is square and invertible. We can solve for $x^$ :
$x^= (A^{T} A)^{- 1} A^{T} b$
Compute the Projection: Now that we have the coefficients $x^$ , we can find the projection vector $p$ :
$p = A x^= A (A^{T} A)^{- 1} A^{T} b$

This gives us both a method and a formula. The matrix $P = A (A^{T} A)^{- 1} A^{T}$ is the projection matrix that projects any vector $b$ onto the column space of $A$ .

3. A Step-by-Step Recipe and Example

The LibreTexts resource provides a clear, step-by-step recipe for this process. This formalizes the method for computation.

6.3: Orthogonal Projection

This reading from LibreTexts provides a concise, recipe-style guide to computing orthogonal projections, along with a worked example.

Read the sections titled "Recipe: Compute an Orthogonal Decomposition" and "Corollary 1". The recipe gives you a clear algorithm. Focus on Example 11, which shows a concrete calculation for projection onto a plane in R^3.

Let's summarize the recipe for projecting $b$ onto a subspace $W$ :

Find a basis for $W$ .
Create the matrix $A$ whose columns are the basis vectors.
Compute the matrix $A^{T} A$ and the vector $A^{T} b$ .
Solve the system of equations $A^{T} A x^= A^{T} b$ to find the coefficient vector $x^$ .
The projection is $p = A x^$ .

Test your understanding!

Let $W$ be the plane in $R^{3}$ spanned by the vectors $a_{1} = 110$ and $a_{2} = 011$ . Compute the orthogonal projection of the vector $b = 600$ onto $W$ .

Show answer

Form the matrix A:
$A = 110011$
Compute $A^{T} A$ and $A^{T} b$ :
$A^{T} A = (101101) 110011 = (2112)$
$A^{T} b = (101101) 600 = (60)$
Solve for $x^$ in $A^{T} A x^= A^{T} b$ :
We need to solve $(2112) x^= (60)$ .
The inverse of $A^{T} A$ is $(A^{T} A)^{- 1} = \frac{1}{4 - 1} (2 - 1 - 1 2) = \frac{1}{3} (2 - 1 - 1 2)$ .
So, $x^= (A^{T} A)^{- 1} A^{T} b = \frac{1}{3} (2 - 1 - 1 2) (60) = \frac{1}{3} (12 - 6) = (4 - 2)$ .
Compute the projection $p = A x^$ :
$p = 110011 (4 - 2) = 42 - 2$
The orthogonal projection of $(6, 0, 0)^{T}$ onto the plane $W$ is $(4, 2, - 2)^{T}$ .

4. The Special Case: Projecting onto an Orthogonal Basis

The general formula $P = A (A^{T} A)^{- 1} A^{T}$ can be computationally intensive because of the matrix inversion. However, if our basis for the subspace $W$ is orthogonal, the process simplifies dramatically. This is why we spent time on the Gram-Schmidt process in the last lesson!

If the columns of $A$ (let's call it $Q$ for an orthogonal basis) are orthogonal, the matrix $Q^{T} Q$ becomes a simple diagonal matrix. This means the projection onto the subspace is just the sum of the individual projections onto each orthogonal basis vector.

Orthogonal Projections on Subspaces — This image illustrates how the projection of vector $\mathbf{u}$ onto a subspace $W$ spanned by an ORTHOGONAL basis $\{\mathbf{v}_1, \mathbf{v}_2\}$ is simply the sum of the individual projections onto each basis vector.

If ${u_{1}, u_{2}, \dots, u_{n}}$ is an orthogonal basis for $W$ , the projection of $b$ onto $W$ is:

p = \frac{⟨ b , u _{1} ⟩}{⟨ u _{1} , u _{1} ⟩} u_{1} + \frac{⟨ b , u _{2} ⟩}{⟨ u _{2} , u _{2} ⟩} u_{2} + \dots + \frac{⟨ b , u _{n} ⟩}{⟨ u _{n} , u _{n} ⟩} u_{n}

This formula should look familiar! It's exactly the sum of projection components we used in the Gram-Schmidt process.

If the basis is orthonormal (orthogonal and unit length), it's even simpler. In this case, $Q^{T} Q = I$ , so the projection matrix becomes $P = Q Q^{T}$ .

Conclusion

Today we have developed a complete framework for projecting a vector onto any subspace. This is a concept of immense practical and theoretical importance.

Key Takeaways:

The orthogonal projection $p$ of a vector $b$ onto a subspace $W$ is the vector in $W$ that is closest to $b$ .
The defining property is that the error vector, $b - p$ , is orthogonal to the subspace $W$ .
For a general subspace with basis matrix $A$ , the projection is calculated via the normal equations: solve $A^{T} A x^= A^{T} b$ for $x^$ , and then find $p = A x^$ .
If you have an orthogonal basis for the subspace, the projection simplifies to the sum of the individual projections onto each basis vector, avoiding a complex matrix inversion.

Preview of the Next Lesson:

We now have all the tools we need to solve one of the most common problems in science and engineering: finding the "best fit" solution to an overdetermined system of linear equations. In the next lesson, we will decompose a vector into components parallel and orthogonal to a subspace, which directly leads to the method of least squares, a cornerstone of machine learning, data analysis, and experimental physics.