SVD for Matrix Analysis and Subspace Geometry

In our previous lesson, we successfully derived the Singular Value Decomposition (SVD), establishing that any $m \times n$ matrix $A$ can be factored into $A = U Σ V^{T}$ . We saw how this decomposition arises from the eigendecomposition of the symmetric matrices $A^{T} A$ and $A A^{T}$ .

Now that we have this powerful tool in hand, we can move beyond its derivation and explore its applications. This lesson will focus on using the SVD to reveal the fundamental properties of a matrix. You will learn how the SVD provides a complete, robust, and geometrically intuitive picture of a matrix's rank, its numerical stability via the condition number, and the structure of the four fundamental subspaces. This is akin to having a universal diagnostic tool that gives a full report on any linear transformation.

SVD and the Four Fundamental Subspaces

In our first module, we analyzed the four fundamental subspaces: the column space $C (A)$ , the nullspace $N (A)$ , the row space $C (A^{T})$ , and the left nullspace $N (A^{T})$ . The SVD provides a remarkably elegant description of these spaces by giving an orthonormal basis for each one.

Recall the decomposition $A = U Σ V^{T}$ , where $U$ is an $m \times m$ orthogonal matrix, $V$ is an $n \times n$ orthogonal matrix, and $Σ$ is an $m \times n$ diagonal matrix of singular values $σ_{1} \geq σ_{2} \geq \dots \geq σ_{r} > 0$ , with $r$ being the rank of $A$ .

The columns of $U$ and $V$ (the singular vectors) are precisely the basis vectors we need:

Column Space, $C (A)$ : The first $r$ columns of $U$ , ${u_{1}, \dots, u_{r}}$ , form an orthonormal basis.
Left Nullspace, $N (A^{T})$ : The remaining $m - r$ columns of $U$ , ${u_{r + 1}, \dots, u_{m}}$ , form an orthonormal basis.
Row Space, $C (A^{T})$ : The first $r$ columns of $V$ , ${v_{1}, \dots, v_{r}}$ , form an orthonormal basis.
Nullspace, $N (A)$ : The remaining $n - r$ columns of $V$ , ${v_{r + 1}, \dots, v_{n}}$ , form an orthonormal basis.

This clean separation is one of the most profound results in linear algebra. The following notes from MIT provide a concise statement of this structure.

Chapter 7 The Singular Value Decomposition (SVD)

This document provides a clear and structured overview of the SVD. We'll start by focusing on its connection to the four fundamental subspaces.

Please read the first page of the notes, focusing on the section titled "Bases and Matrices in the SVD". Pay close attention to how the vectors u_i and v_i are partitioned based on the rank r to form bases for the four subspaces.

For a quick visual recap of this concept, the following video segment is very effective.

Further Matrix Decompositions: LU, Cholesky, QR, and SVD

This video from "Professor Dave Explains" gives a rapid, high-level summary of how to read the bases for the four fundamental subspaces directly from the SVD components.

Watch the segment from four fundamental subspaces. This will reinforce the mappings between the columns of U and V and the specific subspaces.

Rank, Geometry, and Effective Rank

The connection between SVD and the fundamental subspaces makes determining the rank of a matrix straightforward: the rank $r$ is simply the number of non-zero singular values.

In practical data science applications, however, matrices are often contaminated with noise. A matrix that is theoretically rank-deficient might appear to be full-rank numerically due to small perturbations. The SVD gives us a way to handle this. The singular values are ordered by magnitude, $σ_{1} \geq σ_{2} \geq \dots \geq σ_{r} > 0$ . If there is a large drop-off in the magnitude of the singular values, for example, $σ_{k} ≫ σ_{k + 1} \approx 0$ , we can define an effective rank of $k$ . This is the idea that the matrix is "close" to a rank- $k$ matrix, a concept we will explore fully in the next lesson on low-rank approximation.

Geometrically, the singular values and vectors tell the full story of the linear transformation. As we saw previously, the action of $A$ on the unit sphere in $R^{n}$ can be visualized.

The transformation $M$ is decomposed into a rotation $V^*$, scaling $\Sigma$, and another rotation $U$. The right singular vectors (in $V$) are rotated into the left singular vectors (in $U$), and their lengths are scaled by the singular values $\sigma_i$.

The SVD formalizes this: $A$ maps the orthonormal basis of right singular vectors ${v_{i}}$ to an ellipsoid in $R^{m}$ whose semi-axes are the vectors ${σ_{i} u_{i}}$ . The singular values dictate the lengths of these axes.

The following reading provides more detail on this geometric picture and the notion of effective rank.

A Singularly Valuable Decomposition: The SVD of a Matrix

This paper by Dan Kalman provides an excellent and readable exploration of the SVD's properties. We'll look at two sections.

First, read the section titled "A geometric interpretation" (page 5). This formalizes the idea that A maps the unit sphere to an ellipsoid. Next, read the subsection within "Applications in Brief" (page 8) that discusses rank. Focus on the paragraph beginning with "As a second application", which introduces the concept of "effective rank" in the presence of noise.

Condition Number and Numerical Stability

One of the most important practical applications of SVD is in diagnosing the numerical sensitivity of a matrix. The condition number of a matrix $A$ , denoted $κ (A)$ , is defined as the ratio of its largest singular value to its smallest non-zero singular value:

κ (A) = \frac{σ _{1}}{σ _{r}}

A matrix with a large condition number is called ill-conditioned. This means that small changes in the input vector $x$ can lead to large changes in the output vector $b$ when solving $A x = b$ . From a deep learning perspective, this is highly relevant to understanding optimization landscapes. An ill-conditioned Hessian matrix, for example, corresponds to a loss surface that is extremely steep in some directions and very flat in others, making optimization difficult.

The SVD provides the most numerically stable way to compute the condition number and solve linear systems. A common alternative is to use the normal equations to solve the least-squares problem $min_{x} ∥ A x - b ∥^{2}$ , which involves computing $(A^{T} A)^{- 1}$ . While theoretically sound, this approach is numerically inferior.

Why? Let's analyze the condition number of $A^{T} A$ . The eigenvalues of $A^{T} A$ are $σ_{i}^{2}$ . Therefore, the condition number of $A^{T} A$ is:

κ (A^{T} A) = \frac{λ _{m a x}}{λ _{m i n}} = \frac{σ _{1}^{2}}{σ _{r}^{2}} = (\frac{σ _{1}}{σ _{r}})^{2} = (κ (A))^{2}

The condition number is squared! If $κ (A) = 1000$ , which is not uncommon, then $κ (A^{T} A) = 1 0^{6}$ . This squaring of the condition number can lead to a significant loss of numerical precision when working with finite-precision floating-point numbers. Explicitly forming $A^{T} A$ is often a source of numerical instability, a fact well-known in numerical linear algebra but sometimes overlooked in machine learning contexts.

The following reading provides an excellent, detailed analysis of this issue. Given your background, the connection between floating-point arithmetic and the magnitude of eigenvalues will be very familiar.

A Singularly Valuable Decomposition: The SVD of a Matrix

This final reading delves into the practical consequences of ill-conditioning and why the SVD is superior to the normal equations for solving least-squares problems.

Please read the entire section starting from the heading "ATA" on page 11 through the end of the subsequent section on page 14. This is a masterful explanation of why \kappa(A^TA) = (\kappa(A))^2 is so problematic in practice. Pay close attention to the example with floating-point arithmetic; it clearly illustrates how digits of precision are lost.

Conclusion

In this lesson, we have seen how the SVD is much more than just a matrix factorization; it is a complete analytical tool. By decomposing a matrix $A$ into $U Σ V^{T}$ , we can immediately understand its most important properties.

Key Takeaways:

The singular vectors in $U$ and $V$ provide orthonormal bases for the four fundamental subspaces.
The rank of a matrix is equal to the number of its non-zero singular values. The concept of "effective rank" extends this to noisy, real-world data.
The condition number, $κ (A) = σ_{1} / σ_{r}$ , is a crucial measure of a matrix's sensitivity to perturbations and its numerical stability.
Solving problems using the SVD of $A$ is numerically superior to methods involving the formation of $A^{T} A$ , as the latter squares the condition number, potentially leading to catastrophic loss of precision.

We have now established what the SVD is and how to use it for analysis. In our next lesson, we will build on the idea of effective rank and explore how to use a truncated SVD to find the best low-rank approximation of a matrix. This is not just a theoretical curiosity; it is the mathematical foundation for powerful techniques in model compression and data analysis, including a precursor to methods like LoRA (Low-Rank Adaptation) used to fine-tune large language models.

Recommended resources

SVD and the Four Fundamental Subspaces

Rank, Geometry, and Effective Rank

Condition Number and Numerical Stability

Conclusion