SVD for Dimensionality Reduction: Truncation Explained

In the last lesson, we broke down the Singular Value Decomposition, establishing that any matrix $A$ can be factored into $A = U Σ V^{T}$ , representing a sequence of rotation, scaling, and another rotation. The crucial insight was that the singular values in the $Σ$ matrix are sorted by magnitude, $σ_{1} \geq σ_{2} \geq \dots \geq 0$ , quantifying the "importance" of different directions in the transformation.

Today, we'll build directly on that foundation to answer a key interview question: "How is SVD used for dimensionality reduction?" We will see how the ordered nature of the singular values allows us to create a simplified, lower-rank approximation of our original data matrix by strategically "truncating" the decomposition. This is one of the most powerful applications of SVD in data science and AI.

The Core Idea: Importance is Everything

Recall the geometric interpretation of SVD: it finds the directions of maximum "stretch" for a given transformation. A large singular value $σ_{i}$ means the matrix $A$ stretches vectors significantly along the corresponding singular vector direction $u_{i}$ . A small singular value means the transformation has very little effect in that direction.

This diagram shows how a linear transformation maps a unit circle to an ellipse. The singular values, $\sigma_1$ and $\sigma_2$, are the lengths of the semi-axes of the resulting ellipse, indicating the amount of stretch in the principal directions, $u_1$ and $u_2$.

This gives us a powerful idea: if a singular value is very small, perhaps the information in that direction is not very important. It might be noise or fine-grained detail we can afford to lose. Dimensionality reduction with SVD is the process of systematically discarding the directions associated with small singular values to create a "good enough" approximation of the original data.

The following text provides a concise explanation of this principle.

Geometric Interpretation of SVD

Please read the section "Visualizing the Transformation."

Focus on the final paragraph, which starts with the key insight. This connects the size of the singular values directly to the concept of retaining the most significant features of our data.

From Full Decomposition to Truncated Approximation

In the previous lesson, we saw that a matrix $A$ can be expressed as a sum of rank-1 matrices, weighted by the singular values:

A = σ_{1} u_{1} v_{1}^{T} + σ_{2} u_{2} v_{2}^{T} + \dots + σ_{r} u_{r} v_{r}^{T}

Since $σ_{1} \geq σ_{2} \geq \dots$ , the first term $σ_{1} u_{1} v_{1}^{T}$ is the most significant component of the matrix, the second term is the next most significant, and so on.

Dimensionality reduction is achieved by simply stopping this summation early. If we decide that we only need the first $k$ components to capture the essence of our data (where $k$ is less than the matrix rank $r$ ), we can create an approximation of $A$ :

A_{k} = σ_{1} u_{1} v_{1}^{T} + σ_{2} u_{2} v_{2}^{T} + \dots + σ_{k} u_{k} v_{k}^{T}

This is called a rank-k approximation. We have "truncated" the expansion after the k-th term. The following video from Luis Serrano Academy provides a great visual intuition for this. He shows how a 2D transformation that creates a very "skinny" ellipse has one large and one very small singular value. By setting the small singular value to zero, he approximates the 2D data with a 1D line, effectively reducing its dimensionality.

Singular Value Decomposition (SVD) and Image Compression

This video from Luis Serrano Academy gives a fantastic intuitive feel for truncation.

Watch the segment from dimensionality reduction, where he discusses a nearly "degenerate" matrix and how its small singular value can be zeroed out. Then, watch the next segment from the rank-1 expansion to see how this connects to the sum of rank-1 matrices we discussed above.

The Mathematics of Truncation

The sum of rank-1 matrices is equivalent to building the approximation $A_{k}$ using truncated versions of the $U$ , $Σ$ , and $V$ matrices.

$U_{k}$ is the matrix containing the first $k$ columns of $U$ .
$Σ_{k}$ is the top-left $k \times k$ square matrix of $Σ$ , containing the top $k$ singular values.
$V_{k}^{T}$ is the matrix containing the first $k$ rows of $V^{T}$ .

The approximation is then given by the product:

A_{k} = U_{k} Σ_{k} V_{k}^{T}

This is visualized perfectly in the diagram below. We start with a large $m \times n$ matrix and approximate it with the product of three much smaller matrices.

The process of SVD truncation. The original matrix A is approximated by $A_k$, which is the product of the truncated matrices $U_k$ ($m \times k$), $\Sigma_k$ ($k \times k$), and $V_k^T$ ($k \times n$). This reduces the storage and complexity required to represent the data.

Now, let's dive into a more formal reading that brings all these pieces together.

Using SVD for Dimensionality Reduction

This article provides a complete overview of the process.

First, read the introduction and the section on the core mechanism. This will formalize the truncation process we just discussed. Next, read how this reduces dimensions, connecting the matrix math to the idea of projecting data into a lower-dimensional space. Finally, read about the connection to PCA. This is a very common point of discussion in AI interviews.

The Best Possible Approximation

A critical point to make in your interview is that this truncated SVD is not just a good approximation—it's the best possible approximation. The Eckart-Young theorem formally states that for a given matrix $A$ , the matrix $A_{k}$ found by truncating the SVD is the optimal rank-k approximation. This means it minimizes the error between the original matrix and the approximation.

Steve Brunton explains this concept very clearly.

Singular Value Decomposition (SVD): Matrix Approximation

This video provides a more formal look at matrix approximation and the Eckart-Young theorem.

Please watch the section from "truncate at rank R". Focus on his explanation of the Eckart-Young theorem, which guarantees that SVD gives you the best low-rank approximation.

Crafting Your Interview Answer

Let's synthesize this into a clear, confident response for your interview.

Question: "Can you explain how SVD is used for dimensionality reduction?"

Your Answer Framework:

The Core Principle: "SVD is ideal for dimensionality reduction because it reorients the data and ranks the new dimensions by their importance. The singular values, $σ_{i}$ , measure how much variance or 'energy' is captured along each new dimension. They are sorted from largest to smallest."
The Mechanism (Truncation): "The process works by truncation. We perform the SVD to get $A = U Σ V^{T}$ . Then, we choose a smaller number of dimensions, $k$ , that we want to keep. We create an approximate matrix, $A_{k}$ , by keeping only the first $k$ singular values and their corresponding vectors in $U$ and $V$ . We essentially set all the smaller singular values to zero, discarding the information in those less important dimensions."
The Guarantee (Eckart-Young): "The power of this method is that the resulting matrix, $A_{k}$ , is guaranteed to be the best possible rank-k approximation of the original matrix. This is proven by the Eckart-Young theorem. It means we're capturing the most information possible for that chosen number of dimensions."
The Result: "In practice, if our data matrix $A$ has, say, 1000 features (columns), we can use SVD to create a new representation with only 50 features that still captures the essential structure of the original data. This is also the fundamental mechanism behind Principal Component Analysis (PCA), where the right singular vectors of the data matrix give you the principal components."

Conclusion

You have now connected the components of SVD to its most powerful application: creating an optimal, lower-dimensional representation of data. This is a cornerstone of modern data analysis, used to de-noise signals, compress data, and power recommendation engines.

Key Takeaways:

SVD enables dimensionality reduction by ranking data dimensions via singular values ( $σ_{i}$ ).
The process is called truncation: we keep the top $k$ singular values and vectors to form a rank-k approximation, $A_{k} = U_{k} Σ_{k} V_{k}^{T}$ .
The Eckart-Young theorem guarantees that $A_{k}$ is the best possible rank-k approximation of the original matrix $A$ .
This process is computationally equivalent to performing Principal Component Analysis (PCA) on centered data.

In our final lesson on linear algebra, we will look at a concrete real-world example of SVD in action—image compression—to make these abstract ideas tangible and memorable for your interview.