Subspaces of the Weight Matrix

The Four Fundamental Subspaces

For any matrix $A\in {\mathbb{R}}^{m\times n}$, there are four associated subspaces that define its behavior as a linear map from ${\mathbb{R}}^n$ to ${\mathbb{R}}^m$. We've met two; let's now meet all four.

1. Column Space, $C(A)$: The span of the columns of $A$. A subspace of the output space, ${\mathbb{R}}^m$.
2. Null Space, $N(A)$: The set of all vectors $\mathbf{x}\in {\mathbb{R}}^n$ such that $A\mathbf{x}=0$. A subspace of the input space, ${\mathbb{R}}^n$.
3. Row Space, $C(A^T)$: The span of the rows of $A$. Since rows are conventionally vectors, we can think of this as the column space of the transposed matrix, $A^T$. It is a subspace of the input space, ${\mathbb{R}}^n$.
4. Left Null Space, $N(A^T)$: The null space of the transposed matrix $A^T$. It contains all vectors $\mathbf{y}\in {\mathbb{R}}^m$ such that $A^T\mathbf{y}=0$, which is equivalent to ${\mathbf{y}}^{T}A=0^T$. It's a subspace of the output space, ${\mathbb{R}}^m$.

Professor Gilbert Strang's lectures provide a classic, step-by-step introduction to these four spaces using a concrete example. His intuitive approach is perfect for establishing the definitions.

The Big Picture: Orthogonality and Decomposition

The true power of this framework comes from the relationships between the subspaces. The four spaces form two pairs of orthogonal complements.

• The row space $C(A^T)$ is orthogonal to the null space $N(A)$. Together, they span the entire input space ${\mathbb{R}}^n$.
• The column space $C(A)$ is orthogonal to the left null space $N(A^T)$. Together, they span the entire output space ${\mathbb{R}}^m$.

This structure has a profound implication. Any vector $\mathbf{x}$ in the input space can be uniquely decomposed into two orthogonal components: one in the row space (${\mathbf{x}}_{row}$) and one in the null space (${\mathbf{x}}_{null}$).
$\mathbf{x}={\mathbf{x}}_{row}+{\mathbf{x}}_{null}$
When we apply the transformation $A$, look what happens:
$A\mathbf{x}=A({\mathbf{x}}_{row}+{\mathbf{x}}_{null})=A{\mathbf{x}}_{row}+A{\mathbf{x}}_{null}$
By definition, $A{\mathbf{x}}_{null}=0$, so the equation simplifies to:
$A\mathbf{x}=A{\mathbf{x}}_{row}$
This is a critical insight: the matrix transformation $A$ only acts on the component of the input that lies in its row space. The null space component is completely annihilated. The row space can be seen as the domain of "effective inputs."

The following short video provides an excellent visualization of this decomposition and mapping.

Finding the Subspaces: The Role of SVD

This theoretical framework is beautiful, but how do we find orthonormal bases for these subspaces in practice? The answer lies in the Singular Value Decomposition (SVD).

As a brief refresher, the SVD of any matrix $A\in {\mathbb{R}}^{m\times n}$ is a factorization of the form:
$A=U\Sigma V^T$
Where:

• $U$ is an $m\times m$ orthogonal matrix whose columns are the left singular vectors.
• $\Sigma$ is an $m\times n$ diagonal matrix containing the non-negative singular values.
• $V$ is an $n\times n$ orthogonal matrix whose columns are the right singular vectors.

Geometrically, this factorization decomposes any linear transformation into a sequence of three fundamental operations: a rotation ($V^T$), a scaling along axes ($\Sigma$), and another rotation ($U$).

The true magic is that the matrices $U$ and $V$ directly provide orthonormal bases for the four fundamental subspaces. The following resource explains this connection with remarkable clarity.

Let's consolidate the key result from the reading. If the rank of $A$ is $r$, we can partition the columns of $U$ and $V$ accordingly:

• The first $r$ columns of $V$, denoted $V_L$, form an orthonormal basis for the row space of $A$.
• The remaining $n-r$ columns of $V$, denoted $V_R$, form an orthonormal basis for the null space of $A$.
• The first $r$ columns of $U$, denoted $U_L$, form an orthonormal basis for the column space of $A$.
• The remaining $m-r$ columns of $U$, denoted $U_R$, form an orthonormal basis for the left null space of $A$.

The SVD doesn't just tell us these subspaces exist; it hands us a perfectly structured, orthogonal basis for each one. This makes it an exceptionally powerful tool for analysis.

Information Flow in a Neural Network Layer

Let's apply this complete picture to a linear layer with weight matrix $W\in {\mathbb{R}}^{m\times n}$, mapping from an $n$-dimensional input space to an $m$-dimensional output space.

• Row Space $C(W^T)$: This is the subspace of inputs that the layer is "sensitive" to. Any component of an input vector $\mathbf{x}$ that is orthogonal to this space (i.e., in the null space) will be annihilated. Thus, $C(W^T)$ is the effective input space. A low-rank weight matrix implies that the layer is only paying attention to a small subspace of its possible inputs.

• Null Space $N(W)$: This is the subspace of "invisible" inputs. Any patterns or features described by vectors in $N(W)$ are completely discarded. This is the source of information loss or compression. In a layer like nn.Linear(1024, 128), the null space has a dimension of at least $1024-128=896$. The layer is explicitly designed to ignore most of the input space.

• Column Space $C(W)$: As we knew, this is the activation space—the range of possible outputs. Any output from this layer must lie in this subspace. If the rank $r<m$, the layer cannot produce any vector with a component outside of this $r$-dimensional subspace, creating an information bottleneck.

• Left Null Space $N(W^T)$: This is the "unreachable" subspace in the output domain. It consists of all directions in the output space that the layer can never produce, no matter the input. If $\mathbf{y}\in N(W^T)$, then for any $\mathbf{x}$, the output $W\mathbf{x}$ will be orthogonal to $\mathbf{y}$. This describes the built-in constraints on the layer's output geometry. For an autoencoder's decoder, this would represent shapes or patterns that the network is fundamentally incapable of generating.

By analyzing the four fundamental subspaces of a weight matrix, you gain a complete geometric understanding of how it transforms information—what it preserves, what it discards, what it can produce, and what it cannot.

Conclusion

Today, we've assembled the complete picture of how a matrix transforms a vector space.

Key Takeaways:

• Any matrix $A$ has four fundamental subspaces: the row space, null space, column space, and left null space.
• These subspaces form two pairs of orthogonal complements, which decompose the input and output spaces entirely: ${\mathbb{R}}^n=C(A^T)\oplus N(A)$ and ${\mathbb{R}}^m=C(A)\oplus N(A^T)$.
• The transformation $A\mathbf{x}$ annihilates the null space component of $\mathbf{x}$ and maps the row space component of $\mathbf{x}$ to the column space.
• The Singular Value Decomposition (SVD), $A=U\Sigma V^T$, provides orthonormal bases for all four subspaces through the columns of $U$ and $V$.
• In a neural network, these subspaces define the information flow: the row space is the effective input, the null space is information loss, the column space is the achievable output, and the left null space represents unreachable outputs.

This understanding of the complete structure of a linear map is not just theoretical. In our next lesson, we will see how we can exploit this structure by using the SVD to create low-rank approximations of matrices, a technique with direct connections to modern model compression methods like LoRA.