Matrix Multiplication: Linear Maps, Outer Products, and Neural Networks

Matrix Multiplication as Composition of Linear Maps

At its heart, a matrix is a way to represent a linear transformation—a function that takes a vector as input and produces a vector as output, all while keeping the grid lines of the space parallel and evenly spaced. Multiplying a vector by a matrix is equivalent to applying that transformation to the vector.

When we have a sequence of such transformations, like in a multi-layer perceptron (MLP), we apply one after another. The combined effect of this sequence is itself a single, composite linear transformation. The matrix representing this composite transformation is precisely the product of the individual matrices.

To build a strong visual intuition for this, the following video from the "Essence of linear algebra" series by 3Blue1Brown is an excellent starting point.

As the video demonstrates, if we have a transformation $T_1$ represented by matrix $M_1$ and a transformation $T_2$ represented by $M_2$, applying $T_1$ and then $T_2$ to a vector $v$ is computed as $M_2(M_{1}v)$. Because matrix multiplication is associative, this is equivalent to $(M_2{M}_1)v$. The product $M_{comp}=M_2{M}_1$ is a single matrix that captures the entire composite transformation.

This is exactly what happens in a neural network. A sequence of layers, each with its own weight matrix, composes a series of transformations. For a simple two-layer linear network, the output $y$ is a function of the input $x$:

The product of the weight matrices $W_2{W}_1$ defines the single, equivalent linear map from the input space directly to the output space.

The following StatQuest video provides a concrete, practical walkthrough of these matrix operations within a simple neural network, connecting the math directly to the structure of nn.Linear in PyTorch.

Visualizing Compositions in Neural Networks

This compositional view gives us a powerful geometric handle on network architecture. For example, a standard feed-forward network (FFN) block in a Transformer often expands the dimension of the data, applies a nonlinearity, and then projects it back down. This can be seen as a sequence of matrix multiplications, e.g., $x_{\text{out}}=(x_{\text{in}}{W}_{\text{up}})W_{\text{down}}$. The shapes of these matrices determine whether information is being expanded into a higher-dimensional space or compressed through a bottleneck.

The PyTorch blog post "Inside the Matrix" offers a unique 3D visualization for these compositions.

Matrix Multiplication as a Sum of Outer Products

A second, equally important interpretation of matrix multiplication is as a sum of outer products. The outer product of a column vector $u\in {\mathbb{R}}^m$ and a row vector $v^T\in {\mathbb{R}}^{1\times n}$ is the matrix $u{v}^T$, whose entries are $(u{v}^T{)}_{ij}=u_i{v}_j$. This results in an $m\times n$ matrix. A key property is that this matrix always has a rank of 1.

The product of two matrices $C=AB$ can be expressed as a sum of outer products. If we let $a_k$ be the $k$-th column of $A$ and $b_k^T$ be the $k$-th row of $B$, then:

where $n$ is the common dimension (number of columns in $A$ and rows in $B$). Each term in the sum is a rank-1 matrix, and we are adding them up to form the final matrix $C$.

This perspective is crucial for understanding low-rank approximations. If we can approximate a large matrix $W$ with the product of two "thin" matrices, $B$ and $A$, where $W\approx BA$, we are essentially saying that $W$ can be well-approximated by a short sum of outer products. This is the central idea behind Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning technique for large language models.

LoRA freezes the massive pre-trained weights of a model and learns only the low-rank factors of the change in weights. This dramatically reduces the number of trainable parameters.

Unifying the Views

These two viewpoints—composition of maps and sum of outer products—are simply different ways of decomposing the same underlying computation. The conventional "dot product" algorithm, where you compute each element of the result matrix $C_{ij}$ by taking the dot product of the $i$-th row of $A$ and the $j$-th column of $B$, is a third way.

The "Inside the Matrix" article proposes a unified geometric picture that holds all these views.

Conclusion

In this lesson, we established a geometric foundation for understanding matrix multiplication, the cornerstone of deep learning computations.

Key Takeaways:

• Composition of Linear Maps: Multiplying matrices $W_2{W}_1$ is equivalent to applying transformation $W_1$ followed by $W_2$. This helps us reason about the flow of information through sequential layers in a neural network.
• Sum of Outer Products: A matrix product $AB$ can be seen as $\sum a_k{b}_k^T$. This view is critical for understanding low-rank structures and parameter-efficient techniques like LoRA.
• A Unified View: These different interpretations are just different ways to parse the same fundamental operation, which can be visualized as a 3D computation involving the matrix dimensions.

Your familiarity with the PyTorch stack and mathematical background will allow you to fluidly switch between these interpretations depending on the context—from analyzing the architecture of a network to implementing efficient training strategies.

In the next lesson, we will build directly on this geometric foundation. We will use the concepts of vector subspaces, span, basis, and dimension to more formally describe the "spaces" that weight matrices create and where activations live. This will give us the tools to analyze information flow, bottlenecks, and the expressive capacity of neural network layers.