Matrix Operations: Multiplication and Transpose

1. The Matrix Transpose

Let's start by demystifying the superscript 'T' we saw in the previous lesson. It stands for the transpose operation, which is a simple but essential way to reconfigure a matrix.

The transpose of a matrix $\mathbf{A}$, denoted as ${\mathbf{A}}^T$, is formed by swapping its rows and columns. If the original matrix $\mathbf{A}$ has dimensions $m\times n$, its transpose ${\mathbf{A}}^T$ will have dimensions $n\times m$. The element that was in row $i$, column $j$ of $\mathbf{A}$ moves to row $j$, column $i$ of ${\mathbf{A}}^T$.

For example, if we have a matrix $\mathbf{A}$:

Its transpose ${\mathbf{A}}^T$ is:

Now, the notation ${\vec{a}}^T\vec{b}$ from our last lesson should be clearer. In linear algebra, vectors are typically treated as column vectors (matrices with one column). The dot product of two column vectors $\vec{a}$ and $\vec{b}$ is undefined under the rules of matrix multiplication. By transposing the first vector, ${\vec{a}}^T$ becomes a row vector, making the multiplication possible.

2. Matrix Multiplication: The Mechanics

At its core, matrix multiplication is a series of dot products. To find the element in the $i$-th row and $j$-th column of the resulting matrix, you take the dot product of the $i$-th row of the first matrix with the $j$-th column of the second matrix.

The Dimension Rule

A crucial rule for matrix multiplication is that the "inner" dimensions must match. If you are multiplying matrix $\mathbf{A}$ by matrix $\mathbf{B}$ to get $\mathbf{C}=\mathbf{AB}$, and $\mathbf{A}$ is an $m\times n$ matrix, then $\mathbf{B}$ must be an $n\times p$ matrix. The resulting matrix $\mathbf{C}$ will have dimensions $m\times p$.

Rule: $(m\times \mathbf{n})\cdot (\mathbf{n}\times p)\to (m\times p)$

If the number of columns in $\mathbf{A}$ does not equal the number of rows in $\mathbf{B}$, the multiplication is undefined.

The following reading provides a fantastic visual guide to this "row-times-column" method.

3. The "Why" Behind Matrix Multiplication

Why is matrix multiplication defined in this seemingly complex way? The answer lies in its geometric meaning: matrix multiplication corresponds to the composition of linear transformations.

Imagine you apply one transformation to a vector (e.g., a rotation), and then you apply a second transformation to the result (e.g., a shear). The composition is a single new transformation that takes you from the start to the end in one step. The matrix representing this new, combined transformation is the product of the two original matrices.

This perspective is incredibly powerful and provides a deep intuition that is far more useful than just memorizing the row-by-column algorithm.

As the video masterfully explains, applying a rotation then a shear is geometrically different from applying a shear then a rotation. This is the intuitive reason why, in general, $\mathbf{AB}\ne \mathbf{BA}$. The order matters.

4. Matrix Operations in Neural Networks

Matrix multiplication is the engine of modern deep learning. A single layer of a neural network performs a linear transformation on its inputs, which is exactly what matrix multiplication does.

The calculation for a layer's output before the activation function is often written as:

Where:

• $\vec{x}$ is the vector of inputs from the previous layer.
• $\mathbf{W}$ is the weight matrix, containing the strengths of the connections. Each row of $\mathbf{W}$ corresponds to a neuron in the current layer.
• $\vec{b}$ is the bias vector, which is added after the multiplication.
• $\vec{z}$ is the resulting vector of outputs.

The StatQuest channel provides a clear, step-by-step walkthrough of this process in a simple neural network.

It's also worth noting that there is another type of matrix multiplication called the Hadamard Product (or element-wise product), denoted by $\mathbf{A}\odot \mathbf{B}$. In this case, the matrices must have the same dimensions, and the result is a new matrix where each element is simply the product of the elements in the same position. While different from standard matrix multiplication, the Hadamard product also appears in some advanced ML architectures, such as LSTMs and attention mechanisms.

5. Practice Problems

Let's put this into practice. Given the following matrices:

1. What are the dimensions of $\mathbf{A}$ and $\mathbf{B}$?
2. Is the product $\mathbf{AB}$ defined? If so, what will its dimensions be? Calculate $\mathbf{AB}$.
3. Is the product $\mathbf{BA}$ defined? If so, what will its dimensions be? Calculate $\mathbf{BA}$.
4. What is ${\mathbf{A}}^T$?

Conclusion

In this lesson, we moved from the dot product to the more general operations of matrix multiplication and transpose. These are not just arbitrary rules but the language used to describe complex transformations and computations efficiently.

Key Takeaways:

• The transpose (${\mathbf{A}}^T$) of a matrix flips its rows and columns and is often used to make dimensions compatible for multiplication.
• Matrix multiplication is a series of dot products. The element at position $(i,j)$ of the product is the dot product of row $i$ of the first matrix and column $j$ of the second.
• The deeper meaning of matrix multiplication is the composition of linear transformations.
• Matrix multiplication is not commutative ($\mathbf{AB}\ne \mathbf{BA}$), a direct consequence of the fact that the order of transformations matters.
• These operations are the computational backbone of neural networks, allowing us to express the transformation of data through a layer with the equation $\mathbf{W}\mathbf{x}+\vec{b}$.

Now that we understand how matrices act on vectors and on each other, we are ready to explore the geometric spaces they create. In our next lesson, we will delve into the concepts of linear independence and span, which will help us understand the range of all possible outputs a transformation can produce.