Matrix Operations

Hello! Welcome to the fourth lesson in our course on the mathematical foundations of AI.

In our previous lessons, we built up a solid understanding of vector spaces, culminating in the crucial concepts of basis and dimension. We established that a basis provides a unique coordinate system for a vector space, and its size, the dimension, is a fundamental property of that space.

Today, we shift from the objects within these spaces (vectors) to the operators that act upon them: matrices. This lesson will cover the fundamental rules of matrix algebra. Our learning outcome is to perform matrix operations including addition, multiplication, and transposition.

While you may have encountered these operations in your practical work with NumPy, our focus here will be on the formal definitions, their algebraic properties, and the deeper geometric meaning behind them. This theoretical understanding is precisely what's needed to grasp why algorithms in machine learning are structured the way they are.

1. Matrix Addition and Scalar Multiplication

The most straightforward matrix operations are addition and scalar multiplication. They work exactly as you might expect: element by element. For two matrices to be added, they must have the exact same dimensions.

To see the formal definitions and a list of their algebraic properties, please review the following section from a set of linear algebra notes.

As you read, note that these properties (commutativity, associativity, etc.) are very similar to the properties of real number addition and multiplication. This is because these matrix operations are direct extensions of their scalar counterparts.

To summarize the properties from Theorem 9.4:

• Commutativity of Addition: $A+B=B+A$
• Associativity of Addition: $(A+B)+C=A+(B+C)$
• Distributive Properties: $\alpha (A+B)=\alpha A+\alpha B$ and $(\alpha +\beta )A=\alpha A+\beta A$

2. Matrix Multiplication: The Core Operation

Matrix multiplication is by far the most complex and powerful of these operations. It is not performed element-wise. The reason for its specific definition is that it represents something profound: the composition of linear transformations.

The Geometric "Why": Composition of Transformations

Before we dive into the formula, it's essential to understand what matrix multiplication does. In the next lesson, we will formally define linear transformations, which are functions that move vectors around in space (e.g., rotations, shears, projections). A matrix is a way to encode such a transformation.

Multiplying two matrices, $A$ and $B$, to get a new matrix $C=AB$, corresponds to finding a single transformation $C$ that has the same effect as applying transformation $B$ first, and then applying transformation $A$.

This video provides an outstanding geometric intuition for this concept.

The Formal Definition and Computational Rule

The video provides the intuition. Now, let's formalize it. The notes from MATH 233 define matrix multiplication in exactly this way—as the standard matrix of a composite mapping.

This definition leads to the standard computational rule you may have seen before: the entry in the $i$-th row and $j$-th column of the product $AB$ is the dot product of the $i$-th row of $A$ and the $j$-th column of $B$.

Let $A$ be an $m\times n$ matrix and $B$ be an $n\times p$ matrix. Their product $C=AB$ will be an $m\times p$ matrix, where each element $C_{ij}$ is given by:

$$C_{ij}=\sum _{k=1}^n{A}_{ik}{B}_{kj}$$

Notice that for the product to be defined, the number of columns in $A$ must match the number of rows in $B$.

Properties of Matrix Multiplication

Matrix multiplication has several important properties, some of which are non-intuitive.

1. Non-Commutativity: Order Matters!
In general, $AB\ne BA$. Applying a rotation then a shear is not the same as applying a shear then a rotation. The 3Blue1Brown video explains this geometrically.

2. Associativity: $(AB)C=A(BC)$
While the order of matrices cannot be changed, the order of operations doesn't matter. Thinking in terms of transformations makes this property obvious: applying C, then B, then A is the same regardless of how you group the operations.

Other key properties include distributivity over addition ($A(B+C)=AB+AC$) and the existence of an identity matrix $I$ such that $AI=IA=A$. These are summarized in Theorem 9.8 of the MATH 233 notes.

Test your understanding!

Let $A=\left(\begin{array}{cc}1& 2\\ 3& 4\end{array}\right)$ and $B=\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)$.

1. Compute $AB$.
2. Compute $BA$.
3. What do these results tell you about matrix multiplication? (Can you interpret matrix B geometrically?)

Show answer

1. Compute AB:

   $$AB=\left(\begin{array}{cc}1& 2\\ 3& 4\end{array}\right)\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)=\left(\begin{array}{cc}(1)(0)+(2)(1)& (1)(1)+(2)(0)\\ (3)(0)+(4)(1)& (3)(1)+(4)(0)\end{array}\right)=\left(\begin{array}{cc}2& 1\\ 4& 3\end{array}\right)$$

2. Compute BA:

   $$BA=\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)\left(\begin{array}{cc}1& 2\\ 3& 4\end{array}\right)=\left(\begin{array}{cc}(0)(1)+(1)(3)& (0)(2)+(1)(4)\\ (1)(1)+(0)(3)& (1)(2)+(0)(4)\end{array}\right)=\left(\begin{array}{cc}3& 4\\ 1& 2\end{array}\right)$$

3. Interpretation:
   Clearly, $AB\ne BA$, confirming that matrix multiplication is not commutative.

   Geometrically, matrix $B$ swaps the basis vectors $\mathbf{i}$ and $\mathbf{j}$. Applying it to a vector $(x,y)$ results in $(y,x)$ — it reflects the vector across the line $y=x$.

   • $AB$ represents first swapping the columns of $A$.
   • $BA$ represents first swapping the rows of $A$.
     These are different operations, leading to different results.

3. The Matrix Transpose

The final fundamental operation is the transpose, which flips a matrix along its main diagonal, turning its rows into columns and vice-versa. We denote the transpose of $A$ as $A^T$.

Definition and Basic Properties

If $A$ is an $m\times n$ matrix, then $A^T$ is an $n\times m$ matrix where $(A^T{)}_{ij}=A_{ji}$.

The simple properties are:

1. $(A^T{)}^T=A$
2. $(A+B{)}^T=A^T+B^T$
3. $(\alpha A{)}^T=\alpha A^T$

The Transpose of a Product: A Crucial Identity

The most important and non-obvious property concerns the transpose of a product:
$(AB{)}^T=B^T{A}^T$
Note the reversal of order. This identity is extremely important and appears constantly in machine learning derivations, such as the normal equations for linear regression. Given your goal of understanding the theory with mathematical rigor, watching its proof is essential.

This proof, which manipulates indices and summations, is a great example of the mathematical machinery used to formalize linear algebra.

Conclusion

In this lesson, we have formally defined the fundamental operations of matrix algebra. More importantly, we've explored the "why" behind them.

Key Takeaways:

• Matrix Addition and Scalar Multiplication are element-wise operations that follow familiar algebraic rules like commutativity and associativity.
• Matrix Multiplication is defined to represent the composition of linear transformations. Its row-by-column computational rule is a consequence of this definition.
• Matrix multiplication is associative but not commutative. The order of application matters.
• The Transpose of a matrix flips its rows and columns.
• A critical identity to remember and understand is the transpose of a product: $(AB{)}^T=B^T{A}^T$, where the order of the matrices is reversed.

The concept of matrix multiplication as a representation of composite transformations serves as a direct bridge to our next topic. In the upcoming lesson, we will formally define linear transformations, explore their properties, and solidify the connection between the abstract transformation and its concrete matrix representation.