Vector Calculus for Matrix Expressions

1. The Building Blocks: Gradients of Linear and Quadratic Forms

While the chain rule we learned is powerful, many of the gradients you'll need can be derived directly by writing out the expressions in summation notation and taking partial derivatives element by element. This "first principles" approach is excellent for building intuition and is a good skill to have.

The following document provides clear, step-by-step derivations for the gradients of two fundamental expressions: the linear form ${\mathbf{a}}^T\mathbf{w}$ and the quadratic form ${\mathbf{w}}^{T}A\mathbf{w}$.

Let's summarize these crucial results (using $\mathbf{x}$ for consistency with our course):

• Linear form: For $f(\mathbf{x})={\mathbf{a}}^T\mathbf{x}$, the gradient is $\nabla f(\mathbf{x})=\mathbf{a}$. This is the direct multi-dimensional analogue of $\frac{d}{dx}(ax)=a$.

• Quadratic form: For $f(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}$, the gradient is $\nabla f(\mathbf{x})=(A+A^T)\mathbf{x}$.

  • A very important special case is when the matrix $A$ is symmetric ($A=A^T$). In this situation, the gradient simplifies to $\nabla f(\mathbf{x})=2A\mathbf{x}$. This is the analogue of $\frac{d}{dx}(a{x}^2)=2ax$, and you will see it frequently.

These rules form the basis of our "cookbook" for matrix calculus.

2. A More Powerful Viewpoint: Calculus Without Coordinates

While the element-wise method is reliable, it can become tedious. A more elegant and often faster approach is to apply calculus rules like the product and chain rule directly to vector and matrix expressions. This "coordinate-free" method treats vectors and matrices as fundamental objects.

The key idea is to find the function's differential, $df$. For a scalar function of a vector, $f(\mathbf{x})$, the differential relates a small change in the input, $d\mathbf{x}$, to the corresponding change in the output, $df$. It is given by:

If we can manipulate our derivative expression into this form, we can simply read off the gradient $\nabla f(\mathbf{x})$.

The following video demonstrates this powerful technique.

As the video shows, for $f(\mathbf{x})={\mathbf{x}}^T\mathbf{x}$, the differential is $df=d({\mathbf{x}}^T\mathbf{x})=(d{\mathbf{x}}^T)\mathbf{x}+{\mathbf{x}}^T(d\mathbf{x})$. Since $(d{\mathbf{x}}^T)\mathbf{x}={\mathbf{x}}^T(d\mathbf{x})$, this simplifies to $df=2{\mathbf{x}}^{T}d\mathbf{x}$. Comparing this to the definition $df=\nabla f(\mathbf{x}{)}^{T}d\mathbf{x}$, we can see that $\nabla f(\mathbf{x})=2\mathbf{x}$. This matches the result from our "cookbook" rule, since ${\mathbf{x}}^T\mathbf{x}={\mathbf{x}}^{T}I\mathbf{x}$ and $I$ is symmetric.

3. The Main Event: Gradient of the Least Squares Objective

Now we'll tackle one of the most important expressions in machine learning: the least squares objective function, $f(\mathbf{x})=\Vert \mathbf{Ax}-\mathbf{b}{\Vert }^2$. Minimising this function is the goal of linear regression. We need its gradient to perform that minimisation.

We can derive this gradient in two ways: by expanding the expression and using our cookbook rules, or by applying the more general differential method.

Method 1: Expansion

First, let's expand the squared norm.

Using the property $(X-Y{)}^T=X^T-Y^T$ and distributing the terms, we get:

Note that ${\mathbf{x}}^T{A}^T\mathbf{b}$ is a scalar, so it is equal to its own transpose: ${\mathbf{x}}^T{A}^T\mathbf{b}=({\mathbf{x}}^T{A}^T\mathbf{b}{)}^T={\mathbf{b}}^{T}A\mathbf{x}$. This allows us to combine the two middle terms:

Now we can differentiate this term by term using our rules from Section 1:

1. The first term is a quadratic form ${\mathbf{x}}^{T}Q\mathbf{x}$ with $Q=A^{T}A$. Note that $Q$ is always symmetric. Therefore, $\nabla ({\mathbf{x}}^T(A^{T}A)\mathbf{x})=2(A^{T}A)\mathbf{x}$.
2. The second term is a linear form $-2({\mathbf{b}}^{T}A)\mathbf{x}$. This is ${\mathbf{c}}^T\mathbf{x}$ with $\mathbf{c}=-2A^T\mathbf{b}$. The gradient is $\mathbf{c}$, which is $-2A^T\mathbf{b}$.
3. The third term, ${\mathbf{b}}^T\mathbf{b}$, is a constant with respect to $\mathbf{x}$, so its gradient is zero.

Combining these gives the final gradient:

This derivation is valuable because it forces you to practice manipulating matrix expressions, a key skill for this course.

Method 2: The Differential Approach

The differential approach is much more direct and is what you would likely use once you are comfortable with it. The video from Prof. Edelman demonstrates this derivation beautifully.

This second method confirms our result elegantly and showcases a powerful technique for handling more complex derivatives you may encounter later on.

Conclusion

In this lesson, we developed a practical ability to differentiate some of the most common and important expressions in machine learning. While the derivations may seem purely mathematical, they are the key to unlocking the algorithms that train our models.

The key takeaways are:

• Gradient of a linear form: $\nabla ({\mathbf{a}}^T\mathbf{x})=\mathbf{a}$.
• Gradient of a quadratic form: $\nabla ({\mathbf{x}}^{T}A\mathbf{x})=(A+A^T)\mathbf{x}$, which simplifies to $2A\mathbf{x}$ when $A$ is symmetric.
• Gradient of the least squares objective: $\nabla (\Vert \mathbf{Ax}-\mathbf{b}{\Vert }^2)=2A^T(A\mathbf{x}-\mathbf{b})$.
• There are two complementary methods for these derivations: the explicit element-wise calculation and the more abstract differential or "coordinate-free" approach.

In our next lesson, we will briefly revisit Singular Value Decomposition (SVD), another cornerstone of linear algebra for machine learning. Then, we will shift our focus from pure calculus to optimisation and explore the concept of convexity. You will see why minimising a function like the least squares objective is a well-posed problem and why we can be confident in finding its single, global minimum.