Gradients of Scalar-Valued Functions (Vector & Matrix)

1. The Gradient: Derivatives with Respect to Vectors

In deep learning, we typically optimize a scalar loss function, $J$, which is a function of potentially millions of parameters (weights and biases) arranged in vectors and matrices. To use gradient descent, we need to compute how a small change in each parameter affects the final loss. This is captured by the gradient.

For a scalar-valued function of a vector, $f:{\mathbb{R}}^n\to \mathbb{R}$, the gradient is the vector containing all the partial derivatives of $f$ with respect to each element of the input vector $\mathbf{x}$.

A crucial, and often confusing, point is how these partial derivatives are arranged into a vector. There are two common conventions, known as layout conventions.

To summarize the conventions from the reading:
Let $f:{\mathbb{R}}^n\to \mathbb{R}$ and $\mathbf{x}\in {\mathbb{R}}^n$.

• Numerator Layout: The gradient ${\nabla }_{\mathbf{x}}f$ is a row vector of size $1\times n$.

  $${\nabla }_{\mathbf{x}}f=\frac{\partial f}{\partial \mathbf{x}}=\left[\begin{array}{cccc}\frac{\partial f}{\partial x_1}& \frac{\partial f}{\partial x_2}& \cdots & \frac{\partial f}{\partial x_n}\end{array}\right]$$

• Denominator Layout: The gradient ${\nabla }_{\mathbf{x}}f$ is a column vector of size $n\times 1$.

  $${\nabla }_{\mathbf{x}}f=\frac{\partial f}{\partial \mathbf{x}}=\left[\begin{array}{c}\frac{\partial f}{\partial x_1}\\ \frac{\partial f}{\partial x_2}\\ ⋮\\ \frac{\partial f}{\partial x_n}\end{array}\right]$$

Neither convention is universally "correct," but different fields and authors have strong preferences. Physics and mathematics often prefer the denominator layout, while some machine learning resources, like the article you just read, use the numerator layout. Deep learning frameworks like PyTorch and TensorFlow handle this ambiguity internally, but when you're deriving gradients by hand or reading papers, it's critical to identify which convention is in use.

The following video provides a concise recap and uses the denominator layout, offering a direct contrast to the article.

For the rest of this lesson, we will derive gradients using a method that makes the layout explicit at the end, and we will state the result in both conventions.

2. A Coordinate-Free Approach to Gradients

While you can always compute a gradient by grinding through the partial derivatives for each element, this becomes tedious and error-prone for complex functions. A more elegant and powerful method, particularly suited to your background, is to work with vectors and matrices holistically. This approach, championed by Alan Edelman in his MIT lectures, treats the derivative as the best linear approximation to a function's change.

The core idea is this: for a small perturbation $d\mathbf{x}$ to a vector $\mathbf{x}$, the change in the function $f(\mathbf{x})$, denoted $df$, is approximated by the inner product of the gradient with the perturbation:

In the denominator layout (gradient is a column vector), this is the dot product $(d\mathbf{x}{)}^T(\nabla f)$. In the numerator layout (gradient is a row vector), this is $(\nabla f)(d\mathbf{x})$. We can write this more symmetrically as $df=({\nabla }_{\mathbf{x}}f{)}^{T}d\mathbf{x}$ if the gradient is a column, or $df=({\nabla }_{\mathbf{x}}f)d\mathbf{x}$ if the gradient is a row. The strategy is to find an expression for $df$ and then rearrange it into one of these forms to identify the gradient.

Let's see this in action.

Let's formalize the derivation from the video. We want to find the gradient of $f(\mathbf{x})={\mathbf{x}}^T\mathbf{x}$.

1. Find the differential $df$: $$df=d({\mathbf{x}}^T\mathbf{x})$$
2. Apply the product rule, $d(uv)=(du)v+u(dv)$: $$df=(d{\mathbf{x}}^T)\mathbf{x}+{\mathbf{x}}^T(d\mathbf{x})$$
3. Recognize that each term is a scalar (a $1\times 1$ matrix). A scalar is equal to its own transpose. So, we can transpose the first term: $$(d{\mathbf{x}}^T)\mathbf{x}=((d{\mathbf{x}}^T)\mathbf{x}{)}^T={\mathbf{x}}^T(d{\mathbf{x}}^T{)}^T={\mathbf{x}}^{T}d\mathbf{x}$$ As Edelman points out, this is because the dot product is commutative.
4. Substitute this back into the expression for $df$: $$df={\mathbf{x}}^{T}d\mathbf{x}+{\mathbf{x}}^{T}d\mathbf{x}=2{\mathbf{x}}^{T}d\mathbf{x}$$
5. Now, we match this to our target forms:
   • Numerator Layout: We are looking for $df=(\nabla f)d\mathbf{x}$. Comparing this with $df=(2{\mathbf{x}}^T)d\mathbf{x}$, we identify the gradient as the row vector $\nabla f=2{\mathbf{x}}^T$.
   • Denominator Layout: We are looking for $df=(d\mathbf{x}{)}^T(\nabla f)$. We can write $df=(d\mathbf{x}{)}^T(2\mathbf{x})$, so we identify the gradient as the column vector $\nabla f=2\mathbf{x}$.

This method is powerful because it relies on algebraic manipulation of vectors and matrices, which is much cleaner than element-wise calculus for complex expressions.

3. Gradients of Common Machine Learning Functions

Let's apply this technique to a few more examples that are central to machine learning.

A. Least Squares Loss

A fundamental problem in ML is minimizing the squared error, $f(\mathbf{x})=\Vert \mathbf{Ax}-\mathbf{b}{\Vert }^2$. This is the loss function for linear regression. Let's find its gradient with respect to $\mathbf{x}$.

The derivation is as follows:

1. Rewrite the norm: $f(\mathbf{x})=(\mathbf{Ax}-\mathbf{b}{)}^T(\mathbf{Ax}-\mathbf{b})$.
2. Apply the product rule: $df=d((\mathbf{Ax}-\mathbf{b}{)}^T)(\mathbf{Ax}-\mathbf{b})+(\mathbf{Ax}-\mathbf{b}{)}^{T}d(\mathbf{Ax}-\mathbf{b})$.
3. The differential of $(\mathbf{Ax}-\mathbf{b})$ is simply $\mathbf{A}d\mathbf{x}$, since $\mathbf{A}$ and $\mathbf{b}$ are constants.
4. Substitute this in: $df=(\mathbf{A}d\mathbf{x}{)}^T(\mathbf{Ax}-\mathbf{b})+(\mathbf{Ax}-\mathbf{b}{)}^T(\mathbf{A}d\mathbf{x})$.
5. Simplify the transposed term: $df=(d\mathbf{x}{)}^T{\mathbf{A}}^T(\mathbf{Ax}-\mathbf{b})+(\mathbf{Ax}-\mathbf{b}{)}^T\mathbf{A}d\mathbf{x}$.
6. Again, both terms are scalars, so they are equal. $$df=2(\mathbf{Ax}-\mathbf{b}{)}^T\mathbf{A}d\mathbf{x}$$
7. Identify the gradients:
   • Numerator Layout: $\nabla f=2(\mathbf{Ax}-\mathbf{b}{)}^T\mathbf{A}$
   • Denominator Layout: $\nabla f=(2(\mathbf{Ax}-\mathbf{b}{)}^T\mathbf{A}{)}^T=2{\mathbf{A}}^T(\mathbf{Ax}-\mathbf{b})$

Setting the gradient to zero, $2{\mathbf{A}}^T(\mathbf{Ax}-\mathbf{b})=0$, gives the famous normal equations ${\mathbf{A}}^T\mathbf{Ax}={\mathbf{A}}^T\mathbf{b}$ for the least-squares solution.

B. Gradient with Respect to a Matrix

The same principle extends to finding the gradient of a scalar function with respect to a matrix, $f:{\mathbb{R}}^{m\times n}\to \mathbb{R}$. Here, the gradient ${\nabla }_{W}f$ is a matrix of the same dimensions as $W$, where each element is $\frac{\partial f}{\partial W_{ij}}$.

The linearized change $df$ is now expressed using the Frobenius inner product:

Our goal is to manipulate $df$ into this trace form to identify ${\nabla }_{W}f$.

Let's follow his derivation for $f(W)=\text{tr}(W^2)$.

1. Find the differential: $df=d(\text{tr}(W^2))$.
2. Use linearity of trace and product rule: $df=\text{tr}(d(W^2))=\text{tr}((dW)W+W(dW))$.
3. Use linearity again: $df=\text{tr}((dW)W)+\text{tr}(W(dW))$.
4. Use the cyclic property of the trace on the first term: $\text{tr}((dW)W)=\text{tr}(W(dW))$.
5. Combine terms: $df=2\text{tr}(W(dW))$.
6. To match the target form $\text{tr}(G^{T}dW)$, we rewrite $W$ as $(W^T{)}^T$. $$df=\text{tr}((2W^T{)}^{T}dW)$$
7. By inspection, we identify the gradient: ${\nabla }_{W}f=2W^T$.

This method, combining differentials with the product rule and trace properties, is a remarkably efficient way to derive gradients for complex matrix expressions, which are ubiquitous in deep learning.

4. Back to the Computation Graph

Let's connect this back to a simple neural network. Consider the forward pass for a single neuron, which is essentially logistic regression.

Our goal in training is to compute ${\nabla }_{w}J$ and ${\nabla }_{b}J$ to update the parameters. The tools we've developed in this lesson are the first step. For example, if the loss function is Mean Squared Error, $J=\frac{1}{m}\sum (a_i-y_i{)}^2$, this is a scalar function of the vector $\mathbf{a}$. We can now compute $\frac{\partial J}{\partial \mathbf{a}}$ using the techniques from today. The next lesson will show how to use the chain rule to propagate this gradient further backward to get the gradients with respect to $w$ and $b$.

Conclusion

In this lesson, we established the fundamental tools of vector calculus needed for deep learning. You've learned how to formalize the concept of a derivative with respect to a vector or matrix.

Key Takeaways:

• The gradient of a scalar function $f(\mathbf{x})$ is the collection of its partial derivatives, $\frac{\partial f}{\partial x_i}$.
• Layout conventions are critical: the gradient can be a row vector (numerator layout) or a column vector (denominator layout). They are transposes of each other, and you must be consistent.
• A powerful, coordinate-free method for finding gradients is to compute the differential $df$ and manipulate it into the form $df=\text{tr}(G^{T}dX)$. The matrix $G$ is then the gradient ${\nabla }_{X}f$.
• This method relies on the product rule for vectors/matrices and the linearity and cyclic properties of the trace.

We now have the ability to differentiate the final stage of a computation graph—the loss function. In the next lesson, we will introduce the Jacobian and the chain rule, which will allow us to combine these derivatives and compute gradients for arbitrarily deep composite functions, forming the basis of backpropagation.