Computing the Jacobian Matrix

Recommended resources

The freely available resources this lesson will direct you to.

Welcome back. In our previous lesson, we established how the multivariable chain rule allows us to calculate the derivative of a final scalar output with respect to any parameter in a complex computational graph. This is the heart of how we determine the influence of a single weight on a final loss value.

However, modern systems rarely have just one output. A transformation in computer graphics might map 2D texture coordinates to a 3D surface point. A layer in a neural network takes a vector of inputs and produces a vector of outputs (activations). We need to generalize our calculus from scalar-valued functions $f : R^{n} \to R$ to vector-valued functions $f : R^{n} \to R^{m}$ .

This lesson introduces the Jacobian matrix, the natural generalization of the gradient for these vector-valued functions. You will learn how to compute it and, more importantly, understand its role as the best linear approximation of a complex transformation at a specific point—a concept fundamental to both graphics and the mechanics of deep learning.

From Gradient to Jacobian

Recall that for a scalar-valued function $f (x_{1}, \dots, x_{n})$ , the gradient $\nabla f$ is a row vector containing all its first-order partial derivatives:

\nabla f = [\frac{\partial f}{\partial x _{1}} \frac{\partial f}{\partial x _{2}} \dots \frac{\partial f}{\partial x _{n}}]

It tells us how the single output $f$ changes in response to infinitesimal changes in each input variable.

Now, consider a vector-valued function $f$ that maps from $R^{n}$ to $R^{m}$ . This function is really a stack of $m$ scalar-valued functions, which we call component functions:

f (x) = f_{1} (x) f_{2} (x) ⋮ f_{m} (x)

Here, $x$ is the input vector $(x_{1}, \dots, x_{n})$ .

Before I give you the definition, take a moment to reason from first principles. If we want to capture all the first-order derivative information of this system, we need to know how each of the $m$ outputs changes with respect to each of the $n$ inputs.

How many partial derivatives would this entail, and what is the most logical structure to organize them?

You would correctly conclude that there are $m \times n$ such partial derivatives. The most natural way to organize them is in an $m \times n$ matrix. This matrix is the Jacobian matrix, denoted as $J_{f}$ or $D f$ . Each row of the Jacobian is simply the gradient of the corresponding component function:

J_{f} = \nabla f_{1} \nabla f_{2} ⋮ \nabla f_{m} = \frac{\partial f _{1}}{\partial x _{1}} \frac{\partial f _{2}}{\partial x _{1}} ⋮ \frac{\partial f _{m}}{\partial x _{1}} \frac{\partial f _{1}}{\partial x _{2}} \frac{\partial f _{2}}{\partial x _{2}} ⋮ \frac{\partial f _{m}}{\partial x _{2}} \dots \dots ⋱ \dots \frac{\partial f _{1}}{\partial x _{n}} \frac{\partial f _{2}}{\partial x _{n}} ⋮ \frac{\partial f _{m}}{\partial x _{n}}

The following article provides a clear, textual summary of this construction, starting from the gradient of a scalar function and building up to the full Jacobian matrix.

The Jacobian Matrix: Introducing Vector Calculus - Programmathically

This article from Programmathically provides a concise definition of the Jacobian matrix, relating it directly to the gradient.

Please read the introductory section and then skip to the section titled "Jacobian Matrix". Focus on how the article defines the gradient as the Jacobian for a scalar-valued function in the "Gradient Definition" part, and then see how it generalizes this concept to a full matrix for vector-valued functions under the "Jacobian Matrix" section.

The Geometric Interpretation: Local Linearization

The Jacobian is far more than just a notational convenience for storing derivatives. Its profound importance comes from its geometric meaning: the Jacobian matrix of a function $f$ at a point $x_{0}$ defines the best linear transformation that approximates $f$ near $x_{0}$ .

Any complex, non-linear transformation—like warping an image or passing data through a neural network layer—behaves like a simple linear transformation if you zoom in far enough on a single point. The Jacobian is the matrix of that linear transformation.

This idea is illustrated perfectly in the following diagram. The top shows a non-linear function f distorting a grid. The bottom shows how the Jacobian J at a specific point provides a linear approximation (a shear and scale) of that distortion locally.

The following video from Khan Academy provides a dynamic and intuitive explanation of this exact concept. It's one of the clearest explanations of what the Jacobian truly represents.

The Jacobian matrix

This video, "The Jacobian matrix" by Khan Academy, builds the geometric intuition for why the Jacobian is structured the way it is.

Please watch from the beginning until the formal definition. Pay close attention to how a tiny step in an input direction (e.g., along the x-axis) results in a new vector after the transformation. The video shows how the components of this new vector are determined by partial derivatives and form the first column of the Jacobian. The same logic applies to a step in the y-direction for the second column. This directly connects the partial derivatives to the basis vectors of the local linear transformation.

This interpretation is crucial. It means that to understand how a small input vector $dx$ is transformed into a small output vector $df$ , we can use a simple matrix-vector multiplication:

df \approx J_{f} (x_{0}) \cdot dx

This is the multivariable, vector-output equivalent of the single-variable approximation $df \approx f^{'} (x_{0}) d x$ .

Computing the Jacobian

Let's solidify this with a concrete computation. Consider a function $f : R^{2} \to R^{2}$ that maps polar coordinates $(r, θ)$ to Cartesian coordinates $(x, y)$ .
$f (r, θ) = [x (r, θ) y (r, θ)] = [r cos θ r sin θ]$
To find the Jacobian matrix, we follow the definition step-by-step:

Identify component functions:
- $f_{1} (r, θ) = r cos θ$
- $f_{2} (r, θ) = r sin θ$
Compute the first row (gradient of $f_{1}$ ):
- $\frac{\partial f _{1}}{\partial r} = cos θ$
- $\frac{\partial f _{1}}{\partial θ} = - r sin θ$
Compute the second row (gradient of $f_{2}$ ):
- $\frac{\partial f _{2}}{\partial r} = sin θ$
- $\frac{\partial f _{2}}{\partial θ} = r cos θ$
Assemble the Jacobian matrix: $J_{f} (r, θ) = [\frac{\partial f _{1}}{\partial r} \frac{\partial f _{2}}{\partial r} \frac{\partial f _{1}}{\partial θ} \frac{\partial f _{2}}{\partial θ}] = [cos θ sin θ - r sin θ r cos θ]$

This Jacobian tells you how a small change in radius $d r$ and angle $d θ$ will translate to a change in position $d x$ and $d y$ .

To get some hands-on practice, work through the following interactive exercises. They will guide you through the process of identifying component functions, taking their partial derivatives, and assembling the final Jacobian matrix.

Vector-valued functions and the Jacobian matrix - Math Insight

This interactive page from Math Insight provides excellent practice for the mechanics of computing Jacobians.

Start by reviewing the examples of component functions. Then, focus on the exercises that ask you to compute partial derivatives for a given function \mathbf{f}. Finally, use those results to assemble the Jacobian matrix in the subsequent exercises.

The Chain Rule, Revisited with Jacobians

The true power of the Jacobian becomes apparent when we revisit the chain rule. In the last lesson, we saw that calculating the total derivative involved summing up contributions from different paths. With Jacobians, this composition becomes elegant matrix multiplication.

Consider two functions composed together: $y = f (x)$ and $z = g (y)$ . The chain rule states that the Jacobian of the composite function $h (x) = g (f (x))$ is the product of the individual Jacobians:

J_{h} (x) = J_{g} (f (x)) \cdot J_{f} (x)

This is the mathematical engine of backpropagation. Each layer of a neural network is a vector-valued function. The process of calculating the gradient of the final loss with respect to the first layer's weights involves multiplying the Jacobians of all intermediate layers.

The following video provides a superb walkthrough of this exact application.

The Jacobian : Data Science Basics

The video "The Jacobian : Data Science Basics" by ritvikmath offers a clear, practical example of how Jacobians are used.

First, watch the section from the explicit definition for a concise summary of how to build the Jacobian for a 2x2 case. Then, watch the crucial application segment from the neural network example. This will connect everything we've discussed today directly to the process of training a neural network, showing how the chain rule becomes a simple product of Jacobian matrices.

Conclusion

In this lesson, we have moved from the gradient of a scalar function to the Jacobian matrix of a vector function. This is a critical step in building the mathematical machinery for deep learning and advanced computer graphics.

Key Takeaways:

The Jacobian matrix is the generalization of the gradient for vector-valued functions $f : R^{n} \to R^{m}$ . It is an $m \times n$ matrix where each row is the gradient of a component function.
Geometrically, the Jacobian at a point defines the local linear approximation of the function, describing how it scales, rotates, and shears an infinitesimal region of the input space.
For a composition of functions, the chain rule is elegantly expressed as the product of their Jacobian matrices. This is the formal basis for the backpropagation algorithm used to train neural networks.

We have now covered first-order derivatives extensively. However, the derivative only tells us the direction of steepest ascent. It doesn't tell us anything about the shape or curvature of the function's landscape. To understand that, we need to go to the second derivative. In our next lesson, we will introduce the Hessian matrix, the second-order equivalent of the Jacobian, which will allow us to classify critical points and analyze the convexity of our loss functions.