Multivariable Chain Rule for Composite Functions

Recommended resources

The freely available resources this lesson will direct you to.

Welcome to our next lesson. In our previous session, we defined the gradient, $\nabla f$ , and established its fundamental property: it is a vector that points in the direction of the steepest ascent of a function. This gave us a clear strategy for optimization—to find a minimum, we simply need to move in the direction of the negative gradient.

However, the functions encountered in deep learning are not simple, clean expressions. They are massive, deeply nested compositions, where the parameters we need to optimize (like the weights of the first layer) are buried under dozens or even hundreds of subsequent operations. A direct application of our partial derivative rules is intractable. This lesson introduces the critical mechanism for navigating these complex structures: the multivariable chain rule. It provides a systematic, divide-and-conquer strategy for calculating how a change in any parameter, no matter how deeply nested, propagates through the entire system to affect the final output.

The Core Idea: Propagating Change

At its heart, the chain rule is about understanding how change in one variable propagates through a sequence of dependent variables. Let's start with the simplest multivariable case, which you might encounter in physics or graphics when tracking an object's path through a field.

Imagine a function $F (x, y)$ that describes the temperature at any point $(x, y)$ on a 2D plate. Now, imagine a particle moving along a path on this plate, where its position at any time $t$ is given by $(x (t), y (t))$ . If we want to know how the temperature experienced by the particle changes over time, we need to find the derivative of the composite function $F (x (t), y (t))$ with respect to $t$ .

A change in $t$ doesn't affect $F$ directly. Instead, a change in $t$ causes a change in $x$ and a change in $y$ . These changes, in turn, cause a change in $F$ . The total change in $F$ is the sum of the changes propagated through each path.

To visualize and formalize this, we can use a dependency diagram. The video below introduces this concept and derives the chain rule for this scenario.

The Multi-Variable Chain Rule: Derivatives of Compositions

This video, "The Multi-Variable Chain Rule" from Dr. Trefor Bazett, provides an excellent, structured explanation using dependency diagrams.

Please watch the segments from dependency diagrams to see how we map out these relationships, and then from the derivation, where the chain rule formula is built from this diagram. Pay close attention to the distinction between the full derivative symbol d and the partial derivative symbol \partial.

As the video explains, the relationship is captured by the following formula:

\frac{d F}{d t} = \frac{\partial F}{\partial x} \frac{d x}{d t} + \frac{\partial F}{\partial y} \frac{d y}{d t}

Notice the notation. We use the full derivative $\frac{d F}{d t}$ because, from the perspective of $t$ , the composite function is ultimately a function of a single variable. However, we use partial derivatives like $\frac{\partial F}{\partial x}$ because $F$ itself has multiple inputs, and we are considering the change with respect to only one of them at a time. Each term in the sum represents the contribution from one of the dependency paths:

$\frac{\partial F}{\partial x} \frac{d x}{d t}$ : How much a change in $t$ affects $F$ through its effect on $x$ .
$\frac{\partial F}{\partial y} \frac{d y}{d t}$ : How much a change in $t$ affects $F$ through its effect on $y$ .

The Leibniz notation makes this intuitive. It looks as though the $d x$ and $d y$ terms "cancel out", leaving you with two contributions to $\frac{d F}{d t}$ . While not technically a cancellation, it's a powerful mnemonic that reflects the underlying structure of how these rates of change are composed.

Generalizing the Chain Rule

Now, let's extend this to the more general case where the intermediate variables are themselves functions of multiple variables. Suppose we have a function $f (x, y)$ , where $x = x (s, t)$ and $y = y (s, t)$ . This structure is common in computer graphics, for instance, when mapping a 2D texture (with coordinates $s, t$ ) onto a 3D surface (where point coordinates are given by $x$ and $y$ ).

We now have two independent variables, $s$ and $t$ , and can ask how $f$ changes with respect to each of them. The principle remains the same: sum the contributions from all paths.

A dependency diagram showing a function f that depends on intermediate variables x and y, which in turn depend on independent variables s and t. The formulas for the partial derivatives of f with respect to s and t are derived by summing the influences along the paths.

To find the rate of change of $f$ with respect to $s$ , we trace all paths from $s$ to $f$ :

Path 1: $s \to x \to f$
Path 2: $s \to y \to f$

Summing their contributions gives the chain rule for $\frac{\partial f}{\partial s}$ :

\frac{\partial f}{\partial s} = \frac{\partial f}{\partial x} \frac{\partial x}{\partial s} + \frac{\partial f}{\partial y} \frac{\partial y}{\partial s}

Similarly, the rate of change of $f$ with respect to $t$ is:

\frac{\partial f}{\partial t} = \frac{\partial f}{\partial x} \frac{\partial x}{\partial t} + \frac{\partial f}{\partial y} \frac{\partial y}{\partial t}

Notice that all derivatives are now partial derivatives because every function in the chain has multiple inputs.

The Chain Rule as a Computational Process

For deep learning, it's most useful to think of the chain rule not just as a formula, but as a computational algorithm applied to a graph. Any mathematical expression, no matter how complex, can be decomposed into a sequence of simple operations, forming what's called a computational graph.

The chain rule gives us a way to compute the derivative of the graph's final output with respect to any node, including the initial inputs. This process is the essence of backpropagation.

Let's consider the following computational graph for the function $o = (x y \cdot y z)^{2}$ .

A computational graph visualizing the function $o = (h_1 h_2)^2$ where $h_1=xy$ and $h_2=yz$. The gray boxes are inputs, blue boxes are intermediate operations, and the orange box is the final output. The numbers show the values for a 'forward pass' with x=3, y=2, z=1.

To compute the derivative of the output $o$ with respect to an input, say $y$ , we must account for all paths through which $y$ influences $o$ .

Path 1: $y \to h_{1} \to h \to o$
Path 2: $y \to h_{2} \to h \to o$

When a variable influences the output through multiple paths, we must use the total derivative, which sums the contributions from each path. The following article provides a clear, step-by-step breakdown of this exact process.

The Matrix Calculus You Need For Deep Learning

This article from explained.ai gives a rigorous yet practical guide to the calculus needed for deep learning. It brilliantly frames the chain rule as an algorithmic process.

Please read the section titled "Single-variable total-derivative chain rule". This section is key because it explicitly addresses how to handle a variable (like y in our graph) that influences the output through multiple branches. Focus on how it uses a dataflow diagram and formulates the derivative as a sum of all contributing paths.

Now, let's apply this principle to find $\frac{\partial o}{\partial y}$ for our graph. The total derivative rule tells us:

\frac{\partial o}{\partial y} = \frac{\partial o}{\partial h} (\frac{\partial h}{\partial h _{1}} \frac{\partial h _{1}}{\partial y} + \frac{\partial h}{\partial h _{2}} \frac{\partial h _{2}}{\partial y})

This formula represents the total influence of $y$ on $h$ , which is then propagated to $o$ . Let's compute each component derivative from our graph's operations:

$\frac{\partial o}{\partial h} = \frac{\partial}{\partial h} (h^{2}) = 2 h$
$\frac{\partial h}{\partial h _{1}} = \frac{\partial}{\partial h _{1}} (h_{1} h_{2}) = h_{2}$
$\frac{\partial h _{1}}{\partial y} = \frac{\partial}{\partial y} (x y) = x$
$\frac{\partial h}{\partial h _{2}} = \frac{\partial}{\partial h _{2}} (h_{1} h_{2}) = h_{1}$
$\frac{\partial h _{2}}{\partial y} = \frac{\partial}{\partial y} (y z) = z$

Now, substitute these back into the formula:

\frac{\partial o}{\partial y} = 2 h (h_{2} \cdot x + h_{1} \cdot z)

This expression gives us the derivative symbolically. We can also evaluate it numerically using the values from the forward pass: $x = 3, y = 2, z = 1$ , which gives $h_{1} = 6, h_{2} = 2, h = 12$ .

\frac{\partial o}{\partial y} = 2 (12) \cdot (2 \cdot 3 + 6 \cdot 1) = 24 \cdot (6 + 6) = 24 \cdot 12 = 288

This systematic, step-by-step process of breaking a complex function into a graph and chaining local derivatives is exactly how automatic differentiation engines in frameworks like PyTorch and TensorFlow work. Understanding this process from first principles is fundamental to understanding how neural networks are trained.

Conclusion

In this lesson, we have demystified the process of finding derivatives for complex, composite functions—the kind that define neural networks.

Key Takeaways:

The multivariable chain rule is a method for computing the derivative of a composite function.
It works by tracing all paths of influence from an independent variable to the final output in a dependency diagram or computational graph.
The total rate of change is the sum of the rates of change propagated along each distinct path.
This "divide and conquer" approach allows us to compute gradients for arbitrarily complex functions by chaining together simple, local derivatives. This is the core mechanism of backpropagation.

We have seen how to apply the chain rule to scalar outputs. But neural networks often have multiple outputs (e.g., logits for each class in a classification problem). This requires us to compute the derivatives of a vector of functions with respect to a vector of inputs.

In our next lesson, we will generalize our work by organizing these partial derivatives into the Jacobian matrix. This will give us the full linear algebraic picture of the chain rule and set the stage for deriving the backpropagation algorithm.