Chain Rule for Composite Functions

1. What are Composite Functions?

A composite function is simply a function that is nested inside another function. If we have two functions, $f(x)$ and $g(x)$, we can create a composite function by plugging the output of one into the input of the other.

For example, let $f(x)=x^2+1$ and $g(u)=u^3$. The composite function $h(x)=g(f(x))$ would be:
$h(x)=(x^2+1{)}^3$
Here, $f(x)$ is the inner function and $g(u)$ is the outer function.

You can visualize this as a sequence or a "chain" of operations, where the input flows through the first function, and its output becomes the input for the second function.

To further solidify this concept, let's review a brief article that defines composite functions and provides another example.

2. The Intuition: How a Small Nudge Propagates

Now, how do we find the derivative of such a composite function, $h(x)=g(f(x))$? We want to find $\frac{dh}{dx}$, which represents how much the final output h changes for a tiny nudge in the input x.

Since the functions are chained together, a small nudge dx first causes a change in the output of the inner function, $f(x)$. Let's call this intermediate change df. This change df then serves as the input nudge for the outer function g, which in turn causes the final change dh. The chain rule formalizes the idea that the total rate of change is the product of the rates of change at each step in this chain.

The following video from 3Blue1Brown provides a brilliant visual intuition for this propagation of changes.

As the video explained, if we call the output of the inner function $u=f(x)$, so that $h(x)=g(u)$, the chain rule can be stated as:

$\frac{dh}{dx}=\frac{dh}{du}\cdot \frac{du}{dx}$

This formula beautifully captures the intuition: the overall rate of change ($\frac{dh}{dx}$) is the rate of change of the outer function with respect to its input ($\frac{dh}{du}$), multiplied by the rate of change of the inner function with respect to its input ($\frac{du}{dx}$).

3. Applying the Chain Rule: The "Inside-Outside" Method

While the Leibniz notation ($\frac{dh}{du}$) is great for intuition, a more practical way to apply the rule is to think in terms of "inside" and "outside" functions. The rule can be stated as:

The derivative of a composite function is the derivative of the outer function (leaving the inner function untouched) multiplied by the derivative of the inner function.

Let's use this method on our example, $h(x)=(x^2+1{)}^3$.

1. Identify Outer and Inner functions:

   • Outer function: $g(u)=u^3$
   • Inner function: $f(x)=x^2+1$

2. Take the derivative of the outer function, keeping the inner function inside:

   • The derivative of $u^3$ is $3u^2$.
   • So, we get $3(x^2+1{)}^2$.

3. Take the derivative of the inner function:

   • The derivative of $x^2+1$ is $2x$.

4. Multiply the results from steps 2 and 3:
   $h^{\prime }(x)=\underset{\text{Derivative of outer}}{\underbrace{3(x^2+1{)}^2}}\cdot \underset{\text{Derivative of inner}}{\underbrace{(2x)}}=6x(x^2+1{)}^2$

The following video from StatQuest demonstrates this "stuff inside" method clearly.

This "inside-outside" process is a powerful and repeatable technique for any composite function.

4. Why the Chain Rule is Essential for Machine Learning

You might be wondering how this abstract rule connects to your goal of mastering machine learning. The connection is profound.

Neural networks are essentially giant, deeply nested composite functions.

An input vector x is passed through the first layer (a function), its output is passed through the second layer (another function), and this continues until the final layer produces a prediction. A loss function (yet another function) then compares this prediction to the true value.

To train the network, we use an algorithm called gradient descent, which needs the gradient of the loss function with respect to every weight in the network. The chain rule is the engine that powers this calculation. It allows us to calculate how a change in a weight in an early layer propagates through all the subsequent layers to affect the final loss. This process of using the chain rule to compute gradients in a neural network is famously known as backpropagation.

Conclusion

Today, we've added the final core tool to our calculus toolkit. By understanding and applying the chain rule, you've unlocked the ability to differentiate the complex, nested functions that form the bedrock of modern machine learning.

Key Takeaways:

• A composite function is a function nested within another, like $g(f(x))$.
• The chain rule provides a method for finding the derivative of a composite function by multiplying the derivatives of its constituent parts.
• In Leibniz notation: $\frac{dh}{dx}=\frac{dh}{du}\cdot \frac{du}{dx}$.
• In practice, we use the "inside-outside" method: (derivative of the outer function) $\times$ (derivative of the inner function).
• The chain rule is the mathematical foundation for backpropagation, the algorithm used to calculate gradients and train nearly all neural networks.

Preview of the Next Lesson:
We now have all the individual pieces: we can find derivatives of multivariable functions (partial derivatives) and derivatives of composite functions (chain rule). In the next lesson, we will put these together to compute gradients of vector-valued functions. This will lead directly into our lesson on Gradient Descent, where we will finally see how these gradients are used to iteratively "descend" a loss function surface and find the optimal parameters for a model.