Backpropagation and the Chain Rule

The Goal: Minimizing Error

At its heart, training a machine learning model is an optimization problem. We want to find the set of internal parameters (often called weights and biases) that makes the model's predictions as close as possible to the true values. We quantify this "closeness" with a loss function, a single number that measures how wrong the model is. The lower the loss, the better the model.

The challenge is to adjust potentially millions of parameters to find the combination that results in the minimum possible loss. Doing this randomly is incredibly inefficient. We need a smarter way.

Let's start by watching a video that sets up this problem beautifully using an analogy of a "curve fitter" machine with adjustable knobs.

The Tool: Derivatives and Gradients

The video hinted that the "intelligent" approach involves a property called differentiability. Since you have an engineering background, you'll recall that a derivative measures the instantaneous rate of change. If our loss function is differentiable, we can use derivatives to guide our search for the minimum.

• For a function with one parameter, the derivative at a point tells us the slope of the function. To minimize the function, we should take a small step in the direction opposite to the sign of the derivative. If the slope is positive (going up), we move left. If it's negative (going down), we move right.
• For a function with many parameters (like our model), we use the gradient. The gradient is a vector of partial derivatives; each component tells us the slope of the loss function with respect to one parameter. The gradient vector points in the direction of the steepest ascent. Therefore, to minimize the loss, we take a small step in the direction opposite to the gradient. This iterative process is called gradient descent.

Let's continue with the same video to solidify this intuition.

So, the plan is clear: calculate the gradient of the loss function with respect to all model parameters, and nudge the parameters in the opposite direction. But how do we calculate this gradient for a complex function like a neural network?

The Mechanism: The Chain Rule

A neural network is essentially a deeply nested function. The output of one layer becomes the input to the next, which in turn feeds the next, and so on, until we get a final prediction and calculate the loss.

To find the derivative of such a nested (or "composite") function, we use the chain rule. The chain rule tells us how to find the derivative of the final output with respect to an initial input by multiplying the derivatives of each intermediate step along the chain.

If we have a function $f(g(x))$, its derivative with respect to $x$ is:

You can think of this as: the change in $f$ caused by a change in $x$ is equal to (the change in $f$ caused by a change in $g$) times (the change in $g$ caused by a change in $x$).

The "connected machines" analogy in the next part of the video is a great way to visualize this.

Backpropagation: Applying the Chain Rule Backwards

We can represent the flow of calculations in a neural network as a computational graph, where each node is a simple operation (like addition or multiplication). The process of calculating the output from the inputs is called the forward pass.

To get the gradients, we use the chain rule to work our way backwards from the final loss, through all the nodes in the graph, until we reach the input parameters. This process is called backpropagation.

At each node, we calculate the local gradient (the derivative of that node's output with respect to its inputs) and multiply it by the gradient coming from the node ahead of it in the backward direction.

Let's walk through the backward pass in the image above:

1. The final output is f. The gradient of f with respect to itself, ∂f/∂f, is 1.
2. The last operation is f = q * z. The local gradients are ∂f/∂q = z and ∂f/∂z = q.
3. To get the gradient of f with respect to q, we apply the chain rule: ∂f/∂q = (∂f/∂f) * (∂f/∂q) = 1 * z = -4.
4. To get the gradient of f with respect to z, we do the same: ∂f/∂z = (∂f/∂f) * (∂f/∂z) = 1 * q = 3.
5. Now we move further back. The next node is q = x + y. The local gradients are ∂q/∂x = 1 and ∂q/∂y = 1.
6. To get the gradient of f with respect to x, we chain the gradients: ∂f/∂x = (∂f/∂q) * (∂q/∂x) = (-4) * 1 = -4.
7. Similarly for y: ∂f/∂y = (∂f/∂q) * (∂q/∂y) = (-4) * 1 = -4.

We have now successfully propagated the gradients all the way back to the inputs! This is the essence of backpropagation.

Now, let's watch the final part of the video, which puts this all together in a more complex example.

Backpropagation in Neural Network Notation

The principles we've just seen apply directly to neural networks. The language just changes slightly, using terms like activations, weights, and biases.

To connect this general intuition to the specific formulas you'll see in ML literature, we'll turn to a classic 3Blue1Brown video. It breaks down the math for a simple network. For a balanced approach, you can either watch the video or read the companion article which presents the same ideas.

If you prefer to read and digest the equations at your own pace, the article below covers the same ground.

The Secret to Efficiency: Memoization

A crucial insight, especially for someone with an engineering mindset, is that backpropagation is not just the chain rule; it's a computationally efficient way to implement it.

When we calculate the gradients backward from the output, we can store (or "memoize") the gradient at each node. When we need that value again for a path that branched off earlier, we can simply reuse the stored value instead of recomputing it. This avoids a massive amount of redundant calculation.

Therefore, you can think of the algorithm as:
Backpropagation = Chain Rule (for the logic) + Memoization (for efficiency)

This final resource explains this point very clearly.

Conclusion

Today we connected a fundamental concept from calculus to the most important algorithm in deep learning.

Key Takeaways:

• Training a model involves minimizing a loss function by adjusting its parameters.
• Gradient Descent is an iterative algorithm that does this by taking small steps in the direction opposite to the gradient of the loss function.
• The chain rule is the mathematical tool that allows us to calculate the derivative of a complex, nested function like a neural network.
• Backpropagation is a computationally efficient algorithm that applies the chain rule backward through a network's computational graph to find the gradient of the loss with respect to all parameters.
• The efficiency of backpropagation comes from memoization: caching and reusing intermediate gradient calculations as it moves backward through the network.

In our next lesson, we will continue our exploration of optimization by looking at the "landscape" of the loss functions we are trying to minimize. We will distinguish between convex and non-convex functions and explain the implications for optimization, which will help you understand some of the challenges encountered during model training.