Backpropagation: The Chain Rule in Action

Recommended resources

The freely available resources this lesson will direct you to.

Introduction

In our previous lesson, we constructed the architecture of a feedforward neural network and detailed the process of forward propagation, the mechanism that generates a prediction from an input. We now have a network that can produce an output, but we haven't yet discussed how it learns to produce the correct output. The key to learning is to quantify the model's error using a loss function and then systematically adjust the network's parameters—its weights and biases—to minimize this error.

This lesson introduces the theoretical heart of how neural networks learn: the backpropagation algorithm. You will learn how to apply the chain rule from calculus recursively, propagating the gradient of the loss function backward through the network. This process allows us to efficiently compute the sensitivity of the loss with respect to every single weight and bias. These gradients are precisely the signals we need to update our parameters using an optimization algorithm like gradient descent, a topic we will explore in detail in the next module.

The Chain Rule on a Computational Graph

Before diving into a full neural network, let's build intuition by viewing a simple mathematical expression as a computational graph. This perspective, which should feel natural given your computer science background, breaks a complex function down into a series of simple, interconnected operations or "gates". Backpropagation is simply the process of applying the chain rule backward through this graph.

Let's consider the function $f (x, y, z) = (x + y) z$ . We can decompose this into two operations:

$q = x + y$
$f = q z$

The forward pass involves computing these values sequentially. The backward pass computes the gradients in reverse order. The CS231n notes from Stanford provide an excellent, intuitive walkthrough of this idea.

Neural Networks: Backpropagation

This article frames backpropagation as a message-passing algorithm on a "real-valued circuit," which is a very powerful mental model.

Start by reading the introduction to understand the motivation. Then, focus on the section "Compound expressions with chain rule". This section demonstrates the core mechanism using the f(x,y,z) = (x + y)z example. Note how the gradient df/dq is passed "upstream" and multiplied by the "local" gradients dq/dx and dq/dy. Finally, read the section on the intuitive understanding to solidify this concept of gates communicating with each other via gradient signals.

The diagram below, taken from the reading, visualizes this process. The forward pass (green) computes the output value. The backward pass (red) starts with the gradient of the output with respect to itself ( $\frac{\partial f}{\partial f} = 1$ ) and recursively applies the chain rule to compute the gradients for all inputs.

A computational graph for the function f(x,y,z) = (x+y)z. The forward pass computes the values in green, while the backward pass computes the gradients (the sensitivity of f to each variable) in red.

To see a narrated walkthrough, the following video explains this concept with exceptional clarity.

CS231n Winter 2016: Lecture 4: Backpropagation, Neural Networks 1

This lecture by Andrej Karpathy provides a fantastic, step-by-step explanation of the computational graph approach.

First, watch the introduction from why we need computational graphs. Then, carefully follow the detailed walkthrough of the simple (x+y)z example from the forward and backward pass. Finally, watch the summary of the general principle from how a gate operates in a backpropagation framework.

The core insight is that backpropagation is a local process. Each gate only needs to know how to compute its own output and the local derivatives of its output with respect to its inputs. During the backward pass, it receives the gradient from upstream and uses the chain rule (multiplication) to pass the appropriate gradients down to its own inputs.

Backpropagation in a Neural Network

A neural network is simply a very large, structured computational graph. The same principles apply. To make the transition, let's start with a simplified network and then build up to the full, vectorized case.

The following video from 3Blue1Brown provides a masterful, visual derivation of the calculus, starting with a network with only one neuron per layer. This removes the notational complexity of matrices and indices, allowing you to focus on the core chain rule application.

Backpropagation calculus | Deep Learning Chapter 4

This video connects the abstract computational graph to the specific structure of a neural network.

Watch the segment from the start of the derivation to the end of the bias sensitivity. Pay close attention to how the desire to compute \frac{\partial C}{\partial w^{(L)}} is broken down into three distinct partial derivatives via the chain rule. Then, watch the next segment explaining how the process iterates backward, calculating the sensitivity of the cost to the previous layer's activation. This is the "propagation" step.

As the video shows, to compute the gradient of the cost $C$ with respect to a weight $w^{(L)}$ in the final layer, we use the chain rule:

\frac{\partial C}{\partial w ^{(L)}} = \frac{\partial z ^{(L)}}{\partial w ^{(L)}} \cdot \frac{\partial a ^{(L)}}{\partial z ^{(L)}} \cdot \frac{\partial C}{\partial a ^{(L)}}

Let's analyze each term:

$\frac{\partial z ^{(L)}}{\partial w ^{(L)}} = a^{(L - 1)}$ : The impact of the weight on the pre-activation $z$ is proportional to the activation of the neuron from the previous layer.
$\frac{\partial a ^{(L)}}{\partial z ^{(L)}} = σ^{'} (z^{(L)})$ : The impact of the pre-activation on the final activation depends on the slope of the activation function at that point.
$\frac{\partial C}{\partial a ^{(L)}}$ : The impact of the final activation on the cost depends on the derivative of the loss function. For a squared error loss $(a^{(L)} - y)^{2}$ , this is $2 (a^{(L)} - y)$ .

The crucial step for recursion is computing $\frac{\partial C}{\partial a ^{(L - 1)}}$ . This reuses parts of the previous calculation and propagates the error signal backward:

\frac{\partial C}{\partial a ^{(L - 1)}} = \frac{\partial z ^{(L)}}{\partial a ^{(L - 1)}} \cdot \frac{\partial a ^{(L)}}{\partial z ^{(L)}} \cdot \frac{\partial C}{\partial a ^{(L)}}

Notice that the last two terms are the same as before. The new term, $\frac{\partial z ^{(L)}}{\partial a ^{(L - 1)}}$ , is simply the weight $w^{(L)}$ . This value, $\frac{\partial C}{\partial a ^{(L - 1)}}$ , becomes the "upstream gradient" for calculating the derivatives in layer $L - 1$ .

The Full Algorithm: A Rigorous Derivation

Now we are ready to formalize and generalize this process for a multi-layer perceptron with arbitrarily sized, fully-connected layers. This involves matrix calculus, but the core ideas remain identical. The following reading provides a rigorous, step-by-step derivation that is perfectly suited to your mathematical background.

14 Backpropagation

This text from an MIT course on computer vision provides a complete, formal treatment of backpropagation for MLPs.

First, read section 14.2, "The Trick of Backpropagation". This introduces the core idea of efficiency through dynamic programming. Next, study section 14.3, "Backward for a Generic Layer". This defines the key recurrence relation \mathbf{g}_{\texttt{in}} = \mathbf{g}_{\texttt{out}}\mathbf{L}^{\mathbf{x}}, where \mathbf{g} is the gradient of the cost and \mathbf{L}^{\mathbf{x}} is the local Jacobian. Read section 14.4, "The Full Algorithm", to see how the forward and backward passes are orchestrated. Finally, dive deep into section 14.6, "Example: Backpropagation for an MLP". This section derives the specific backward operations for a Linear layer, a pointwise non-linearity (like ReLU), and the L2 loss. Focus on the final matrix equations for the gradients.

This rigorous derivation leads to a set of update rules that are the foundation of training neural networks. Let's define $δ^{(l)} = \frac{\partial C}{\partial z ^{(l)}}$ as the "error" of the pre-activations in layer $l$ . The entire backpropagation algorithm can be summarized by four key equations:

Error at the output layer ( $L$ ):
$δ^{(L)} = \nabla_{a^{(L)}} C ⊙ σ^{'} (z^{(L)})$
This is the element-wise product of the gradient of the cost function and the derivative of the activation function.
Error at a hidden layer ( $l$ ):
$δ^{(l)} = ((W^{(l + 1)})^{T} δ^{(l + 1)}) ⊙ σ^{'} (z^{(l)})$
This is the heart of backpropagation. The error from layer $l + 1$ is propagated backward through the weights $W^{(l + 1)}$ to compute the error at layer $l$ . The image below illustrates this flow.

This diagram shows how the error gradient with respect to the output of layer j, $\frac{\partial E}{\partial z_j}$, is used to calculate the error with respect to the output of a neuron in the previous layer, $\frac{\partial E}{\partial y_i}$. This corresponds to propagating the error vector $\delta^{(l+1)}$ back through the weight matrix $W^{(l+1)}$.

Gradient with respect to the biases:
$\frac{\partial C}{\partial b ^{(l)}} = δ^{(l)}$
The gradient for the biases is simply the error term for that layer.
Gradient with respect to the weights:
$\frac{\partial C}{\partial W ^{(l)}} = δ^{(l)} (a^{(l - 1)})^{T}$
The gradient for the weights is the outer product of the error at layer $l$ and the activations from layer $l - 1$ .

A crucial implementation detail, which your CS background makes you well-positioned to appreciate, is that we rarely form the full Jacobian matrices explicitly. For element-wise operations like ReLU or sigmoid, the Jacobian is a diagonal matrix. Multiplying by a diagonal matrix is equivalent to a more efficient element-wise product. Modern deep learning libraries are heavily optimized to take advantage of this structure.

Conclusion

In this lesson, we have demystified the backpropagation algorithm, revealing it to be a clever and efficient application of the chain rule on the computational graph of a neural network. You now understand how a network can calculate the gradient of its loss function with respect to every one of its millions of parameters.

Key takeaways from this lesson are:

Computational Graphs: Viewing neural networks as computational graphs provides a powerful framework for understanding both forward and backward passes.
Chain Rule Recursion: Backpropagation is a recursive algorithm that starts from the final loss and propagates error gradients backward through the network, layer by layer.
Local vs. Upstream Gradients: Each "gate" or layer in the network computes its local gradients and combines them with the upstream gradient it receives from the layer ahead of it.
Efficiency through Dynamic Programming: The algorithm is highly efficient because it reuses computations. The error signal calculated for one layer is used directly to calculate the error for the preceding layer.
Vectorized Equations: The entire process can be expressed elegantly using matrix and vector operations, leading to the four fundamental equations of backpropagation.

We have now established how a network can compute the gradients it needs for learning. However, the process of training very deep networks is not without its perils. In our next lesson, we will explore notorious problems that can arise during training, such as vanishing and exploding gradients, and discuss their underlying causes.