Backpropagation: Recursive Chain Rule for Neural Networks

Introduction

In our last lesson, we successfully derived the gradient of the loss function with respect to the final layer's logits, ${\nabla }_{{\mathbf{z}}^{(L)}}\mathcal{L}$. For the common case of softmax activation and cross-entropy loss, this gave us the elegant result $\mathbf{p}-\mathbf{y}$. This gradient is the initial "error signal" that tells our network how wrong its final output was.

Now, we will tackle the central algorithm of neural network training: backpropagation. Our goal is to derive a method to take this initial error signal and propagate it backward through every layer of the network, layer by layer, to compute the gradient of the loss function with respect to every weight and bias. This is the ultimate application of the multivariable chain rule you've mastered. Backpropagation isn't an optimization algorithm itself; rather, it's a highly efficient computational procedure for calculating the gradients that optimizers like SGD need to update the model's parameters.

Backpropagation as Flow on a Computational Graph

Before diving into the matrix-vector calculus, it's incredibly insightful to visualize a neural network as a computational graph. Your background in computer science makes this a natural starting point. In this view, every mathematical operation—a matrix multiply, a vector addition, an activation function—is a "node" or "gate" in a directed acyclic graph.

The forward pass involves feeding inputs ($\mathbf{x}$, weights $\mathbf{W}$, biases $\mathbf{b}$) into the graph and flowing them through the gates to compute the final loss, $\mathcal{L}$. The backward pass, or backpropagation, is the process of starting from the final loss and flowing gradients backward through the graph.

At each gate, we perform a simple, local calculation. The gate receives a gradient from "upstream" (closer to the loss function) and uses its own local gradient to compute the gradients for its inputs, which it then passes "downstream" (closer to the network inputs). The chain rule is the mechanism that governs this flow, telling us that to find the gradient of a downstream variable, we multiply the upstream gradient by the local gradient.

To build a solid intuition for this process, let's watch a segment from Andrej Karpathy's renowned lecture on this topic. He masterfully illustrates this concept with a simple scalar-valued graph.

The Four Fundamental Equations of Backpropagation

With the computational graph intuition in place, let's formalize this for a multi-layer neural network. We will derive four fundamental equations that constitute the backpropagation algorithm.

First, let's establish our notation for an arbitrary layer $l$:

• ${\mathbf{W}}^{(l)}$ and ${\mathbf{b}}^{(l)}$: The weight matrix and bias vector for layer $l$.
• ${\mathbf{a}}^{(l-1)}$: The activation vector from the previous layer, which is the input to layer $l$.
• ${\mathbf{z}}^{(l)}$: The pre-activation (or "logit") vector for layer $l$, computed as ${\mathbf{z}}^{(l)}={\mathbf{W}}^{(l)}{\mathbf{a}}^{(l-1)}+{\mathbf{b}}^{(l)}$.
• $g(\cdot )$: The element-wise activation function (e.g., ReLU, sigmoid).
• ${\mathbf{a}}^{(l)}$: The activation vector for layer $l$, computed as ${\mathbf{a}}^{(l)}=g({\mathbf{z}}^{(l)})$.
• $\mathcal{L}$: The scalar loss function at the end of the network.

Our core quantity of interest is the error ${\delta }^{(l)}$ at each layer $l$, which we define as the gradient of the loss with respect to the pre-activation vector ${\mathbf{z}}^{(l)}$:

$${\delta }^{(l)}\equiv \frac{\partial \mathcal{L}}{\partial {\mathbf{z}}^{(l)}}$$

This vector tells us how the final loss is influenced by the logits at layer $l$. If we can compute ${\delta }^{(l)}$ for every layer, we can easily find the gradients for the weights and biases.

We will now derive the four equations that allow us to do this. The following resource provides an exceptionally detailed, first-principles derivation, which we will use as our primary guide.

1. Error in the Output Layer (${\delta }^{(L)}$)

This is the starting point of the backward pass. The error ${\delta }^{(L)}$ at the final layer $L$ is given by:

$${\delta }^{(L)}=\frac{\partial \mathcal{L}}{\partial {\mathbf{a}}^{(L)}}\odot g^{\prime }({\mathbf{z}}^{(L)})$$

where $\odot$ denotes element-wise (Hadamard) product.

As we discovered in the previous lesson, for the most common classification setup (Softmax activation and Cross-Entropy loss), this equation has a beautifully simple form:

$${\delta }^{(L)}={\mathbf{a}}^{(L)}-\mathbf{y}$$

Where ${\mathbf{a}}^{(L)}$ is the vector of predicted probabilities ($\mathbf{p}$) and $\mathbf{y}$ is the one-hot encoded true label vector. This is our initial "prediction - label" error.

2. Error Propagation (${\delta }^{(l)}$ in terms of ${\delta }^{(l+1)}$)

This is the recursive heart of backpropagation. It tells us how to compute the error at layer $l$ once we know the error at layer $l+1$.

Question: Using the chain rule, how can we express ${\delta }^{(l)}=\frac{\partial \mathcal{L}}{\partial {\mathbf{z}}^{(l)}}$ in terms of ${\delta }^{(l+1)}=\frac{\partial \mathcal{L}}{\partial {\mathbf{z}}^{(l+1)}}$? What intermediate variable connects ${\mathbf{z}}^{(l)}$ to ${\mathbf{z}}^{(l+1)}$?

Answer

The variable connecting them is the activation of layer $l$, $\mathbf{a}^{(l)}$. The full chain is $\mathcal{L} \rightarrow \mathbf{z}^{(l+1)} \rightarrow \mathbf{a}^{(l)} \rightarrow \mathbf{z}^{(l)}$. Therefore, the chain rule gives: $$ \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(l)}} = \left(\frac{\partial \mathbf{z}^{(l+1)}}{\partial \mathbf{a}^{(l)}}\right)^T \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(l+1)}} \odot \frac{\partial \mathbf{a}^{(l)}}{\partial \mathbf{z}^{(l)}} $$ Note the transpose on the Jacobian. In element-wise form, this is: $$ \delta_j^{(l)} = \frac{\partial \mathcal{L}}{\partial z_j^{(l)}} = \sum_k \frac{\partial \mathcal{L}}{\partial z_k^{(l+1)}} \frac{\partial z_k^{(l+1)}}{\partial a_j^{(l)}} \frac{\partial a_j^{(l)}}{\partial z_j^{(l)}} $$

Let's evaluate the pieces.

• $\frac{\partial \mathcal{L}}{\partial z_k^{(l+1)}}={\delta }_k^{(l+1)}$ (by definition).
• $\frac{\partial a_j^{(l)}}{\partial z_j^{(l)}}=g^{\prime }(z_j^{(l)})$ (derivative of the activation function).
• Since ${\mathbf{z}}^{(l+1)}={\mathbf{W}}^{(l+1)}{\mathbf{a}}^{(l)}+{\mathbf{b}}^{(l+1)}$, the partial derivative $\frac{\partial z_k^{(l+1)}}{\partial a_j^{(l)}}$ is simply the weight $W_{kj}^{(l+1)}$.

Putting this together in matrix form yields the second fundamental equation:

$${\delta }^{(l)}=(({\mathbf{W}}^{(l+1)}{)}^T{\delta }^{(l+1)})\odot g^{\prime }({\mathbf{z}}^{(l)})$$

This equation shows how the error signal ${\delta }^{(l+1)}$ is transformed by the transpose of the weights of the next layer, and then modulated by the local gradient of the activation function.

To see this derivation with all the intermediate Jacobians, please review the article.

This recursive relationship is visualized below. The error from a later layer is passed backward through the transposed weights to compute the error for the current layer.

3. Gradient with respect to Biases (${\nabla }_{\mathbf{b}}\mathcal{L}$)

Once we have the error ${\delta }^{(l)}$ for a layer, finding the gradient for its bias vector is straightforward.

Question: Given that ${\mathbf{z}}^{(l)}={\mathbf{W}}^{(l)}{\mathbf{a}}^{(l-1)}+{\mathbf{b}}^{(l)}$ and ${\delta }^{(l)}=\frac{\partial \mathcal{L}}{\partial {\mathbf{z}}^{(l)}}$, use the chain rule to find an expression for $\frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{(l)}}$. What is the Jacobian $\frac{\partial {\mathbf{z}}^{(l)}}{\partial {\mathbf{b}}^{(l)}}$?

Answer

The chain rule gives $\frac{\partial \mathcal{L}}{\partial \mathbf{b}^{(l)}} = \left(\frac{\partial \mathbf{z}^{(l)}}{\partial \mathbf{b}^{(l)}}\right)^T \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(l)}}$. Since a change in $\mathbf{b}^{(l)}$ causes an identical change in $\mathbf{z}^{(l)}$ (it's just an addition), the Jacobian $\frac{\partial \mathbf{z}^{(l)}}{\partial \mathbf{b}^{(l)}}$ is the identity matrix, $\mathbf{I}$. Therefore, $\frac{\partial \mathcal{L}}{\partial \mathbf{b}^{(l)}} = \mathbf{I}^T \delta^{(l)} = \delta^{(l)}$.

This gives us our third fundamental equation:

$$\frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{(l)}}={\delta }^{(l)}$$

The gradient for the biases is simply the error at that layer.

4. Gradient with respect to Weights (${\nabla }_{\mathbf{W}}\mathcal{L}$)

Finally, we find the gradient for the weight matrix ${\mathbf{W}}^{(l)}$. This gradient tells us how to adjust the connection strengths. Similar to the bias, we apply the chain rule:

$$\frac{\partial \mathcal{L}}{\partial W_{jk}^{(l)}}=\frac{\partial \mathcal{L}}{\partial z_j^{(l)}}\frac{\partial z_j^{(l)}}{\partial W_{jk}^{(l)}}$$

We know $\frac{\partial \mathcal{L}}{\partial z_j^{(l)}}={\delta }_j^{(l)}$. From the definition ${\mathbf{z}}^{(l)}={\mathbf{W}}^{(l)}{\mathbf{a}}^{(l-1)}+{\mathbf{b}}^{(l)}$, we can see that $z_j^{(l)}={\sum }_k{W}_{jk}^{(l)}{a}_k^{(l-1)}+b_j^{(l)}$. The partial derivative $\frac{\partial z_j^{(l)}}{\partial W_{jk}^{(l)}}$ is simply $a_k^{(l-1)}$.

Thus, $\frac{\partial \mathcal{L}}{\partial W_{jk}^{(l)}}={\delta }_j^{(l)}{a}_k^{(l-1)}$. When we assemble this for all $j$ and $k$, we get the fourth fundamental equation in matrix form:

$$\frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{(l)}}={\delta }^{(l)}({\mathbf{a}}^{(l-1)}{)}^T$$

This is an outer product between the error vector ${\delta }^{(l)}$ and the activation vector from the previous layer $({\mathbf{a}}^{(l-1)}{)}^T$. Intuitively, the change to a weight is proportional to the error at its destination neuron and the activation of its source neuron. This is a mathematical formalization of Hebbian learning ("neurons that fire together, wire together").

The LINK resource derives this result in the sections you've just reviewed.

The Backpropagation Algorithm Summarized

With these four equations, we can now state the full algorithm.

1. Input: A set of training examples. For each example $\mathbf{x}$:
2. Forward Pass: Set ${\mathbf{a}}^{(0)}=\mathbf{x}$. For each layer $l=1,2,...,L$, compute:
   • ${\mathbf{z}}^{(l)}={\mathbf{W}}^{(l)}{\mathbf{a}}^{(l-1)}+{\mathbf{b}}^{(l)}$
   • ${\mathbf{a}}^{(l)}=g({\mathbf{z}}^{(l)})$
   • (During this pass, you must cache the ${\mathbf{z}}^{(l)}$ and ${\mathbf{a}}^{(l-1)}$ values for use in the backward pass. This is a key point for implementation and relates to your interest in systems performance, as it implies a memory cost.)
3. Backward Pass:
   • Compute the output error ${\delta }^{(L)}$ using Equation 1.
   • For $l=L-1,L-2,...,1$, propagate the error backward by computing ${\delta }^{(l)}$ using Equation 2.
   • Compute the desired gradients for all layers using Equations 3 and 4:
     • ${\nabla }_{{\mathbf{b}}^{(l)}}\mathcal{L}={\delta }^{(l)}$
     • ${\nabla }_{{\mathbf{W}}^{(l)}}\mathcal{L}={\delta }^{(l)}({\mathbf{a}}^{(l-1)}{)}^T$
4. Gradient Descent Step: Average the gradients over a mini-batch of examples and use an optimizer (like SGD or Adam) to update all ${\mathbf{W}}^{(l)}$ and ${\mathbf{b}}^{(l)}$.

Conclusion

You have now derived the complete backpropagation algorithm from first principles. By viewing the network as a computational graph and recursively applying the chain rule, we arrived at four simple, powerful equations that allow us to efficiently compute the gradient of the loss with respect to every parameter in the network.

Key Takeaways:

• Backpropagation is an algorithm for computing gradients by recursively applying the chain rule.
• It can be understood as passing error signals backward through a computational graph.
• The error ${\delta }^{(l)}$ is the core quantity, representing the gradient of the loss with respect to a layer's logits.
• The four fundamental equations provide a complete recipe for computing all necessary gradients:
  1. ${\delta }^{(L)}={\mathbf{a}}^{(L)}-\mathbf{y}$ (for Softmax/CE)
  2. ${\delta }^{(l)}=(({\mathbf{W}}^{(l+1)}{)}^T{\delta }^{(l+1)})\odot g^{\prime }({\mathbf{z}}^{(l)})$
  3. $\frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{(l)}}={\delta }^{(l)}$
  4. $\frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{(l)}}={\delta }^{(l)}({\mathbf{a}}^{(l-1)}{)}^T$

Preview of the Next Lesson:

Look closely at Equation 2. The error signal is repeatedly multiplied by weight matrices as it propagates backward ($...{\mathbf{W}}^{(l+2)T}{\mathbf{W}}^{(l+1)T}...$). If the values in these matrices are consistently large or small, what might happen to the magnitude of the gradient in the early layers of a deep network? This line of inquiry leads directly to the notorious problems of exploding and vanishing gradients, which we will dissect in our next lesson.