Backpropagation with Chain Rule: Two-Layer Network

1. The Big Picture: Computational Graphs and the Chain Rule

At its core, a neural network is a large, nested mathematical function. The forward pass is simply the evaluation of this function. To figure out how to adjust the parameters (weights and biases) to reduce the error, we need to calculate the derivative of the loss function with respect to each of these parameters. Doing this for a deep network with millions of parameters seems daunting, but backpropagation is a clever algorithm that makes it efficient.

Backpropagation is essentially an organized application of the chain rule from calculus, applied backward through the network's structure, which can be visualized as a computational graph.

To build a strong intuition for this, let's watch a segment from Stanford's renowned deep learning course, CS231n.

As you saw, the key idea is to break a complex function into simple stages or "gates." During the backward pass, each gate is responsible for one thing: taking the gradient from the stage after it (the upstream gradient) and multiplying it by its own local gradient to pass it to the stage before it.

2. A Concrete Example: Deriving Gradients by Hand

Now let's apply this concept to a simple two-layer neural network. We'll walk through a complete example with specific numbers, which is often the best way to solidify understanding.

The following video provides a step-by-step walkthrough of both a forward and backward pass for a small network. We'll go through it in parts.

The Setup

The video uses a network with:

• 2 input neurons $(x_1,x_2)$
• 2 hidden neurons $(h_1,h_2)$ with ReLU activation
• 1 output neuron with a linear activation (i.e., no activation function)
• A single training example: input $[1,0]$ with a true target $y=2$.
• The loss function is $L=(y_{hat}-y{)}^2$.

The forward pass calculated a prediction of $y_{hat}=2.5$, resulting in a loss of $L=(2.5-2{)}^2=0.25$. Now, our goal is to find $\frac{\partial L}{\partial W}$ for every weight $W$ in the network.

Step 1: Gradients for the Output Layer ($W_5,W_6,B_3$)

The weights and bias of the output layer directly influence the final loss. Let's start with $W_5$. The loss $L$ is a function of $y_{hat}$, which is a function of $W_5$.

Using the chain rule:

• $\frac{\partial L}{\partial y_{hat}}$: The derivative of $(y_{hat}-y{)}^2$ with respect to $y_{hat}$ is $2(y_{hat}-y)$. This is the upstream gradient.
• $\frac{\partial y_{hat}}{\partial W_5}$: Since $y_{hat}=h_1{W}_5+h_2{W}_6+B_3$, the derivative with respect to $W_5$ is simply $h_1$. This is the local gradient.

So, $\frac{\partial L}{\partial W_5}=2(y_{hat}-y)\times h_1$.

Now, watch how this is calculated in the video.

Step 2: Gradients for the Hidden Layer ($W_1,W_2,W_3,W_4,B_1,B_2$)

This is where the "back" part of backpropagation becomes clear. The weight $W_1$ doesn't directly affect the loss $L$. It affects $h_1$, which affects $y_{hat}$, which in turn affects $L$. We need to chain all these dependencies together.

Where $z_1$ is the pre-activation value of the first hidden neuron. Let's break this down:

• $\frac{\partial L}{\partial y_{hat}}$: The gradient we already know from the output layer.
• $\frac{\partial y_{hat}}{\partial h_1}$: The derivative of $y_{hat}=h_1{W}_5+h_2{W}_6+B_3$ with respect to $h_1$ is $W_5$. This tells us how much $h_1$ contributed to the final output.
• $\frac{\partial h_1}{\partial z_1}$: The derivative of the ReLU activation function. It's 1 if $z_1>0$ and 0 otherwise.
• $\frac{\partial z_1}{\partial W_1}$: The derivative of $z_1=x_1{W}_1+x_2{W}_3+B_1$ with respect to $W_1$ is $x_1$.

The video walks through this multi-step chain rule application with concrete numbers, making it much easier to follow.

3. Generalizing to Vectorized Equations

The manual, scalar-by-scalar calculation is excellent for understanding, but in practice, we use matrices and vectors for efficiency. Your background in NumPy makes you well-prepared for this transition. The logic remains the same, but the operations are performed on entire batches of data at once.

Let's look at the final, clean, vectorized formulas for a two-layer network. These are the equations you will implement in the next lesson.

For a more formal mathematical treatment, the document "Backpropagation Derivation for a Two-Layer Neural Network" (LINK) lays out each step of the chain rule using matrix calculus notation. The "Summary of Gradients" section provides the same final equations we will use.

Conclusion

In this lesson, we demystified the core mechanism of network training. By peeling back the layers of abstraction, you've seen that backpropagation is not magic, but a systematic application of the chain rule.

Key Takeaways:

• Computational graphs help visualize a neural network as a series of simple operations ("gates").
• Backpropagation is the process of moving backward through this graph, calculating the gradient of the loss with respect to each parameter.
• The Chain Rule is the engine of backpropagation. At each gate, the gradient is calculated as: downstream_gradient = upstream_gradient * local_gradient.
• We manually derived the gradient formulas for all weights and biases in a simple two-layer network, first with scalars and then with generalized vector equations.

Preview of the Next Lesson:
With the "why" and "how" of the math covered, we are ready to code. In the next lesson, you will implement the backpropagation algorithm to compute gradients for all parameters in our deep MLP. We will add a backward method to our DenseLayer and MLP classes, bringing our from-scratch network one step closer to a complete training loop.