Backpropagation for Fully-Connected Networks

1. The Goal: Learning by Minimizing Error

A neural network is essentially a mathematical function parameterized by weights ($W$) and biases ($b$). During a forward pass, an input $x$ travels through the network's layers, and the network produces an output, or prediction, which we'll call $\hat{y}$.

The process at each layer $l$ looks like this:

1. Weighted Sum (Pre-activation): A linear combination of the previous layer's activations $a^{[l-1]}$ is computed.
   $z^{[l]}=W^{[l]}{a}^{[l-1]}+b^{[l]}$
2. Activation: A non-linear activation function $\sigma$ (like Sigmoid or ReLU) is applied to the weighted sum.
   $a^{[l]}=\sigma (z^{[l]})$

The final activation, $a^{[L]}$, is the network's prediction $\hat{y}$.

To learn, we need a way to measure how wrong our network's prediction is. We use a loss function (or cost function), $L(\hat{y},y)$, which compares the prediction $\hat{y}$ to the true target $y$. For simplicity in our derivation, we'll use the Mean Squared Error (MSE) loss, though the principles apply to others like Cross-Entropy.

$L=\frac{1}{2}(\hat{y}-y{)}^2$
(The $\frac{1}{2}$ is just a mathematical convenience that will simplify our derivatives later.)

The goal of training is to find the weights and biases that minimize this loss. We do this using an optimization algorithm called gradient descent. The idea is to calculate the gradient of the loss function with respect to all the parameters. The gradient, $\nabla L$, is a vector of partial derivatives ($\frac{\partial L}{\partial w_1},\frac{\partial L}{\partial b_1},\dots$) that points in the direction of the steepest increase in the loss. To minimize the loss, we take small steps in the opposite direction:

$\theta \leftarrow \theta -\alpha {\nabla }_{\theta }L$

Here, $\theta$ represents any parameter (a weight or bias) and $\alpha$ is the learning rate.

The central question then becomes: How do we efficiently compute the gradient ${\nabla }_{\theta }L$? This is where backpropagation comes in.

2. The Core Mechanism: The Chain Rule

Backpropagation is a clever application of the chain rule from calculus. It allows us to calculate how much a small change in any weight or bias, even in the earliest layers, will affect the final loss.

Let's start with a single weight in the final layer, $w^{[L]}$, connecting the last hidden layer's activation $a^{[L-1]}$ to the final output neuron. To find its contribution to the loss, we need to compute $\frac{\partial L}{\partial w^{[L]}}$.

We can trace the influence of this weight forward:

1. A change in $w^{[L]}$ affects the weighted sum $z^{[L]}$.
2. A change in $z^{[L]}$ affects the final activation $a^{[L]}$ (our prediction $\hat{y}$).
3. A change in $a^{[L]}$ affects the loss $L$.

The chain rule lets us express this relationship as a product of derivatives:
$\frac{\partial L}{\partial w^{[L]}}=\frac{\partial L}{\partial a^{[L]}}\cdot \frac{\partial a^{[L]}}{\partial z^{[L]}}\cdot \frac{\partial z^{[L]}}{\partial w^{[L]}}$

To see a great visual intuition of this process, let's turn to a classic video from 3Blue1Brown.

As the video explained, we can calculate each of those three parts:

1. $\frac{\partial L}{\partial a^{[L]}}$: The derivative of the loss with respect to the output activation.

   • For our MSE loss $L=\frac{1}{2}(a^{[L]}-y{)}^2$, this is simply $(a^{[L]}-y)$. This term represents the output error.

2. $\frac{\partial a^{[L]}}{\partial z^{[L]}}$: The derivative of the activation function.

   • If we're using the sigmoid function, $\sigma (z)$, its derivative is $\sigma (z)(1-\sigma (z))$. We can write this as ${\sigma }^{\prime }(z^{[L]})$.

3. $\frac{\partial z^{[L]}}{\partial w^{[L]}}$: The derivative of the weighted sum with respect to the weight.

   • For $z^{[L]}=w^{[L]}{a}^{[L-1]}+b^{[L]}$, this is just the activation from the previous layer, $a^{[L-1]}$.

Putting it all together for a weight in the output layer:
$\frac{\partial L}{\partial w^{[L]}}=(a^{[L]}-y)\cdot {\sigma }^{\prime }(z^{[L]})\cdot a^{[L-1]}$

What about the bias, $b^{[L]}$? The derivation is almost identical, but the last term changes. Since $\frac{\partial z^{[L]}}{\partial b^{[L]}}=1$, the gradient for the bias is:
$\frac{\partial L}{\partial b^{[L]}}=(a^{[L]}-y)\cdot {\sigma }^{\prime }(z^{[L]})$

3. Propagating the Error Backward

Now for the main event: calculating the gradient for a weight in a hidden layer, say $w^{[l]}$ in layer $l$. This is the "propagation" part of backpropagation.

A neuron in a hidden layer affects the loss through all the neurons it connects to in the next layer. This is a crucial point.

To calculate $\frac{\partial L}{\partial w^{[l]}}$, we first need to find the "error" at the output of hidden layer $l$, which is $\frac{\partial L}{\partial a^{[l]}}$. A change in $a^{[l]}$ affects the loss through its influence on the next layer's weighted sum, $z^{[l+1]}$.

Using the chain rule, the error at a hidden neuron's activation is the error from the next layer ($\frac{\partial L}{\partial z^{[l+1]}}$), weighted by the connection weight ($w^{[l+1]}$):
$\frac{\partial L}{\partial a^{[l]}}=\frac{\partial L}{\partial z^{[l+1]}}\frac{\partial z^{[l+1]}}{\partial a^{[l]}}=\frac{\partial L}{\partial z^{[l+1]}}\cdot w^{[l+1]}$
If neuron $a_j^{[l]}$ connects to multiple neurons in layer $l+1$, we must sum up all these weighted errors.

This recursive nature is what makes backpropagation efficient. We can define an error term, ${\delta }_j^{[l]}$, for each neuron $j$ in layer $l$ as the partial derivative of the loss with respect to its pre-activation input, $z_j^{[l]}$.

${\delta }_j^{[l]}=\frac{\partial L}{\partial z_j^{[l]}}$

With this notation, we can establish two key relationships:

1. Error at the output layer ($l=L$):
   ${\delta }_j^{[L]}=\frac{\partial L}{\partial a_j^{[L]}}{\sigma }^{\prime }(z_j^{[L]})$
2. Error at a hidden layer ($l<L$):
   ${\delta }_j^{[l]}=\left({\sum }_k{w}_{kj}^{[l+1]}{\delta }_k^{[l+1]}\right){\sigma }^{\prime }(z_j^{[l]})$
   This second equation is the heart of backpropagation. It shows how the error at layer $l$ is calculated from the weighted sum of errors at layer $l+1$.

Let's walk through a formal derivation to make this concrete.

4. The Four Core Equations of Backpropagation

From the derivations, we can summarize the entire backpropagation algorithm into four fundamental equations. For a given training example:

1. Forward Pass: Compute all activations $a^{[l]}$ and weighted sums $z^{[l]}$ for $l=1,\dots ,L$.

2. Output Error (${\delta }^{[L]}$): Compute the error vector for the output layer. In vectorized form:
   ${\delta }^{[L]}={\nabla }_{a}L\odot {\sigma }^{\prime }(z^{[L]})$
   ($\odot$ denotes element-wise multiplication. ${\nabla }_{a}L$ is the gradient of the loss with respect to the output activations, e.g., $a^{[L]}-y$ for MSE).

3. Backpropagate Error (${\delta }^{[l]}$): For each layer $l=L-1,L-2,\dots ,2$, compute the error:
   ${\delta }^{[l]}=\left((W^{[l+1]}{)}^T{\delta }^{[l+1]}\right)\odot {\sigma }^{\prime }(z^{[l]})$

4. Compute Gradients: The gradients of the loss with respect to the weights and biases of each layer are:
   $\frac{\partial L}{\partial b^{[l]}}={\delta }^{[l]}$
   $\frac{\partial L}{\partial W^{[l]}}={\delta }^{[l]}(a^{[l-1]}{)}^T$

These four equations give us a complete and efficient procedure for calculating the gradient of the loss function with respect to every single parameter in the network.

Conclusion

In this lesson, we have demystified the "magic" of how neural networks learn. We started from a simple loss function and, using the chain rule, derived the backpropagation algorithm from first principles.

Key Takeaways:

• Training a neural network involves minimizing a loss function using gradient descent.
• Backpropagation is an efficient algorithm for computing the gradient of the loss function with respect to all network parameters.
• The algorithm works by first performing a forward pass and then propagating an "error signal" backward from the output layer to the input layer.
• The error at any hidden layer is a weighted sum of the errors from the layer that follows it.

This foundational knowledge is critical. It separates you from someone who can only call model.fit() and enables you to reason about why your model might not be training correctly.

Preview of the Next Lesson:
Theory is essential, but practice is where understanding truly solidifies. In our next lesson, you will take these four equations and implement forward and backward propagation from scratch in Python for a multi-layer perceptron. This will be a challenging but incredibly rewarding exercise.