Backpropagation Explained: ANN Training Fundamentals

1. The Goal: Minimizing Prediction Error

Before we dive into the algorithm, let's clarify its purpose. When we first initialize a neural network, its weights and biases are set to random values. Its predictions are, therefore, completely arbitrary and incorrect. The goal of training is to systematically adjust these weights and biases so that the network's output gets progressively closer to the correct answer.

To do this, we first need a way to measure how "wrong" the network's predictions are. This is the job of a loss function (also called a cost or error function). It takes the network's prediction and the true target value and computes a single number representing the error. A common example, especially for regression-style problems, is the Mean Squared Error (MSE).

The formula shown in the image, $L=0.5\ast (y-a_3{)}^2$, is a classic example. If the output $a_3$ is very far from the target $y$, the loss will be high. If they are close, the loss will be low. The entire training process is an optimization problem: we want to find the set of weights and biases that results in the minimum possible loss.

Backpropagation is the algorithm that tells us how to adjust the weights and biases to achieve this minimum.

2. The Core Idea: Propagating "Blame" Backwards

Imagine a network with millions of weights. After seeing a large error, how do we know which weights to adjust? Should we increase some? Decrease others? Which adjustments will have the biggest impact? This is the "credit assignment" or "blame assignment" problem.

Backpropagation solves this intuitively. Instead of trying to figure out the effect of each weight individually, it starts from the error at the very end and works its way backward through the network, layer by layer.

The 3Blue1Brown video below provides an exceptionally clear and intuitive walkthrough of this process, framing it as a series of "desired nudges" that propagate backward.

As the video explains, the process for a single training example works like this:

1. The output neurons know how they need to change. For example, in a digit classifier, if the input is a '2', the '2' neuron's activation should be nudged up, and all others should be nudged down.
2. This "desire" is propagated back to the last hidden layer. Each neuron in this layer receives a "nudge" based on how strongly it contributed to the output error. Neurons connected with strong positive weights to an output that needs to increase will be told to "increase your activation."
3. This aggregated list of desired changes for the hidden layer is then used to calculate the desired changes for the next layer back, and so on.
4. This backward flow of information tells us precisely how sensitive the final error is to each and every weight and bias in the network.

3. The Four Steps of the Backpropagation Algorithm

Now that you have the intuition, let's formalize it into the conceptual steps you'll need for an exam. The training process for a neural network using backpropagation is an iterative cycle. The following flowchart provides a great high-level summary.

This cycle can be broken down into four key steps.

Let's summarize those steps:

1. Forward Pass: You've seen this in the last lesson. A single training example (e.g., the pixels of an image) is fed into the input layer. The network processes the input layer by layer, applying weights, biases, and activation functions, until it produces a prediction at the output layer.

2. Error Calculation: The loss function compares the network's prediction to the actual target label from the training data. This computes the error value, quantifying how wrong the prediction was.

3. Backward Pass (Backpropagation): This is the core of the algorithm. The error is propagated backward from the output layer. Using the chain rule from calculus, the algorithm calculates the gradient of the loss function with respect to each weight and bias. You don't need to perform the calculus for your exam, but you should know what the gradient represents: it is a vector that points in the direction of the steepest increase in the loss. In simpler terms, it tells us how much a small change in each weight/bias will affect the total error.

4. Weight Update: The gradients calculated in the backward pass are used to update all the weights and biases in the network. This is done using an optimization algorithm, most commonly Gradient Descent. The basic update rule is:
   $\text{new weight}=\text{old weight}-(\text{learning rate}\times \text{gradient})$
   We subtract the gradient because we want to move in the opposite direction of the steepest ascent—we want to go "downhill" on the loss curve. The learning rate is a small number that controls how big of a step we take.

This entire four-step cycle is repeated thousands of times, each time with a new batch of training data, causing the network's loss to gradually decrease and its predictions to become more accurate.

4. A Walkthrough Exam-Style Example

To solidify your understanding, it's helpful to see how these steps apply to a simple network with actual numbers. You will not be expected to perform these calculations from scratch in an exam, but you should be able to follow the logic.

The following resource provides a step-by-step numerical example.

By following the example, you can see the abstract steps in action:

• The Forward Pass computes the final output (0.67).
• The Error Calculation finds the difference between the target (0.5) and the output, resulting in an error of -0.17.
• The Backward Pass uses this error to calculate error terms (deltas) for each neuron and then calculates the required change for each weight (Δw).
• The Weight Update step applies these changes to get the new weights, completing one iteration. The process would then repeat with these new weights.

5. Practicalities and Key Terminology

For your exam, it's good to be aware of a few related concepts.

• Efficiency: Why use such a complex algorithm? Because it is incredibly efficient. The alternative would be to tweak each weight one by one and see how it affects the error, which would require millions of forward passes. Backpropagation calculates the entire gradient for all weights in just one forward pass and one backward pass.

• Stochastic Gradient Descent (SGD): Calculating the average gradient over the entire training dataset for every single update step is computationally very expensive. In practice, we use a technique called Stochastic Gradient Descent. The training data is shuffled and broken into small mini-batches (e.g., 32 or 64 examples). The algorithm then performs the four-step cycle on just one mini-batch to update the weights. This is much faster and, while the path to the minimum loss is more "drunken" and less direct, it works extremely well in practice.

Conclusion

In this lesson, we demystified the process of how a neural network learns. You now have a conceptual and practical understanding of backpropagation, the engine of modern deep learning.

Here are the key takeaways for your exam:

• Purpose: Backpropagation is an algorithm to train ANNs by efficiently computing the gradient of the loss function.
• Core Idea: It works by propagating an error signal backward through the network to assign "blame" to each weight and bias for the overall error.
• The Four Steps: The training loop consists of a Forward Pass (make a prediction), Error Calculation (compute loss), Backward Pass (calculate gradients), and Weight Update (apply changes via gradient descent).
• Key Components: The process relies on a loss function to measure error, the chain rule to calculate gradients, and an optimizer like Gradient Descent to update weights using a learning rate.

This lesson concludes our module on advanced supervised models. We've covered SVMs and the architecture and training of ANNs. In our next module, we will switch from supervised to unsupervised learning and explore methods for discovering patterns in data without predefined labels, starting with Association Rule Mining and the Apriori algorithm.