Building a Multi-Layer Neural Network from Scratch in Python

Introduction

In our previous lessons, we built a complete theoretical picture of a neural network. We traced the flow of information forward to generate predictions (the forward pass) and then derived the backpropagation algorithm to calculate how the error changes with respect to every weight and bias in the network (the backward pass). We also explored the potential instabilities of this process, namely the vanishing and exploding gradient problems.

Today, we transition from theory to practice. This lesson is dedicated to implementing the core engine of a neural network from scratch. Our goal is to translate the mathematical equations for the forward and backward passes into efficient, vectorized Python code using NumPy. This is a crucial step for any deep learning practitioner, as it demystifies what high-level libraries like TensorFlow and PyTorch do under the hood. Your background in systems engineering will be valuable here, as we'll see how vectorized computations in NumPy provide a massive performance advantage over naïve, loopy implementations.

By the end of this lesson, you will have built the two fundamental functions—forward_prop and back_prop—that form the heart of a neural network's training process.

The Anatomy of Our Network

To make our implementation concrete, we will build a simple two-layer neural network designed to solve a classic problem: handwritten digit classification using the MNIST dataset.

Our network will have the following structure:

1. Input Layer (${\mathbf{A}}^{(0)}$): This layer takes flattened 28x28 pixel images, resulting in 784 input neurons.
2. Hidden Layer (${\mathbf{A}}^{(1)}$): This layer will have 10 neurons, using the ReLU activation function.
3. Output Layer (${\mathbf{A}}^{(2)}$): This layer will have 10 neurons, one for each digit (0-9), using the Softmax activation function to produce a probability distribution.

The entire process of building, training, and testing this network is demonstrated in the video below. We will use this video as our primary guide, dissecting its Python implementation piece by piece.

1. Parameter Initialization

Before we can compute anything, our network needs parameters: the weights ($\mathbf{W}$) and biases ($\mathbf{b}$) for each layer. These must be initialized with some starting values.

Let's consider the dimensions. Our first weight matrix, ${\mathbf{W}}^{(1)}$, connects the 784-dimensional input to the 10-neuron hidden layer. The second weight matrix, ${\mathbf{W}}^{(2)}$, connects the 10-neuron hidden layer to the 10-neuron output layer.

Question: What should the dimensions of the matrices ${\mathbf{W}}^{(1)}$ and ${\mathbf{W}}^{(2)}$ be? What about the bias vectors ${\mathbf{b}}^{(1)}$ and ${\mathbf{b}}^{(2)}$?

We initialize these parameters with small random numbers. This is critical to break symmetry; if all weights start at the same value, all neurons in a layer will learn the same thing.

2. The Forward Pass: From Input to Prediction

The forward pass computes the network's prediction by propagating the input signal through the layers. This involves a sequence of linear transformations followed by non-linear activations.

For a single layer $l$, the computation is:

1. Linear Step: ${\mathbf{Z}}^{(l)}={\mathbf{W}}^{(l)}{\mathbf{A}}^{(l-1)}+{\mathbf{b}}^{(l)}$
2. Activation Step: ${\mathbf{A}}^{(l)}=g({\mathbf{Z}}^{(l)})$

Here, ${\mathbf{A}}^{(l-1)}$ is the activation from the previous layer (or the input data $\mathbf{X}$ for the first layer).

Let's implement the two activation functions we need: ReLU and Softmax.

• ReLU: $g(Z)=\max (0,Z)$. This can be done efficiently in NumPy using np.maximum(0, Z).
• Softmax: $\sigma (\mathbf{z}{)}_i=\frac{e^{z_i}}{{\sum }_{j=1}^K{e}^{z_j}}$. This normalizes the outputs into a probability distribution. This is implemented as np.exp(Z) / np.sum(np.exp(Z), axis=0). The axis=0 argument is crucial for vectorized computation, ensuring we sum over the neurons for each example independently.

With these helpers, we can build the full forward_prop function.

The forward_prop function takes the parameters and input data, and returns not only the final prediction ${\mathbf{A}}^{(2)}$ but also the intermediate values ${\mathbf{Z}}^{(1)},{\mathbf{A}}^{(1)},{\mathbf{Z}}^{(2)}$. We need to cache these values because they are essential for the backward pass.

3. The Backward Pass: From Error to Gradients

This is the most mathematically intensive part, where we implement the backpropagation algorithm derived in the previous module. Our goal is to compute the gradients of the loss function with respect to every weight and bias: $\frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{(1)}}$, $\frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{(1)}}$, $\frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{(2)}}$, and $\frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{(2)}}$.

The process starts at the output and works backward.

Step 1: Output Layer Error
The error signal for the output layer, ${\mathbf{dZ}}^{(2)}$, is simply the difference between the predictions and the true labels:
${\mathbf{dZ}}^{(2)}={\mathbf{A}}^{(2)}-\mathbf{Y}$
Here, $\mathbf{Y}$ represents the true labels, which must be in a "one-hot" format. For example, the digit '4' becomes a 10-element vector [0, 0, 0, 0, 1, 0, 0, 0, 0, 0].

Step 2: Output Layer Gradients
Using ${\mathbf{dZ}}^{(2)}$, we compute the gradients for ${\mathbf{W}}^{(2)}$ and ${\mathbf{b}}^{(2)}$. The equations we derived are:
$\frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{(2)}}=\frac{1}{m}{\mathbf{dZ}}^{(2)}({\mathbf{A}}^{(1)}{)}^T$
$\frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{(2)}}=\frac{1}{m}{\sum }_{i=1}^m{\mathbf{dZ}}_i^{(2)}$
where $m$ is the number of training examples. The bias gradient is an average of the error signal across all examples.

Step 3: Hidden Layer Error
Now we propagate the error backward to the hidden layer:
${\mathbf{dZ}}^{(1)}=(({\mathbf{W}}^{(2)}{)}^T{\mathbf{dZ}}^{(2)})\odot g^{\prime }({\mathbf{Z}}^{(1)})$
This is the core backpropagation step. The term $g^{\prime }({\mathbf{Z}}^{(1)})$ is the derivative of our hidden layer's activation function, which is ReLU. The derivative of ReLU is particularly simple: it's 1 if the input is positive, and 0 otherwise.

Step 4: Hidden Layer Gradients
Finally, we compute the gradients for ${\mathbf{W}}^{(1)}$ and ${\mathbf{b}}^{(1)}$ using the newly computed ${\mathbf{dZ}}^{(1)}$:
$\frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{(1)}}=\frac{1}{m}{\mathbf{dZ}}^{(1)}(\mathbf{X}{)}^T$
$\frac{\partial \mathcal{L}}{\partial {\mathbf{b}}^{(1)}}=\frac{1}{m}{\sum }_{i=1}^m{\mathbf{dZ}}_i^{(1)}$

Now, let's watch how these four steps are translated into vectorized NumPy code.

The result of the back_prop function is a set of gradient matrices (dW1, db1, dW2, db2) that have the same dimensions as our parameter matrices. These gradients tell us the direction of steepest ascent on the loss surface.

Conclusion

In this lesson, you have translated the abstract mathematics of neural networks into concrete, functional Python code. We have successfully implemented the two most important functions in a neural network:

• forward_prop: Takes inputs and current parameters to produce a prediction, caching intermediate values along the way.
• back_prop: Uses the cached values and true labels to compute the gradient of the loss with respect to every single parameter in the network.

These two functions are the essential building blocks for training. You now have an "engine" that can process data and calculate the exact direction to adjust the parameters to improve performance. The vectorized nature of our implementation, using NumPy's optimized C and Fortran backends for matrix operations, is precisely the kind of low-level performance consideration that is crucial in real-world systems.

Key Takeaways:

• A feedforward neural network's computation can be expressed as a series of matrix multiplications and element-wise non-linear functions.
• The forward pass propagates a signal from input to output, while the backward pass propagates an error gradient from output to input.
• Vectorization with libraries like NumPy is key to implementing these passes efficiently, avoiding slow Python loops and operating on entire batches of data at once.
• The back_prop function relies on values cached during the forward_prop function, creating a tight coupling between the two.

Preview of the Next Lesson:

We have built the engine, but we haven't turned the key yet. In our next lesson, we will do just that. We will construct a training loop that repeatedly calls forward_prop and back_prop. We'll use the gradients returned by back_prop to update our network's weights and biases using an optimizer, a process known as gradient descent. This will complete the full training pipeline and allow us to watch our network learn to recognize digits.