Gradient Descent: Weight Update Rule

Hello! Welcome back to our module on building neural networks from scratch.

In our last lesson, we completed the most mathematically intensive part of our journey: implementing backpropagation. You wrote a NumPy function that takes the network's outputs and true labels, and works backward through the layers to compute the gradient of the loss function with respect to every single weight and bias. These gradients, like ${\nabla }_{W}C$ and ${\nabla }_{b}C$, represent the direction of the steepest ascent on the loss landscape.

Today, we'll use those gradients to actually "learn." This lesson focuses on the final, crucial step of the gradient descent algorithm: updating the network's parameters to minimize the loss.

This lesson directly addresses the learning outcome: Implement the weight update step using the computed gradients and a learning rate.

1. The Gradient Descent Update Rule

We have the gradients, which tell us which way is "uphill" for the loss function. To train the network, we need to take a small step in the opposite direction—"downhill"—to reduce the error. This is the essence of gradient descent.

The update rule for any parameter $\theta$ (which can be a weight $W$ or a bias $b$) is defined as:

$${\theta }_{\text{new}}={\theta }_{\text{old}}-\alpha {\nabla }_{\theta }C$$

Let's break this down:

• ${\theta }_{\text{old}}$ is the current value of our parameter.
• ${\nabla }_{\theta }C$ is the gradient of the loss $C$ with respect to that parameter, which we calculated in the last lesson.
• $\alpha$ is the learning rate, a small positive number (e.g., 0.1, 0.01) that we choose. It's a hyperparameter that controls how large of a step we take.

So, for our weights and biases, the specific update rules are:

$$W^{(l)}\leftarrow W^{(l)}-\alpha {\nabla }_{W^{(l)}}C$$ $$b^{(l)}\leftarrow b^{(l)}-\alpha {\nabla }_{b^{(l)}}C$$

The $\leftarrow$ symbol signifies that we are updating the parameter's value in place.

The role of the learning rate $\alpha$ is critical:

• If $\alpha$ is too large, we might overshoot the minimum of the loss function and diverge.
• If $\alpha$ is too small, training will be very slow as we take tiny steps.

Finding a good learning rate is a key part of training neural networks.

To solidify your understanding of this concept, let's review a short text that explains the intuition behind using the derivative and the learning rate.

2. Implementing the Update Step in NumPy

Translating the update rule into code is remarkably straightforward. Continuing with the implementation from our previous lessons, let's see how this is done.

Now, let's see the direct implementation of these formulas in a Python function.

As you saw, the implementation for a two-layer network is as simple as this:

def update_params(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha):
    """
    Updates parameters using the gradient descent rule.
    
    Args:
        W1, b1, W2, b2: Current parameters of the network.
        dW1, db1, dW2, db2: Gradients computed from backpropagation.
        alpha: The learning rate.
        
    Returns:
        Updated W1, b1, W2, b2.
    """
    W1 = W1 - alpha * dW1
    b1 = b1 - alpha * db1
    W2 = W2 - alpha * dW2
    b2 = b2 - alpha * db2
    return W1, b1, W2, b2

Each line performs an element-wise subtraction of the scaled gradient from the corresponding parameter matrix. This is possible because our back_prop function from the last lesson produced gradient matrices (dW1, db1, etc.) with the exact same dimensions as the parameter matrices (W1, b1, etc.).

3. An Object-Oriented Approach: The Optimizer

While the function above works perfectly, in larger projects and deep learning frameworks, this update logic is often encapsulated within an optimizer class. This keeps the code clean and makes it easy to swap out different optimization algorithms (like Adam or RMSProp, which we'll cover later).

Let's look at a simple implementation of a Stochastic Gradient Descent (SGD) optimizer.

The step() method in Daza's SGD class is the key. It iterates through the layers of the model and applies the update rule layer.W = layer.W - self.lr * layer.dW. This design is very similar to how optimizers work in PyTorch, where you would call optimizer.step() to update your model's parameters after computing gradients.

4. The Update Step in the Full Training Loop

We now have all the individual components for training our network:

1. Forward Pass: To make predictions.
2. Backward Pass: To compute gradients.
3. Parameter Update: To apply the gradients and learn.

The training process is simply a loop that repeatedly executes these three steps. Let's watch a final video that puts our new update step into the context of this training loop.

This iterative process of forward pass, backward pass, and parameter update is the fundamental heartbeat of training almost any neural network.

Conclusion

In this lesson, you have implemented the final core mechanic of network training. We have closed the loop of gradient-based optimization.

Key Takeaways:

• The gradient descent update rule is parameter = parameter - learning_rate * gradient. This moves the parameters in the direction that minimizes the loss.
• The learning rate ($\alpha$) is a crucial hyperparameter that controls the step size of each update.
• The implementation in NumPy is a straightforward element-wise operation: W = W - alpha * dW.
• This update logic can be encapsulated in an optimizer class for cleaner, more modular code, which is standard practice in deep learning frameworks.

You now have all the individual building blocks. In our next and final lesson of this module, we will assemble them into a complete train function. You will initialize a network, loop through a dataset, and watch your from-scratch model learn to solve a classification problem, all using the functions you have painstakingly built.