Performing and Interpreting a Gradient Descent Update

From a gradient to a parameter update

Let the parameter vector be

and let $L(\mathbf{\theta })$ be the loss computed on the training data. Gradient descent updates the parameters by

where:

• $t$ is the iteration number;
• $\nabla L({\mathbf{\theta }}^{(t)})$ is the gradient evaluated at the current parameters;
• $\eta >0$ is the learning rate;
• $\eta \nabla L$ is the step calculated from the gradient.

The minus sign is the essential feature. The gradient points in the direction of steepest local increase in loss, so subtracting it moves in the direction of steepest local decrease.

For the two parameters separately, the rule is:

All gradient components must be calculated from the same old parameter setting, and the updates happen simultaneously. Do not update $b$, recompute the model, and then update $w$ using that partially updated model. That would be a different algorithm.

The four-step gradient-descent diagram below captures the wider loop. This lesson focuses closely on the transition from steps 2 to 3: turning a computed gradient into one controlled move.

One complete update by hand

Continue with the two-observation example from the previous lesson:

The model is

and the loss is the half mean squared error:

At the current parameter setting,

the predictions are $1$ and $3$, while the residuals $y_i-{\hat{y}}_i$ are $1$ and $2$. Thus the current loss is

From the previous lesson, the gradient at this point is

Choose a learning rate of

The gradient-descent step is

Add this step to the old parameter vector:

Equivalently, applying the update component by component:

Interpret the signs

Both partial derivatives were negative:

A negative derivative says that a small increase in that parameter should reduce loss locally. Subtracting a negative gradient component correctly increases both $b$ and $w$.

The updated model is

Its predictions are:

The new residuals are therefore $0.50$ and $0.80$, giving

The loss fell from $1.25$ to $0.2225$. This confirms that this particular update improved the model's fit to these training observations.

The larger update to $w$ does not mean the feature is inherently more important. It says only that, at this parameter setting and this feature scale, the loss was more sensitive to a change in $w$ than to an equally sized change in $b$.

The learning rate controls caution, not direction

The gradient determines a direction. The learning rate determines how far to move in that direction.

A small learning rate takes cautious steps:

This may reduce loss reliably, but it can require many iterations before parameters get close to a minimum.

A larger learning rate moves more aggressively. It may reduce loss faster, but it can overshoot the minimum. In our example, try using

with the same initial gradient:

The resulting predictions are $3.5$ and $9$, which are much too high for the target values $2$ and $5$. The loss becomes

So this larger step makes the loss worse than the original loss of $1.25$.

This does not contradict the fact that the negative gradient is a descent direction. It is a local statement: a sufficiently small move in the negative-gradient direction reduces loss to first order. A large move can jump across the valley and land at a higher-loss location.

Loss scaling changes the numerical learning rate

You may encounter either of these squared-error objectives:

They have the same minimizing parameters because one is exactly twice the other:

But their gradients differ:

That is why Google’s walkthrough can report derivatives that look different from yours even when the underlying model and data are equivalent. If a loss is multiplied by $2$, then a learning rate should be divided by $2$ to produce the same parameter step.

The lesson is practical: never interpret a learning rate in isolation. Its useful scale depends on:

• the loss definition;
• the number and scale of features;
• the current local shape of the loss surface;
• whether the loss is averaged over observations.

For example, multiplying a numerical feature by $1000$ can greatly change the associated weight gradient. This is one reason feature scaling becomes important in gradient-based model fitting.

Express the update in matrix form

For a linear model with an intercept, create a design matrix whose first column is all ones:

and write the parameter vector and outcomes as

Then the prediction vector is

For the half-MSE loss, the gradient is

The update is simply

This is the version worth recognizing in real code: matrix multiplication calculates the predictions and accumulates every parameter's derivative efficiently.

import numpy as np

# First column is the intercept feature.
X = np.array([
    [1.0, 1.0],
    [1.0, 3.0]
])

y = np.array([2.0, 5.0])
theta = np.array([0.0, 1.0])  # [b, w]
learning_rate = 0.1

def half_mse_loss(theta, X, y):
    errors = X @ theta - y
    return 0.5 * np.mean(errors ** 2)

def gradient(theta, X, y):
    errors = X @ theta - y
    return X.T @ errors / len(y)

old_loss = half_mse_loss(theta, X, y)
grad = gradient(theta, X, y)

# The right-hand side uses only the old theta and its gradient.
new_theta = theta - learning_rate * grad
new_loss = half_mse_loss(new_theta, X, y)

print("Old parameters:", theta)
print("Gradient:", grad)
print("New parameters:", new_theta)
print("Old loss:", old_loss)
print("New loss:", new_loss)

The key outputs should be approximately:

Gradient: [-1.5 -3.5]
New parameters: [0.15 1.35]
Old loss: 1.25
New loss: 0.2225

A compact audit for any hand calculation or implementation is:

1. State the parameter order.
2. Calculate predictions and loss using the current parameters.
3. Calculate the gradient in that same parameter order.
4. Compute the step $-\eta \nabla L$.
5. Apply the whole vector update simultaneously.
6. Recompute the loss at the new parameters as a diagnostic.

For a one-step calculation, an improved loss is useful evidence that the sign, gradient, and learning-rate scale are sensible. It is not a guarantee that every future step will reduce loss, nor does a decreasing training loss establish that the model will generalize well to new data.

Key takeaways

Gradient descent changes parameters using

The negative gradient supplies a local direction of loss reduction. The learning rate controls the step magnitude: too small can be slow, while too large can overshoot and increase loss.

For the example starting at $(b,w)=(0,1)$, the gradient

and learning rate $0.1$ produce the simultaneous update

which lowers the half-MSE from $1.25$ to $0.2225$.

This completes the optimization foundations module. Next, the course turns to probability foundations, beginning with how marginal, joint, and conditional probabilities are computed from a probability table.