Computing Gradients from Partial Derivatives

Good to see you again. In the previous lesson, you used the product and chain rules to differentiate a loss with respect to one parameter. A predictive model, however, rarely has just one adjustable quantity: even a simple linear model has an intercept and one or more feature weights. The loss is therefore a function over a parameter space, and we need one derivative for each parameter.

This lesson extends scalar derivatives to partial derivatives and combines them into the gradient of a multivariable loss. By the end, you should be able to compute a gradient by holding all but one parameter fixed at a time, and interpret each component as a local sensitivity of model loss.

From an ordinary derivative to partial derivatives

Suppose a model has two parameters:

$b$ : an intercept;
$w$ : the weight attached to a feature $x$ .

Its loss can be written as:

L (b, w)

This notation means that the scalar loss changes when either $b$ , $w$ , or both change. But when we ask, “What happens if I increase $b$ slightly?”, we want to isolate the effect of $b$ while keeping $w$ fixed. This is a partial derivative:

\frac{\partial L}{\partial b}

The curled symbol $\partial$ signals that the function has multiple inputs. The calculation is otherwise familiar: differentiate with respect to the named variable, treating every other variable as a constant.

Likewise,

\frac{\partial L}{\partial w}

measures the local change in loss caused by a small change in $w$ , with $b$ held fixed.

For a two-parameter loss, the gradient collects those derivatives in parameter order:

\nabla L (b, w) = [\frac{\partial L}{\partial b} \frac{\partial L}{\partial w}]

More generally, if the parameter vector is

θ = θ_{1} θ_{2} ⋮ θ_{p}

then the gradient of a scalar loss is

\nabla_{θ} L (θ) = \frac{\partial L}{\partial θ _{1}} \frac{\partial L}{\partial θ _{2}} ⋮ \frac{\partial L}{\partial θ _{p}}

The order matters. The first gradient component must correspond to the first parameter, the second to the second parameter, and so forth. In an implementation, a mismatch between parameter ordering and gradient ordering is a serious bug: the optimizer would change the wrong parameter.

Partial Derivatives and the Gradient of a Function

Watch “Partial Derivatives and the Gradient of a Function” by Professor Dave Explains for a compact visual account of holding one variable fixed and then assembling the resulting partial derivatives into a vector.

Watch partial derivatives to see the “hold the other variable constant” rule applied to a concrete function. Then watch the gradient for the geometric interpretation: the gradient combines coordinate-wise rates of change and points in the direction of steepest local increase.

Compute partial derivatives one coordinate at a time

Consider the two-variable function

f (u, v) = u^{2} v + 3 v^{2}

To calculate the partial derivative with respect to $u$ , treat $v$ as a constant:

\frac{\partial f}{\partial u} = 2 uv

The term $3 v^{2}$ contains no $u$ , so it acts like a constant and disappears.

Now calculate the partial derivative with respect to $v$ . This time $u$ is constant:

\frac{\partial f}{\partial v} = u^{2} + 6 v

Therefore,

\nabla f (u, v) = [2 uv u^{2} + 6 v]

At the particular point $u = 2$ , $v = - 1$ ,

\nabla f (2, - 1) = [2 (2) (- 1) 2^{2} + 6 (- 1)] = [- 4 - 2]

This vector says two things at once:

increasing $u$ slightly, while holding $v = - 1$ , locally decreases $f$ , since the first component is negative;
increasing $v$ slightly, while holding $u = 2$ , also locally decreases $f$ , since the second component is negative.

The gradient is not merely a list of derivatives. It is a vector describing the local shape of a scalar function across all parameter directions.

A reliable hand-calculation workflow is:

Write down the parameter order, for example $(b, w)$ .
Compute one partial derivative per parameter.
For each derivative, explicitly identify which variables are being held fixed.
Apply product and chain rules exactly as in one-variable calculus.
Stack the results in the same order as the parameters.
Substitute numerical parameter values only after deriving the symbolic gradient, unless the loss is already fully numerical.

2.4. Calculus — Dive into Deep Learning 1.0.3 documentation

Read Section 2.4.3, “Partial Derivatives and Gradients,” from Dive into Deep Learning. It formalizes the partial-derivative rule and the gradient-vector definition used throughout machine learning.

Read all of Section 2.4.3, beginning with the transition to multivariable functions. Focus first on the instruction to treat all other variables as constants, then on how the partial derivatives are concatenated into a gradient. The later matrix formulas are useful reference material, but do not need to be memorized yet.

A gradient for a predictive-model loss

Now apply the procedure to a small regression-style model. Suppose each observation has:

feature value $x_{i}$ ;
observed outcome $y_{i}$ ;
prediction $y^_{i} = b + w x_{i}$ .

For now, use the mean squared error with a convenient factor of one half:

L (b, w) = \frac{1}{2 n} i = 1 \sum n [y_{i} - (b + w x_{i})]^{2}

This is a pedagogical loss for practicing gradients. For binary conversion prediction, logistic regression will later use log loss rather than squared error. The differentiation pattern, however, is the same: trace how a parameter changes the prediction, then how the prediction changes the loss.

Define a residual as:

r_{i} = y_{i} - (b + w x_{i})

Then:

L (b, w) = \frac{1}{2 n} i = 1 \sum n r_{i}^{2}

Partial derivative with respect to the intercept

Differentiate one squared residual with respect to $b$ . By the chain rule:

\frac{\partial}{\partial b} [r_{i}^{2}] = 2 r_{i} \frac{\partial r _{i}}{\partial b}

Since

\frac{\partial r _{i}}{\partial b} = - 1

we obtain:

\frac{\partial L}{\partial b} = - \frac{1}{n} i = 1 \sum n r_{i}

Equivalently,

\frac{\partial L}{\partial b} = - \frac{1}{n} i = 1 \sum n [y_{i} - (b + w x_{i})]

The intercept affects every prediction by exactly one unit per unit change in $b$ . That is why no $x_{i}$ appears in this derivative.

Partial derivative with respect to the weight

For $w$ , the inner residual changes at a rate determined by $x_{i}$ :

\frac{\partial r _{i}}{\partial w} = - x_{i}

Applying the same chain rule gives:

\frac{\partial L}{\partial w} = - \frac{1}{n} i = 1 \sum n x_{i} r_{i}

or:

\frac{\partial L}{\partial w} = - \frac{1}{n} i = 1 \sum n x_{i} [y_{i} - (b + w x_{i})]

So the full gradient is:

\nabla L (b, w) = [- \frac{1}{n} \sum_{i = 1}^{n} [y_{i} - (b + w x_{i})] - \frac{1}{n} \sum_{i = 1}^{n} x_{i} [y_{i} - (b + w x_{i})]]

The two components have distinct meanings:

the intercept component aggregates the model’s signed errors;
the weight component aggregates signed errors weighted by the feature value.

This is why a feature’s scale matters during optimization. If one numerical feature has values around $10, 000$ and another has values around $0.01$ , their weight gradients can differ greatly in scale even when both features are relevant. Later, feature scaling and appropriate optimization settings will address this practical issue.

A complete numerical gradient calculation

Take two observations:

(x_{1}, y_{1}) = (1, 2)

(x_{2}, y_{2}) = (3, 5)

Start at:

b = 0

w = 1

The current predictions are:

y^_{1} = 0 + 1 (1) = 1

y^_{2} = 0 + 1 (3) = 3

Thus the residuals are:

r_{1} = 2 - 1 = 1

r_{2} = 5 - 3 = 2

The intercept derivative is:

\frac{\partial L}{\partial b} = - \frac{1}{2} (1 + 2) = - \frac{3}{2}

The weight derivative is:

\frac{\partial L}{\partial w} = - \frac{1}{2} [1 (1) + 3 (2)] = - \frac{7}{2}

Therefore:

\nabla L (0, 1) = [- \frac{3}{2} - \frac{7}{2}]

Both components are negative. At this particular parameter setting, increasing either parameter by a small amount would decrease the loss locally. The larger magnitude for $w$ tells us that, under the current feature scale, loss is more sensitive to a change in $w$ than to an equally sized change in $b$ .

That is a statement about the local loss surface, not a claim that the feature is more important, more predictive in general, or causal. Gradients answer an optimization question: how should the parameters move to change this chosen loss around the current parameter values?

The geometry: why the negative gradient is useful

For a function of two parameters, imagine a landscape where the horizontal axes are $b$ and $w$ , and the vertical axis is the loss $L$ . A parameter setting is a location on this loss surface.

A point on a three-dimensional loss surface with the gradient $\nabla f$ indicating the direction of greatest local increase, and the negative gradient $-\nabla f$ indicating the direction of greatest local decrease; directions perpendicular to the gradient produce zero first-order change.

The gradient points in the direction of greatest local increase. Its negative points in the direction of greatest local decrease.

More precisely, for a small parameter change $Δ θ$ ,

L (θ + Δ θ) \approx L (θ) + \nabla L (θ)^{⊤} Δ θ

If we choose a small move proportional to the negative gradient,

Δ θ = - ϵ \nabla L (θ)

for a positive scalar $ϵ$ , then the first-order change becomes:

\nabla L (θ)^{⊤} Δ θ = - ϵ ∥ \nabla L (θ) ∥^{2}

This quantity is non-positive and is strictly negative whenever the gradient is nonzero. Locally, this is why moving opposite the gradient reduces the loss.

A zero gradient,

\nabla L (θ) = 0

means there is no first-order direction of improvement at that point. Such a point might be a minimum, but it can also be a maximum or a saddle point. The gradient alone identifies a stationary point; further reasoning is needed to classify it.

Check an analytic gradient numerically

In notebooks, a finite-difference check is a useful guard against algebra and implementation errors. It perturbs one parameter at a time and estimates the relevant partial derivative.

import numpy as np

x = np.array([1.0, 3.0])
y = np.array([2.0, 5.0])

def loss(theta):
    b, w = theta
    predictions = b + w * x
    return 0.5 * np.mean((y - predictions) ** 2)

def analytic_gradient(theta):
    b, w = theta
    residuals = y - (b + w * x)

    dL_db = -np.mean(residuals)
    dL_dw = -np.mean(x * residuals)

    return np.array([dL_db, dL_dw])

theta = np.array([0.0, 1.0])
h = 1e-6

numeric_gradient = np.zeros_like(theta)

for j in range(len(theta)):
    perturbation = np.zeros_like(theta)
    perturbation[j] = h

    numeric_gradient[j] = (
        loss(theta + perturbation) - loss(theta - perturbation)
    ) / (2 * h)

print("Analytic gradient:", analytic_gradient(theta))
print("Numeric gradient: ", numeric_gradient)

Both lines should be close to:

[- 1.5 - 3.5]

The central finite difference does not replace a derivation, but it is a strong practical check. It is especially helpful before trusting a custom loss function or a manual implementation of a model-training routine.

Key takeaways

A partial derivative isolates one input’s local effect on a multivariable function while holding the other inputs fixed:

\frac{\partial L}{\partial θ _{j}}

The gradient stacks all partial derivatives of a scalar loss in the same order as its parameters:

\nabla_{θ} L (θ) = \frac{\partial L}{\partial θ _{1}} ⋮ \frac{\partial L}{\partial θ _{p}}

For a squared-error linear model, chain-rule differentiation produces one gradient component for the intercept and one for each feature weight. These components measure local loss sensitivity, not feature importance or causal effect.

Next, you will use a computed gradient to perform and interpret a single gradient-descent parameter update.

Recommended resources