Calculating Partial Derivatives

Hello! Welcome back to our module on Calculus for Machine Learning.

In the last lesson, we mastered the concept of a derivative for single-variable functions. We saw that it represents the slope of the tangent line at a point, telling us the function's instantaneous rate of change. We also learned the basic rules, like the power rule, to calculate these derivatives.

However, the functions we optimize in machine learning are rarely so simple. A model's loss function depends on thousands, sometimes millions, of parameters (weights and biases). This means we're not dealing with a simple curve $f (x)$ , but a high-dimensional surface $L (w_{1}, w_{2}, ..., w_{n})$ . How do we find the "slope" on such a complex landscape?

This brings us to today's learning outcome: to calculate partial derivatives of multivariable functions. A partial derivative allows us to isolate the effect of a single variable on the function's output, which is the first step toward navigating these complex surfaces to find the optimal model parameters.

1. Visualizing Multivariable Functions

Before we can find the slope on a surface, let's first build an intuition for what these surfaces look like. A function of one variable, $f (x)$ , can be plotted on a 2D graph as a curve. A function of two variables, $f (x, y)$ , can be plotted on a 3D graph as a surface.

3D Surface Plots and 2D Contour Plots of Multivariable Functions — This image shows two functions of two variables. The top function, $f(x,y) = x+y$, forms a flat plane. The bottom function, $f(x,y) = x^2+y^2$, forms a bowl shape called a paraboloid. The images on the right are **contour plots**, which are like topographical maps showing lines of constant "elevation" (function value). These are a key tool for visualizing the landscapes we'll explore in machine learning.

When we talk about finding a "slope" on these surfaces, the question becomes: slope in which direction? Are we moving along the x-axis, the y-axis, or some other direction?

2. The Core Idea: Partial Derivatives

A partial derivative answers this question by simplifying the problem. It measures the slope in a direction parallel to one of the coordinate axes. The key idea is to hold all other variables constant and calculate the derivative with respect to just one variable.

Imagine you are standing on the side of a hill (our surface). The partial derivative with respect to 'x' would be the steepness you'd feel if you only walked in the east-west direction. The partial derivative with respect to 'y' would be the steepness if you only walked in the north-south direction.

Let's watch a short video that uses a similar analogy to introduce this concept.

Best Explanation of Partial Derivatives and Gradients

This video from Brain Station Advanced provides a great intuitive explanation of partial derivatives by imagining the function as a bumpy surface.

Watch the segment from 02:17 to 04:03. Focus on the idea of 'fixing' one variable (like x) and tracing a curve along the other variable (y) to find the slope in that specific direction.

This leads us to the central rule of partial differentiation:

To find the partial derivative of a function with respect to one variable, you treat all other variables as if they were constants.

You then apply all the single-variable derivative rules you learned in the previous lesson.

Let's see this in action with a concrete example. Consider the function from the video we're about to watch: $z = f (x, y) = x^{2} - y^{2}$ .

To find the partial derivative of $z$ with respect to $x$ , we treat $y$ as a constant. Since $y^{2}$ is treated as a constant, its derivative is 0. The derivative of $x^{2}$ is $2 x$ . So, we get:
$\frac{\partial z}{\partial x} = 2 x - 0 = 2 x$
To find the partial derivative of $z$ with respect to $y$ , we treat $x$ as a constant. The derivative of $x^{2}$ is now 0. The derivative of $- y^{2}$ is $- 2 y$ . So, we get:
$\frac{\partial z}{\partial y} = 0 - 2 y = - 2 y$

The symbol $\partial$ is called "del", and we read $\frac{\partial z}{\partial x}$ as "the partial derivative of z with respect to x" or simply "del z del x".

The following video provides a fantastic hands-on introduction, connecting the "why" in machine learning to the "how" of calculation and visualization.

What Partial Derivatives Are (Hands-on Introduction) — Topic 67 of Machine Learning Foundations

This video by Jon Krohn will first motivate why we need partial derivatives in machine learning and then walk through calculating them for the function z = x^2 - y^2.

Watch the following two segments: Why Partial Derivatives? (00:14 - 01:33): This explains the need for partials in the context of a simple regression model. Calculating a Partial Derivative (04:05 - 06:20): This segment walks through the calculation of rac{\partial z}{\partial x} and introduces the 'del' notation. Pay close attention to how the variable 'y' is treated as a constant.

3. Step-by-Step Calculation

Now that you have the core concept, let's work through another example to solidify the process. This time, we'll see a function where the variables are multiplied together.

Calculus — ML Glossary documentation

The ML Cheatsheet provides a clear, step-by-step guide for calculating partial derivatives and assembling them into a gradient vector.

Read the subsections 'Partial derivatives' and 'Step-by-step'. The example used is f(x,z) = 2z^3x^2. Follow the logic of how one variable is replaced with a constant 'b' to make the calculation clearer.

Let's recap the example from the reading, $f (x, z) = 2 z^{3} x^{2}$ :

To find $\frac{\partial f}{\partial x}$ :
- Treat $z$ as a constant. This means the entire term $2 z^{3}$ is just a constant multiplier.
- Let's call $c = 2 z^{3}$ . The function is $f (x) = c \cdot x^{2}$ .
- Using the constant multiple and power rules, the derivative is $c \cdot 2 x$ .
- Substituting back, we get $\frac{\partial f}{\partial x} = (2 z^{3}) \cdot 2 x = 4 z^{3} x$ .
To find $\frac{\partial f}{\partial z}$ :
- Treat $x$ as a constant. This means the term $2 x^{2}$ is our constant multiplier.
- Let's call $d = 2 x^{2}$ . The function is $f (z) = d \cdot z^{3}$ .
- The derivative is $d \cdot 3 z^{2}$ .
- Substituting back, we get $\frac{\partial f}{\partial z} = (2 x^{2}) \cdot 3 z^{2} = 6 x^{2} z^{2}$ .

Test your understanding!

Calculate the partial derivatives $\frac{\partial f}{\partial x}$ and $\frac{\partial f}{\partial y}$ for the following function:
$f (x, y) = 5 x^{3} y^{2} - 2 x^{2} + 3 y^{4}$

Show answer

To find $\frac{\partial f}{\partial x}$ :

Treat $y$ as a constant.
Derivative of $5 x^{3} y^{2}$ with respect to $x$ is $(5 y^{2}) \cdot (3 x^{2}) = 15 x^{2} y^{2}$ .
Derivative of $- 2 x^{2}$ with respect to $x$ is $- 4 x$ .
Derivative of $3 y^{4}$ with respect to $x$ is $0$ (since it's treated as a constant).
So, $\frac{\partial f}{\partial x} = 15 x^{2} y^{2} - 4 x$ .

To find $\frac{\partial f}{\partial y}$ :

Treat $x$ as a constant.
Derivative of $5 x^{3} y^{2}$ with respect to $y$ is $(5 x^{3}) \cdot (2 y) = 10 x^{3} y$ .
Derivative of $- 2 x^{2}$ with respect to $y$ is $0$ .
Derivative of $3 y^{4}$ with respect to $y$ is $12 y^{3}$ .
So, $\frac{\partial f}{\partial y} = 10 x^{3} y + 12 y^{3}$ .

4. Code and Geometric Interpretation

We've now calculated partial derivatives, but what do they look like? The Jon Krohn video you started earlier includes a fantastic Python demonstration that plots the function's surface and the tangent lines whose slopes are given by the partial derivatives.

This is a great opportunity to connect the mathematical theory to a practical coding implementation, which is a core part of your learning goal.

What Partial Derivatives Are (Hands-on Introduction) — Topic 67 of Machine Learning Foundations

Let's return to Jon Krohn's video. He will now use Python, NumPy, and Matplotlib to visualize the partial derivatives we calculated earlier for the function z = x^2 - y^2. This will make the concept very concrete.

This is a longer segment, so take your time. You don't need to code along line-by-line, but focus on understanding the workflow: Visualizing ∂z/∂x (08:37 - 21:28): Observe how he holds y constant in the code to generate the 2D slice of the curve, then uses the derivative function 2*x to plot the tangent lines. Visualizing ∂z/∂y (21:28 - 29:10): Watch how the same process is repeated for the y variable, this time using the derivative -2*y. The key is to see how the abstract math translates directly into code that produces a meaningful visualization.

The Gradient Vector

You've now calculated the partial derivatives of a function with respect to each of its variables. So, what do we do with them? We collect them into a vector called the gradient.

For a function $f (x, y)$ , the gradient, denoted $\nabla f$ (pronounced "nabla f" or "del f"), is:
$\nabla f (x, y) = [\frac{\partial f}{\partial x} \frac{\partial f}{\partial y}]$

For the function $f (x, z) = 2 z^{3} x^{2}$ from our earlier example, the gradient is:
$\nabla f (x, z) = [4 z^{3} x 6 x^{2} z^{2}]$

The gradient is one of the most important concepts in machine learning. While each partial derivative gives us the slope along one axis, the gradient vector points in the direction of the steepest ascent on the function's surface. To find the minimum of a function (like minimizing a loss function), we will simply take steps in the direction opposite to the gradient. This algorithm is called Gradient Descent, and it is the topic of an upcoming lesson.

22.4. Multivariable Calculus

To see a more formal definition, let's look at the 'Dive into Deep Learning' text. It defines the gradient and shows how it's used to approximate the function's value.

Read the section '22.4.1. Higher-Dimensional Differentiation', focusing on the part that begins with 'This may look like a mess...'. Specifically, look at equation (22.4.4) which formally defines the gradient vector ( abla_{\mathbf{x}} L).

Conclusion

In this lesson, we made the critical leap from single-variable calculus to multivariable calculus, a necessary step for understanding how machine learning models learn.

Key Takeaways:

Functions of multiple variables, like $f (x, y)$ , define surfaces in higher-dimensional space.
A partial derivative measures the function's rate of change along one specific axis, holding all other variables constant.
To calculate a partial derivative (e.g., $\frac{\partial f}{\partial x}$ ), we apply standard derivative rules while treating all other variables ( $y, z, \dots$ ) as constants.
The gradient ( $\nabla f$ ) is a vector that bundles together all the partial derivatives of a function. It is the cornerstone of many optimization algorithms in machine learning.

Preview of the Next Lesson:
We now know how to find the derivative of a simple function ( $x^{n}$ ) and a multivariable function ( $f (x, y)$ ). But what about a "function of a function," also known as a composite function? For example, how would we find the derivative of something like $sin (x^{2})$ ? This is extremely common in neural networks, where the output of one layer becomes the input to the next.

In our next lesson, we will learn how to apply the chain rule for composite functions, which will give us the final fundamental tool needed to understand how a neural network learns.