Calculating Partial Derivatives

Hello! Welcome to the first lesson in our new module on Multivariate Calculus for Machine Learning.

Introduction

In our last module, we explored how linear algebra, through concepts like eigendecomposition and SVD, allows us to understand and manipulate data matrices. We concluded by seeing how SVD can find the best low-rank approximation of a matrix, a key tool in dimensionality reduction and compression.

Now, we pivot from the linear world of matrices to the continuous world of multivariable functions. This module is the mathematical engine behind training almost all modern machine learning models. The central idea is optimization: finding the set of model parameters that minimizes a cost or error function. To do that, we first need to understand how the function's value changes as we adjust its parameters.

Today, we address the first learning outcome of this module: to compute partial derivatives of multivariable functions. A partial derivative is the natural extension of the single-variable derivative you are familiar with. It measures the instantaneous rate of change of a function with respect to one of its variables, while holding all other variables constant. This is the fundamental building block for the optimization algorithms we will study later.

From Single to Multiple Variables

Recall that for a function of one variable, $f(x)$, the derivative $f^{\prime }(x)$ or $\frac{df}{dx}$ gives us the slope of the tangent line to the function's graph at point $x$. It tells us how sensitive $f$ is to a small change in $x$.

But what about a function of multiple variables, like $f(x,y)$? This function's graph is no longer a curve, but a surface in 3D space. At any point $(x_0,y_0)$ on this surface, there isn't just one "slope"—we could move in the x-direction, the y-direction, or any combination.

To get a handle on this, let's first build our intuition for what these functions and their graphs look like.

As the video discusses, visualizing these functions is key. We have two primary methods:

1. Graphs: We can plot $z=f(x,y)$ as a surface in 3D space. The height $z$ of the surface above the xy-plane at point $(x,y)$ represents the function's value.
2. Contour Plots (Level Curves): We can draw curves in the 2D xy-plane that connect all points where the function has a constant value, like a topographical map showing lines of constant elevation.

The core idea of partial differentiation is to simplify the problem by considering only one direction of change at a time—specifically, change along the coordinate axes.

The Partial Derivative: Definition and Geometric Meaning

To find the rate of change of $f(x,y)$ in the x-direction, we can simply hold $y$ constant at some value $y_0$. The function effectively becomes a single-variable function of $x$, $g(x)=f(x,y_0)$. The derivative of this "partial function" $g(x)$ is what we call the partial derivative of $f$ with respect to $x$.

Notation:
The partial derivative of $f$ with respect to $x$ is denoted by:
$\frac{\partial f}{\partial x}\text{or}{f}_x$
The curly 'd' symbol ($\partial$, called "del" or "partial") is crucial. It signifies that $f$ is a function of multiple variables and we are only differentiating with respect to one of them, holding the others constant.

Formal Definition:
Just like the ordinary derivative, the partial derivative is defined with a limit. The partial derivative of $f(x,y)$ with respect to $x$ at a point $(x,y)$ is:
$\frac{\partial f}{\partial x}={\lim }_{h\to 0}\frac{f(x+h,y)-f(x,y)}{h}$
Similarly, the partial derivative with respect to $y$ is:
$\frac{\partial f}{\partial y}={\lim }_{h\to 0}\frac{f(x,y+h)-f(x,y)}{h}$
Notice that in the definition for $\frac{\partial f}{\partial x}$, the $y$ coordinate remains unchanged, and vice-versa.

Geometric Interpretation:
Geometrically, holding $y$ constant at $y_0$ is equivalent to slicing the 3D surface $z=f(x,y)$ with a vertical plane at $y=y_0$. The intersection of the plane and the surface is a curve. The partial derivative $\frac{\partial f}{\partial x}$ at $(x_0,y_0)$ is simply the slope of the tangent line to that specific curve at that point.

To see this concept in motion, the following video provides an excellent visualization.

The Mechanics of Computation

While the limit definition is fundamental for theory, in practice we almost never use it for computation. The rule is much simpler.

To compute a partial derivative, treat all other variables as constants and apply the standard rules of single-variable differentiation.

For example, to compute $\frac{\partial }{\partial x}$ of $f(x,y)=x^2{y}^3$:

1. We are differentiating with respect to $x$.
2. Therefore, we treat $y$ as a constant. The term $y^3$ is just a constant multiplier.
3. The derivative of $x^2$ with respect to $x$ is $2x$.
4. So, $\frac{\partial f}{\partial x}=(2x)\cdot y^3=2x{y}^3$.

Now, let's compute $\frac{\partial }{\partial y}$ of the same function, $f(x,y)=x^2{y}^3$:

1. We are differentiating with respect to $y$.
2. We treat $x$ as a constant. The term $x^2$ is a constant multiplier.
3. The derivative of $y^3$ with respect to $y$ is $3y^2$.
4. So, $\frac{\partial f}{\partial y}=x^2\cdot (3y^2)=3x^2{y}^2$.

The following resource provides a clear walkthrough of this process.

The rules you know from single-variable calculus, like the product rule, quotient rule, and chain rule, all apply directly when computing partial derivatives.

Conclusion

In this lesson, we have taken the first crucial step into multivariate calculus for machine learning. We have extended the concept of a derivative to functions of multiple variables by defining the partial derivative.

Key Takeaways:

• A multivariable function $f(x_1,\dots ,x_n)$ can be visualized as a surface or through its level curves.
• The partial derivative $\frac{\partial f}{\partial x_i}$ measures the instantaneous rate of change of the function as only the variable $x_i$ changes.
• Geometrically, it represents the slope of the tangent line to the curve formed by slicing the function's graph with a plane that holds all other variables constant.
• Computationally, we find $\frac{\partial f}{\partial x_i}$ by treating all other variables as constants and applying the familiar rules of single-variable differentiation.

Preview of the Next Lesson:
We have learned to measure the rate of change along each coordinate axis individually. But what if we want to know the direction of the steepest ascent on our function's surface? In the next lesson, we will combine all the partial derivatives of a function into a single vector called the gradient. This vector, $\nabla f$, points in the direction of the greatest rate of increase of the function and is the cornerstone of gradient-based optimization methods that train machine learning models.