Partial Derivatives of Multivariable Functions

Recommended resources

The freely available resources this lesson will direct you to.

Introduction

In our last lesson, we established a powerful way to think about the derivative of a single-variable function: it defines the function's best local linear approximation. This concept of "zooming in" until a curve looks like a straight line is the key to generalizing calculus. Today, we take the first and most crucial step into higher dimensions.

We will move from functions of one variable, $f (x)$ , to functions of multiple variables, such as $f (x, y)$ or $f (x_{1}, x_{2}, \dots, x_{n})$ . These functions are everywhere in machine learning and graphics, from a cost function that depends on many model weights to a function describing the color and density at a point $(x, y, z)$ in space.

The question immediately arises: What is "the" derivative of such a function? If the input space is a plane, we can move in infinitely many directions from any given point. The slope of the function's "surface" will be different depending on which direction we choose. This lesson will introduce you to partial derivatives, which are our tool for measuring a function's rate of change along the primary coordinate axes. This is the foundational concept upon which we will build the gradient, a cornerstone of optimization.

From 1D Slopes to 2D Slices

For a function of one variable, $f (x)$ , the derivative $f^{'} (x)$ tells us how the output changes in response to a small nudge in the single possible input direction. For a function of two variables, say $f (x, y)$ , the input is a point $(x, y)$ in a plane. A nudge to this input could be purely in the x-direction, purely in the y-direction, or in any combination.

The core idea of partial differentiation is to simplify the problem by considering only one direction at a time. To find the rate of change with respect to $x$ , we can imagine "freezing" the value of $y$ . By holding $y$ constant, say at a value $y_{0}$ , the function $f (x, y_{0})$ behaves just like a single-variable function of $x$ . Its derivative is then a familiar concept.

Geometrically, if you visualize the function $f (x, y)$ as a surface in 3D space, holding $y$ constant is equivalent to taking a vertical slice through that surface parallel to the x-z plane. The partial derivative with respect to $x$ is simply the slope of the curve created by this slice at a given point.

This image perfectly illustrates the concept. The surface represents a function $f(\mathbf{x})$ of two variables, $x_1$ and $x_2$. By holding $x_2$ constant, we create a 2D slice (the black curve). The partial derivative $\frac{\partial f(\mathbf{x})}{\partial x_1}$ is the slope of the tangent line (red dashed line) to this curve.

Defining and Computing Partial Derivatives

This "freezing" of variables gives us a clear computational strategy. The following video from Khan Academy provides an excellent intuitive and practical introduction to this idea.

Partial derivatives, introduction

This video, "Partial derivatives, introduction," explains the concept by extending the idea of a "tiny nudge" from single-variable calculus to multivariable functions.

Please watch these segments: First, watch the introduction (the concept) to understand the core idea of restricting movement to one axis and the new \partial notation. Next, see how this is used to compute a partial derivative at a specific point (at a point) by plugging in the constant value before differentiating. Finally, learn the general procedure (the general formula), where you simply treat the other variables as constants during differentiation.

As the video explained, the formal definition of the partial derivative of $f (x, y)$ with respect to $x$ is a natural extension of the single-variable limit definition:

\frac{\partial f}{\partial x} (x, y) = h \to 0 lim \frac{f ( x + h , y ) - f ( x , y )}{h}

Notice that the $y$ coordinate is identical in both terms in the numerator. We are only changing $x$ . Symmetrically, the partial derivative with respect to $y$ is:

\frac{\partial f}{\partial y} (x, y) = h \to 0 lim \frac{f ( x , y + h ) - f ( x , y )}{h}

The key takeaway is the computational rule: To find the partial derivative of a function with respect to a variable, treat all other variables as constants and apply the standard rules of single-variable differentiation.

Let's work through an example. Consider the function $f (x, y) = x^{3} y^{2} + 2 x^{2} + sin (y)$ .

Question: Based on the rule above, how would you compute $\frac{\partial f}{\partial x}$ and $\frac{\partial f}{\partial y}$ ?

For $\frac{\partial f}{\partial x}$ , think of every instance of $y$ as just a constant number, like 5. How would you differentiate $x^{3} (5^{2}) + 2 x^{2} + sin (5)$ with respect to $x$ ?
For $\frac{\partial f}{\partial y}$ , think of $x$ as a constant. How would you differentiate $(c^{3}) y^{2} + 2 c^{2} + sin (y)$ with respect to $y$ ?

Solution

Partial derivative with respect to $x$ :
- We treat $y$ as a constant.
- The term $x^{3} y^{2}$ is like $(constant) \cdot x^{3}$ . Its derivative with respect to $x$ is $y^{2} \cdot (3 x^{2})$ .
- The term $2 x^{2}$ has derivative $4 x$ .
- The term $sin (y)$ is treated as a pure constant, so its derivative with respect to $x$ is 0.
- Combining these, we get: $\frac{\partial f}{\partial x} = 3 x^{2} y^{2} + 4 x$ .
Partial derivative with respect to $y$ :
- We treat $x$ as a constant.
- The term $x^{3} y^{2}$ is like $(constant) \cdot y^{2}$ . Its derivative with respect to $y$ is $x^{3} \cdot (2 y)$ .
- The term $2 x^{2}$ is a pure constant, so its derivative is 0.
- The term $sin (y)$ has derivative $cos (y)$ .
- Combining these, we get: $\frac{\partial f}{\partial y} = 2 x^{3} y + cos (y)$ .

Advanced Rules and Deeper Interpretation

All the familiar rules from single-variable calculus, such as the product rule and chain rule, still apply. The next video provides a more rigorous walk-through and demonstrates how to handle these more complex cases.

Oxford Calculus: Partial Differentiation Explained with Examples

This video from Oxford University, "Partial Differentiation Explained," demonstrates the computation process, including for more complex functions.

Please watch two key sections: The demonstration of computing first-order partial derivatives (the computation), which reinforces the main technique. The application of the product and chain rules (advanced rules). This is crucial for handling the complex functions we'll encounter in neural networks.

Your background in computer science and engineering attunes you to the importance of precise definitions. Let's refine our understanding using an idea from the provided text, which distinguishes a function from a formula.

Partial derivatives

This document from the University of Tennessee, Knoxville provides a rigorous mathematical perspective on partial derivatives.

Please read the sections titled "Clarification of the function concept and appropriate notation" and "Interpretation of partial derivatives in the graph of a function". Focus on the "slot machine" analogy. A multivariable function f(·, ·) has multiple input slots. When we compute a partial derivative like \frac{\partial f}{\partial x}, we are conceptually creating a new, single-variable function by fixing the second slot (e.g., f(·, y₀)). The partial derivative is simply the ordinary derivative of this new function.

This perspective is more than just a notational subtlety. It highlights that a partial derivative only tells part of the story. As the document mentions, a function can have well-defined partial derivatives at a point but still be discontinuous or "wild" in other directions. This is a crucial insight that motivates the need for a more complete picture of differentiability, which we will explore in future lessons.

Conclusion

In this lesson, we have extended the concept of the derivative to functions of multiple variables. By focusing on change one dimension at a time, we defined and learned to compute partial derivatives.

Key Takeaways:

A multivariable function's rate of change depends on the direction of movement in its input space.
A partial derivative measures this rate of change along one of the primary coordinate axes (e.g., with respect to $x$ while holding $y$ and $z$ constant).
Computation is straightforward: treat all variables, except the one you are differentiating with respect to, as constants and apply standard differentiation rules.
Geometrically, a partial derivative represents the slope of a curve formed by slicing the function's graph parallel to a coordinate plane.

You now have the primary tool for analyzing how a function with many inputs responds to changes in each of those inputs individually. However, these partial derivatives are just scalar components of a larger, more powerful object. What happens when we assemble them into a vector?

In our next lesson, we will do exactly that. We will define the gradient as the vector of all partial derivatives. This vector will give us the complete picture of local change, pointing in the direction of the steepest ascent of the function—the single most important direction for optimization.