Gradient Vectors and Steepest Ascent

Introduction

In our last lesson, you mastered the computation of partial derivatives, which tell us how a multivariable function changes as we move along the primary axes (like the x-direction or y-direction). This was an essential step into multivariable calculus.

However, on a landscape, the steepest path uphill is rarely due east or due north. It's usually some combination of the two. This lesson tackles that very idea. Our goal is to explain the concept of a gradient vector and its relation to the direction of steepest ascent.

We will define a new object, the gradient vector, by combining all the partial derivatives of a function. You will learn that this vector has a powerful geometric meaning: it always points in the direction where the function's value increases the fastest. This concept is not just a mathematical curiosity; it is the absolute foundation for how most modern AI models learn. Understanding this is a key step toward your goal of mastering AI.

1. Defining the Gradient Vector

Let's start by formally defining the gradient. In the previous lesson, you calculated partial derivatives like $\frac{\partial f}{\partial x}$ and $\frac{\partial f}{\partial y}$ individually. The gradient simply collects all of these partial derivatives into a single vector.

For a function of two variables $f (x, y)$ , the gradient is denoted by $\nabla f$ (pronounced "del f" or "grad f") and is defined as:

\nabla f (x, y) = [\frac{\partial f}{\partial x} \frac{\partial f}{\partial y}] = (\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y})

For a function of three variables $f (x, y, z)$ , it's what you'd expect:

\nabla f (x, y, z) = (\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}, \frac{\partial f}{\partial z})

Essentially, the gradient is a vector-valued function that, for any point in the input space, gives you a vector. Let's ground this with a quick recap of the computation.

Calculus — ML Glossary documentation

The ML Cheatsheet provides a concise definition of the gradient and walks through a clear, step-by-step example of how to construct it from partial derivatives.

Please read the sections titled 'Gradients', 'Partial derivatives', and 'Step-by-step'. This will refresh your memory from the last lesson and show you how the partial derivatives are assembled into the final gradient vector.

Now that we know what the gradient is, let's explore what it does.

2. The Geometric Interpretation: Direction of Steepest Ascent

The gradient's true power lies in its geometric meaning. At any given point, the gradient vector points in the direction you should move to increase the function's value most quickly.

Imagine you are standing on a hillside described by a function $f (x, y)$ . The gradient at your location is a vector on the ground (in the xy-plane) that points you directly up the steepest path.

To build a solid intuition for this, let's watch a video that uses this very analogy.

Geometric Meaning of the Gradient Vector

Dr. Trefor Bazett provides an excellent visual explanation using the analogy of climbing a mountain. He introduces 'level curves' (or contours), which are paths of constant height, and sets up the central question: which direction is steepest?

Watch the first 2 minutes and 25 seconds of the video. Pay close attention to the relationship between the 3D mountain surface and the 2D contour plot. This is a crucial visualization tool in machine learning.

The video shows that moving along a level curve results in zero change in height. Intuitively, the steepest path must be perpendicular to these level curves. The gradient vector provides exactly this direction.

Look at the following image. It shows the 2D contour plot for a bowl-shaped function, $f (x, y) = a (x^{2} + y^{2}) + b$ . The circles are the level curves (constant height). The arrows represent the gradient vectors at various points. Notice how each arrow is perpendicular to the contour line it starts on, pointing "outward" toward higher values of the function.

Gradient Vector Fields Visualization — This image shows both a 2D contour plot (left) and a 3D surface plot (right). The arrows are the gradient vectors, which always point perpendicular to the contour lines in the direction of steepest ascent.

3. The Mathematical Proof: Maximizing the Directional Derivative

We now have the intuition that the gradient points in the direction of steepest ascent. As you prefer a theory-first approach, let's prove this mathematically.

The tool we need is the directional derivative, which we'll denote as $D_{u} f$ . It measures the rate of change (the slope) of the function $f$ at a point, in an arbitrary direction given by a unit vector $u$ .

A key formula in multivariable calculus states that the directional derivative is the dot product of the gradient and the direction vector:

D_{u} f = \nabla f \cdot u

Our question—"Which direction maximizes the rate of change?"—becomes a mathematical one: "For which unit vector $u$ is the dot product $\nabla f \cdot u$ maximized?"

Recall the geometric definition of the dot product:

\nabla f \cdot u = ∣∣\nabla f ∣∣ \cdot ∣∣ u ∣∣ \cdot cos (θ)

where $θ$ is the angle between the gradient vector $\nabla f$ and the direction vector $u$ .

Since $u$ is a unit vector, its magnitude $∣∣ u ∣∣$ is 1. The formula simplifies to:

D_{u} f = ∣∣\nabla f ∣∣ cos (θ)

Now, to maximize this value:

$∣∣\nabla f ∣∣$ is a fixed value at any given point.
The only thing we can change is the direction $u$ , which changes the angle $θ$ .
The value of $cos (θ)$ is maximized when it equals 1. This occurs when $θ = 0$ .

An angle of $θ = 0$ means that the direction vector $u$ points in the exact same direction as the gradient vector $\nabla f$ .

This proves our central point: the rate of change is greatest in the direction of the gradient.

The following video provides a crystal-clear walkthrough of this exact argument.

Why the gradient is the direction of steepest ascent

Khan Academy breaks down the proof step-by-step, focusing entirely on how the properties of the dot product lead to the conclusion that the gradient is the direction of steepest ascent.

Watch the segment from 02:09 to 07:46. This is the mathematical core of today's lesson. Focus on how the problem is framed as maximizing the directional derivative.

4. The Magnitude of the Gradient

We've established the direction of the gradient. But what does its magnitude, $∣∣\nabla f ∣∣$ , represent?

Let's go back to our formula for the directional derivative: $D_{u} f = ∣∣\nabla f ∣∣ cos (θ)$ .

When we move in the direction of steepest ascent, $θ = 0$ and $cos (θ) = 1$ . The formula becomes:

D_{steepest} f = ∣∣\nabla f ∣∣ \cdot 1 = ∣∣\nabla f ∣∣

This gives us the second crucial property of the gradient:
The magnitude of the gradient, $∣∣\nabla f ∣∣$ , is the rate of change in the direction of steepest ascent.

So, a large gradient magnitude means the "hill" is very steep at that point, while a small magnitude means it's relatively flat.

This concept is also clearly explained in a reading resource that ties together the ideas we've discussed.

Demystifying Calculus: Why is the Gradient the Maximum Value of the Directional Derivative

This article from Renda Zhang on Medium provides a great textual summary of the concepts, including a concrete example. It's a good way to reinforce what you've seen in the videos.

Please read the sections 'Geometric Connection Between Gradient and Directional Derivative', 'The Gradient as the Maximum Value of Directional Derivative', and the worked-out 'Example' using the function f(x, y) = x^2 + y^2. This will solidify your understanding of the proof and its implications.

Finally, to see this applied to a real-world problem, let's return to Dr. Bazett's video.

Geometric Meaning of the Gradient Vector

This final segment applies the concepts of level curves and gradients to a real topographical map of a mountain, which is a fantastic real-world application.

Watch the clip from 11:11 to 14:23. Notice how the closeness of the contour lines relates to the steepness of the terrain, and thus to the magnitude of the gradient.

Conclusion

Today, you've moved from calculating simple rates of change along axes to understanding direction and steepness in any dimension. This is a significant leap in your mathematical toolkit for AI.

Key Takeaways:

The gradient vector, $\nabla f$ , is a vector containing all the partial derivatives of a function $f$ .
The gradient's direction points in the direction of the function's steepest ascent at any given point. Geometrically, this is always perpendicular to the level curve at that point.
The gradient's magnitude, $∣∣\nabla f ∣∣$ , represents the actual slope, or rate of change, in that steepest direction.
This relationship is proven by maximizing the directional derivative, $D_{u} f = \nabla f \cdot u$ , which occurs when the direction vector $u$ is aligned with the gradient $\nabla f$ .

Preview of the Next Lesson:
We've established that the gradient, $\nabla f$ , points in the direction of steepest ascent. In machine learning, our goal is almost always to minimize an error or "loss" function, not maximize it. It follows logically that the direction of steepest descent is simply the opposite of the gradient, $- \nabla f$ .

In our next lesson, you will apply this principle to find the minimum of a simple function on paper. This technique, known as gradient descent, is the single most important optimization algorithm in machine learning and is how we train everything from simple models to the most advanced large language models.