Classifying Critical Points with Gradient and Hessian

Recommended resources

The freely available resources this lesson will direct you to.

In our previous lesson, we developed the multivariable Taylor expansion, a powerful tool for creating polynomial approximations of complex functions. We saw how the gradient dictates the linear (tangent plane) approximation and how the Hessian matrix governs the quadratic (curvature) approximation. Now, we will put these tools to their ultimate use in this module.

This lesson marks the culmination of your journey through multivariable calculus. We will synthesize everything you've learned—partial derivatives, gradients, Hessians, and Taylor series—to address one of the most fundamental problems in optimization and machine learning: finding and classifying the extrema of a function. By the end of this lesson, you will be able to use the gradient and Hessian to systematically identify points of interest and determine whether they are local minima, local maxima, or saddle points.

The First Derivative Test: Finding Critical Points

Imagine you are standing on a hilly terrain represented by a function $f (x)$ . A local minimum is a valley floor, a local maximum is a hilltop, and a saddle point is a mountain pass. What geometric feature do all three of these points share?

At any of these locations, the ground is momentarily flat. In mathematical terms, the tangent plane to the function's surface is horizontal. Since the gradient vector defines the orientation of the tangent plane, a horizontal plane means the gradient must be the zero vector. This gives us our first necessary condition.

A critical point of a function $f (x)$ is a point $c$ in its domain where the gradient is zero:

\nabla f (c) = 0

This is the first derivative test in higher dimensions. Finding these points is the first step in any optimization problem, as any local extremum must be a critical point. However, as the existence of saddle points shows, not every critical point is an extremum.

[PDF] CRITICAL POINTS Contents 1. Univariate functions 1 2. The ...

This document from McGill University formalizes this concept.

Please read the section titled "2. The gradient test". Focus on Theorem 2.1 (Fermat's theorem) and Definition 2.2. This establishes the necessary condition that the gradient must be zero at a local maximum or minimum.

The Second Derivative Test: Classifying Critical Points

Once we've found a critical point $c$ , how do we classify it? This is where the second-order Taylor expansion becomes indispensable. Recall the formula from our last lesson for approximating the function near $c$ :

f (c + δ x) \approx f (c) + \nabla f (c) \cdot δ x + \frac{1}{2} δ x^{T} H_{f} (c) δ x

Since $c$ is a critical point, $\nabla f (c) = 0$ . The formula simplifies dramatically. The change in the function's value as we move a small step $δ x$ away from $c$ is given by:

f (c + δ x) - f (c) \approx \frac{1}{2} δ x^{T} H_{f} (c) δ x

This is the key insight: the local behavior of the function around a critical point is entirely governed by the quadratic form defined by the Hessian matrix at that point.

The nature of the critical point now depends on the sign of this quadratic term:

Local Minimum: If $δ x^{T} H_{f} (c) δ x > 0$ for any non-zero step $δ x$ , it means moving in any direction increases the function's value. The surface is shaped like a bowl pointing up.
Local Maximum: If $δ x^{T} H_{f} (c) δ x < 0$ for any non-zero step $δ x$ , it means moving in any direction decreases the function's value. The surface is shaped like a dome pointing down.
Saddle Point: If $δ x^{T} H_{f} (c) δ x$ is positive for some directions $δ x$ and negative for others, it means the function goes up in some directions and down in others. This is a mountain pass.

This brings us to the formal Second Derivative Test, which connects this behavior to the properties of the Hessian matrix.

Definiteness and Eigenvalues

The behavior of the quadratic form $v^{T} H v$ is so important that it has its own terminology. A symmetric matrix $H$ is:

Positive Definite if $v^{T} H v > 0$ for all $v \neq = 0$ .
Negative Definite if $v^{T} H v < 0$ for all $v \neq = 0$ .
Indefinite if $v^{T} H v$ takes on both positive and negative values.

In our linear algebra module, you learned that the eigenvalues of a symmetric matrix tell us how it stretches space. This stretching behavior directly determines the matrix's definiteness.

Positive Definite $⟺$ All eigenvalues are positive.
Negative Definite $⟺$ All eigenvalues are negative.
Indefinite $⟺$ There is at least one positive and one negative eigenvalue.

If any eigenvalue is zero, the test is inconclusive, and we'd need to examine higher-order derivatives.

The following reading formalizes this connection between eigenvalues and the classification of critical points.

[PDF] CRITICAL POINTS Contents 1. Univariate functions 1 2. The ...

Let's return to the "Critical Points" document to see the complete test.

Read the section "8. The Hessian test". Start by looking at the quadratic functions in Example 8.1 and their shapes. Then, read the text that links the behavior to eigenvalues in Remark 8.2. Finally, absorb the formal statement in Theorem 8.4. This theorem is the core of this lesson.

The video below provides a great intuitive summary of the first and second derivative tests, tying all these concepts together.

Gradients, Hessians, and All Those Derivative Tests

The video "Gradients, Hessians, and All Those Derivative Tests" by OptWhiz explains the application to optimization.

Watch the section from the first derivative test to the conclusion of the second derivative test. The narrator explains why the gradient must be zero (first derivative test) and then explains how the sign of the directional second derivative (determined by the Hessian's definiteness) classifies the critical point.

A Practical Test for 2D Functions

Calculating eigenvalues for large matrices is computationally intensive. For the common case of functions with two variables ( $f (x, y)$ ), there is a much simpler test that uses the determinant of the 2x2 Hessian matrix.

Let the Hessian at a critical point be $H = (f_{xx} f_{y x} f_{x y} f_{y y})$ and let $D = det (H) = f_{xx} f_{y y} - f_{x y}^{2}$ .

The signs of the eigenvalues $λ_{1}, λ_{2}$ are related to the determinant ( $D = λ_{1} λ_{2}$ ) and the trace ( $tr (H) = λ_{1} + λ_{2} = f_{xx} + f_{y y}$ ). This gives us the following rules:

If $D > 0$ and $f_{xx} > 0$ : Both eigenvalues are positive. $H$ is positive definite $⟹$ Local Minimum.
If $D > 0$ and $f_{xx} < 0$ : Both eigenvalues are negative. $H$ is negative definite $⟹$ Local Maximum.
If $D < 0$ : The eigenvalues have opposite signs. $H$ is indefinite $⟹$ Saddle Point.
If $D = 0$ : At least one eigenvalue is zero. The test is inconclusive.

The image below shows the classification for the function $f (x, y) = 2 x^{2} - x^{3} - y^{2}$ . Notice the local maximum at $(1.33, 0)$ and the saddle point at $(0, 0)$ , along with the determinant test results.

This visualization shows a function's surface on the right. On the left, a table lists the critical points, the values of $f_{xx}$, and the determinant of the Hessian, correctly classifying one point as a saddle point and the other as a local maximum.

Worked Example

Let's apply this to find and classify the critical points of $f (x, y) = x^{3} - 12 x + y^{2}$ .

Step 1: Find Critical Points (First Derivative Test)
First, we compute the gradient and set it to zero.

\nabla f (x, y) = [\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}] = [3 x^{2} - 12, 2 y]

Setting $\nabla f = 0$ gives us a system of equations:
\begin{align*}
3x^2 - 12 &= 0 \implies x^2 = 4 \implies x = \pm 2 \
2y &= 0 \implies y = 0
\end{align*}
So, we have two critical points: $(2, 0)$ and $(- 2, 0)$ .

Step 2: Compute the Hessian
Next, we find the matrix of second partial derivatives.

H_{f} (x, y) = (f_{xx} f_{y x} f_{x y} f_{y y}) = (6 x 0 02)

Step 3: Classify Critical Points (Second Derivative Test)
Now we evaluate the Hessian at each critical point.

At point (2, 0):
$H_{f} (2, 0) = (6 (2) 0 02) = (12002)$
The determinant is $D = (12) (2) - (0) (0) = 24$ .
Since $D > 0$ and $f_{xx} = 12 > 0$ , the point $(2, 0)$ is a local minimum.
At point (-2, 0):
$H_{f} (- 2, 0) = (6 (- 2) 0 02) = (- 12 0 02)$
The determinant is $D = (- 12) (2) - (0) (0) = - 24$ .
Since $D < 0$ , the point $(- 2, 0)$ is a saddle point.

This systematic procedure allows us to fully characterize the local geometry of a function's surface.

Conclusion

This lesson completes our foundational study of multivariable calculus. You have seen how the concepts of derivatives, gradients, and curvature build upon one another to provide a complete framework for analyzing and approximating functions in higher dimensions.

Key Takeaways:

Critical points, where $\nabla f = 0$ , are candidates for local minima, maxima, or saddle points. This is the First Derivative Test.
The Second Derivative Test uses the Hessian matrix at a critical point to classify it. The behavior is determined by the quadratic form $δ x^{T} H_{f} (c) δ x$ .
The classification depends on the definiteness of the Hessian, which in turn is determined by the signs of its eigenvalues:
- All positive eigenvalues $⟹$ Local Minimum
- All negative eigenvalues $⟹$ Local Maximum
- Mixed positive and negative eigenvalues $⟹$ Saddle Point
For 2D functions, a practical test using the determinant of the Hessian provides a convenient shortcut.

With the tools of linear algebra and multivariable calculus now firmly in your grasp, you are prepared to tackle the core concepts of machine learning.

Preview of the Next Lesson:
We will now pivot from the deterministic world of calculus to the world of uncertainty and data. Our next module, Essentials of Probability and Information Theory, begins by establishing the fundamental axioms of probability. This will be the first step in building the language needed to describe, model, and learn from data, which is at the heart of all modern AI.