Hessian Matrix and Curvature Analysis

Recommended resources

The freely available resources this lesson will direct you to.

In our previous lesson, we established the Jacobian as the matrix of all first-order partial derivatives, representing the best linear approximation of a function at a point. This tells us about the local "slope" or "gradient" of the function's landscape. However, to understand the geometry of this landscape—whether we are at the bottom of a valley, the peak of a hill, or on a Pringle-shaped saddle—we need to analyze its curvature. This requires moving to second derivatives.

This lesson introduces the Hessian matrix, the multivariable generalization of the second derivative. We will explore how to compute it and, more importantly, how to interpret its structure to understand the local concavity and curvature of a function. This concept is fundamental to optimization, as it allows us to classify the critical points found by setting the gradient to zero.

From First to Second Derivatives: The Hessian Matrix

For a single-variable function $f (x)$ , the second derivative $f^{''} (x)$ tells us about concavity. If $f^{''} (x) > 0$ , the function is concave up (like a bowl), and if $f^{''} (x) < 0$ , it's concave down (like a dome).

For a scalar-valued function of multiple variables, $f (x)$ where $x \in R^{n}$ , we have many second-order partial derivatives: $\frac{\partial ^{2} f}{\partial x _{1}^{2}}$ , $\frac{\partial ^{2} f}{\partial x _{1} \partial x _{2}}$ , etc. Just as the Jacobian organized all first-order partials, the Hessian matrix, denoted $H_{f}$ , organizes all second-order partial derivatives.

For a function $f : R^{n} \to R$ , the Hessian is an $n \times n$ matrix defined as:

H_{f} (x) = \frac{\partial ^{2} f}{\partial x _{1}^{2}} \frac{\partial ^{2} f}{\partial x _{2} \partial x _{1}} ⋮ \frac{\partial ^{2} f}{\partial x _{n} \partial x _{1}} \frac{\partial ^{2} f}{\partial x _{1} \partial x _{2}} \frac{\partial ^{2} f}{\partial x _{2}^{2}} ⋮ \frac{\partial ^{2} f}{\partial x _{n} \partial x _{2}} \dots \dots ⋱ \dots \frac{\partial ^{2} f}{\partial x _{1} \partial x _{n}} \frac{\partial ^{2} f}{\partial x _{2} \partial x _{n}} ⋮ \frac{\partial ^{2} f}{\partial x _{n}^{2}}

The entry at position $(i, j)$ is $H_{ij} = \frac{\partial ^{2} f}{\partial x _{i} \partial x _{j}}$ .

A key property for most functions encountered in machine learning is that the mixed partial derivatives are equal (e.g., $\frac{\partial ^{2} f}{\partial x _{1} \partial x _{2}} = \frac{\partial ^{2} f}{\partial x _{2} \partial x _{1}}$ ). This is guaranteed by Clairaut's theorem for functions with continuous second partial derivatives. This makes the Hessian matrix symmetric. As you recall from our linear algebra module, symmetric matrices have very special properties, including having only real eigenvalues and an orthogonal basis of eigenvectors. This is not a coincidence and is crucial for interpreting curvature.

Let's watch a short video that walks through a concrete calculation of a Hessian matrix.

M4ML - Multivariate Calculus - 2.7 The Hessian

The video "M4ML - Multivariate Calculus - 2.7 The Hessian" from Imperial College London provides a clear, step-by-step computation.

Please watch the segment from the example calculation. The instructor first computes the Jacobian (the vector of first derivatives) and then differentiates its components again to build the Hessian. Notice how the symmetry of the final matrix becomes apparent.

The process is mechanical: you compute the gradient first, then you compute the gradient of each component of the gradient vector to form the rows of the Hessian.

This diagram illustrates the computational path for the Hessian. The function $f(x,y)$ is first differentiated with respect to $x$ and $y$ to get the first partials. Each of these is then differentiated again with respect to both $x$ and $y$ to yield the four entries of the 2x2 Hessian matrix.

The Physical Meaning: Curvature in Any Direction

The gradient told us the slope in any direction via the directional derivative $\nabla f \cdot v$ . The Hessian gives us the curvature in any direction.

For a unit vector $v$ , the second directional derivative, which measures how the slope is changing (i.e., the concavity) in the direction $v$ , is given by a quadratic form:

D_{v}^{2} f (x) = v^{T} H_{f} (x) v

This expression is central. It tells us that the Hessian matrix encodes the curvature information for all directions. To find the curvature in a specific direction, we simply "sandwich" the Hessian between the direction vector $v$ .

The following video provides a more formal derivation of this relationship, showing how the Hessian naturally emerges from the definition of a second derivative.

Gradients, Hessians, and All Those Derivative Tests

The video "Gradients, Hessians, and All Those Derivative Tests" by OptWhiz provides a first-principles derivation of the Hessian's role in directional curvature.

Watch the portion from deriving the Hessian. The key idea is to approximate the second derivative along a direction \mathbf{v} by looking at how the first directional derivative changes. This analysis leads directly to the quadratic form \mathbf{v}^T H_f \mathbf{v}.

Interpreting Curvature with Eigenvalues

The relationship $D_{v}^{2} f = v^{T} H_{f} v$ is powerful, but checking it for all possible directions $v$ is impossible. So, how can we characterize the curvature at a point?

This is where the eigendecomposition of the Hessian comes into play. Since $H_{f}$ is a real, symmetric matrix, its eigenvectors form an orthogonal basis that defines the principal axes of curvature at a point. Let's say $(λ, e)$ is an eigenpair of $H_{f}$ , so $H_{f} e = λ e$ .

If we choose our direction vector $v$ to be a unit eigenvector $e$ , the curvature in that direction becomes wonderfully simple:

D_{e}^{2} f = e^{T} H_{f} e = e^{T} (λ e) = λ (e^{T} e) = λ

This is the core insight: The eigenvalues of the Hessian are the curvatures along its principal directions (the eigenvectors).

The eigenvectors tell you in which directions the function curves the most and least, and the corresponding eigenvalues tell you how much it curves in those directions.

Now, consider a critical point $x_{0}$ where $\nabla f (x_{0}) = 0$ . By analyzing the signs of the Hessian's eigenvalues at this point, we can classify it:

All $λ_{i} > 0$ : The function is concave up in all principal directions. The surface looks like a multi-dimensional bowl opening upwards. This is a local minimum. A Hessian with all positive eigenvalues is called positive-definite.
All $λ_{i} < 0$ : The function is concave down in all principal directions, like a dome. This is a local maximum. A Hessian with all negative eigenvalues is called negative-definite.
Mixed signs (some $λ_{i} > 0$ , some $λ_{i} < 0$ ): The function curves up in some directions and down in others. This is a saddle point. A Hessian with mixed eigenvalues is called indefinite.

On the left, a 3D plot of a function with a saddle point at the origin. Notice it curves up along one axis and down along another. On the right, the contour plot shows how the level sets behave near this point, characteristic of an indefinite Hessian.

Some $λ_{i} = 0$ : The curvature is zero in at least one direction. The test is inconclusive. The point could be a minimum, maximum, or a more complex saddle/plateau. The Hessian is called positive/negative semi-definite or degenerate.

To explore this connection between eigenvalues and concavity more deeply, the following resource is excellent.

The Hessian matrix: Eigenvalues, concavity, and curvature

This document provides a rigorous connection between the Hessian, its eigenvalues, and the geometric properties of a surface.

First, read the brief definition of eigenvalues in Definition 3.1. Then, focus on Theorem 3.5 and 3.6, which formally state the eigenvalue test for concavity and for classifying critical points. Finally, review Table 4 in the conclusion, which provides a superb summary of how eigenvalues, the determinant, and concavity all relate.

The Determinant Shortcut (for 2D)

For a 2x2 Hessian, there's a common shortcut that avoids computing eigenvalues directly. It uses the determinant ( $det (H)$ ) and the trace ( $tr (H)$ ). Since you know that $det (H) = λ_{1} λ_{2}$ and $tr (H) = λ_{1} + λ_{2}$ , you can infer the signs of the eigenvalues.

Let $D = det (H) = f_{xx} f_{y y} - f_{x y}^{2}$ .

If $D < 0$ : $λ_{1}$ and $λ_{2}$ must have opposite signs. It's a saddle point.
If $D > 0$ : $λ_{1}$ and $λ_{2}$ have the same sign.
- If $f_{xx} > 0$ (or $f_{y y} > 0$ ), the function is concave up in that direction, so both eigenvalues must be positive. It's a local minimum.
- If $f_{xx} < 0$ (or $f_{y y} < 0$ ), both eigenvalues must be negative. It's a local maximum.
If $D = 0$ : The test is inconclusive.

This "D-test" is just a consequence of the more fundamental eigenvalue properties. Understanding it from the perspective of eigenvalues ensures you grasp the underlying principle that generalizes beyond two dimensions.

Conclusion

In this lesson, we have taken a significant step beyond first-order derivatives. The Hessian matrix provides a rich, second-order description of a function's local geometry.

Key Takeaways:

The Hessian matrix is the $n \times n$ symmetric matrix of a function's second-order partial derivatives.
The curvature (second directional derivative) in any direction $v$ is given by the quadratic form $v^{T} H_{f} v$ .
The eigenvalues of the Hessian represent the curvature along its principal axes (eigenvectors).
The signs of the Hessian's eigenvalues at a critical point allow us to classify it as a local minimum (all positive), local maximum (all negative), or saddle point (mixed signs).

We now have the two key matrices of multivariable calculus: the Jacobian (first-order approximation) and the Hessian (second-order curvature). In our next lesson, we will combine them to build the Taylor expansion of a multivariable function, a powerful tool for approximating complex functions and the foundation for advanced optimization methods like Newton's method.