Computing the Hessian Matrix

Hello! Welcome to the next lesson in our journey through the mathematical foundations of AI.

Introduction

In our previous lesson, we defined and computed the Jacobian matrix, which generalizes the first derivative to vector-valued functions. We established that for a function $\mathbf{f}:{\mathbb{R}}^n\to {\mathbb{R}}^m$, the Jacobian is an $m\times n$ matrix of all first-order partial derivatives.

Today, we will build directly on that foundation to explore second derivatives in a multivariable context. This lesson focuses on defining and computing the Hessian matrix, which is the multivariable analogue of the second derivative for a scalar-valued function. The Hessian is a critical tool in machine learning, forming the bedrock of second-order optimization algorithms (like the Newton-Raphson method) and for analyzing the loss landscape of a model to distinguish between minima, maxima, and saddle points.

Our key insight today will be to understand the Hessian of a scalar function $f$ as the Jacobian of its gradient vector $\nabla f$.

From Gradient to Hessian

Let's formalize this connection. Consider a scalar-valued function $f:{\mathbb{R}}^n\to \mathbb{R}$. We know its first derivative is the gradient, $\nabla f$, which you'll recall is a vector containing all the first-order partial derivatives:

$$\nabla f(\mathbf{x})=\left[\begin{array}{c}\frac{\partial f}{\partial x_1}\\ \frac{\partial f}{\partial x_2}\\ ⋮\\ \frac{\partial f}{\partial x_n}\end{array}\right]$$

Notice that the gradient itself is a vector-valued function, mapping from ${\mathbb{R}}^n$ to ${\mathbb{R}}^n$. From our last lesson, we know that the derivative of a vector-valued function is its Jacobian matrix.

Therefore, the Hessian matrix of $f$, denoted $H_f$ or ${\nabla }^{2}f$, is simply the Jacobian of the gradient function $\nabla f$.

As the notes explain, since the Hessian $H$ is the Jacobian of the gradient $\nabla f:{\mathbb{R}}^n\to {\mathbb{R}}^n$, it will be an $n\times n$ square matrix. Its components are all the second-order partial derivatives of $f$:

$$(H_f{)}_{ij}=\frac{\partial }{\partial x_i}\left(\frac{\partial f}{\partial x_j}\right)=\frac{{\partial }^{2}f}{\partial x_i\partial x_j}$$

The full matrix is:

$$H_f=\left[\begin{array}{cccc}\frac{{\partial }^{2}f}{\partial x_1^2}& \frac{{\partial }^{2}f}{\partial x_1\partial x_2}& \cdots & \frac{{\partial }^{2}f}{\partial x_1\partial x_n}\\ \frac{{\partial }^{2}f}{\partial x_2\partial x_1}& \frac{{\partial }^{2}f}{\partial x_2^2}& \cdots & \frac{{\partial }^{2}f}{\partial x_2\partial x_n}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{{\partial }^{2}f}{\partial x_n\partial x_1}& \frac{{\partial }^{2}f}{\partial x_n\partial x_2}& \cdots & \frac{{\partial }^{2}f}{\partial x_n^2}\end{array}\right]$$

A fundamental property, as mentioned in the reading, is that the Hessian matrix is symmetric, meaning $H_f=H_f^T$. This is due to the equality of mixed partials (also known as Clairaut's or Schwarz's theorem), which states that for functions with continuous second partial derivatives, the order of differentiation does not matter:

$$\frac{{\partial }^{2}f}{\partial x_i\partial x_j}=\frac{{\partial }^{2}f}{\partial x_j\partial x_i}$$

This symmetry property has profound implications in optimization theory and is not just a computational convenience.

Computing the Hessian Matrix

The process for computing the Hessian follows directly from its definition:

1. Calculate the gradient vector, $\nabla f$.
2. Calculate the partial derivative of each component of the gradient with respect to each input variable.
3. Arrange these second-order derivatives into an $n\times n$ matrix.

Let's watch a clear, worked-out example.

Test your understanding!

In machine learning, loss functions are often quadratic in form. Consider the following simple quadratic function $f:{\mathbb{R}}^2\to \mathbb{R}$, where $\mathbf{x}=[x_1,x_2{]}^T$:
$f(\mathbf{x})=2x_1^2-3x_1{x}_2+5x_2^2-7x_1+2x_2+1$
Compute the Hessian matrix $H_f$ for this function.

Show answer

First, we compute the gradient vector $\nabla f$.

$$\frac{\partial f}{\partial x_1}=4x_1-3x_2-7$$ $$\frac{\partial f}{\partial x_2}=-3x_1+10x_2+2$$

So, the gradient is $\nabla f=\left[\begin{array}{c}4x_1-3x_2-7\\ -3x_1+10x_2+2\end{array}\right]$.

Now, we compute the Jacobian of the gradient to find the Hessian.

The first row of the Hessian is the gradient of the first component of $\nabla f$:

$$\frac{\partial }{\partial x_1}(4x_1-3x_2-7)=4$$ $$\frac{\partial }{\partial x_2}(4x_1-3x_2-7)=-3$$

The second row of the Hessian is the gradient of the second component of $\nabla f$:

$$\frac{\partial }{\partial x_1}(-3x_1+10x_2+2)=-3$$ $$\frac{\partial }{\partial x_2}(-3x_1+10x_2+2)=10$$

Assembling these into a matrix gives the Hessian:

$$H_f=\left[\begin{array}{cc}4& -3\\ -3& 10\end{array}\right]$$

Notice that the Hessian for this quadratic function is a constant matrix, and it is symmetric, as expected. This means the curvature of this function is the same everywhere.

Geometric Interpretation: Curvature

We can compute the Hessian, but what does it represent geometrically? In single-variable calculus, the second derivative $f^{\prime \prime }(x)$ tells us about the function's curvature (concave up or down). The Hessian matrix does the same for multivariable functions.

It describes the local curvature of the function's surface at a given point. However, since we're in $n$ dimensions, curvature isn't a single number but a property that can be different in every direction. The Hessian captures all of this directional information.

The following video provides an excellent derivation of the Hessian from the perspective of the "directional second derivative." This is a more profound way to understand its role.

The key formula from the video, $d^{2}f={\mathbf{v}}^{T}H\mathbf{v}$, is central to the theory of optimization. By analyzing the properties of the matrix $H$ (for example, whether this expression is positive for all directions $\mathbf{v}$), we can determine if a point is a local minimum, maximum, or saddle point. We will explore this in detail in a future module.

Conclusion

In this lesson, we generalized the concept of a second derivative to multivariable scalar functions, arriving at the Hessian matrix.

Key Takeaways:

• The Hessian matrix $H_f$ is the second derivative of a scalar-valued function $f:{\mathbb{R}}^n\to \mathbb{R}$.
• Formally, it is the Jacobian of the gradient, $H_f=\nabla (\nabla f)$, and is always an $n\times n$ square matrix.
• Due to the equality of mixed partials (Clairaut's theorem), the Hessian is always a symmetric matrix ($H_f=H_f^T$).
• Geometrically, the Hessian describes the local curvature of the function's surface. The quadratic form ${\mathbf{v}}^T{H}_f\mathbf{v}$ gives the curvature in the direction of vector $\mathbf{v}$.

Preview of the Next Lesson:
We have now defined the gradient (first derivative) and the Hessian (second derivative). Just as the first derivative is used for linear approximation in single-variable calculus ($f(x)\approx f(a)+f^{\prime }(a)(x-a)$), the gradient and Hessian are the key components of multivariable approximations. In our next lesson, we will combine them to construct the second-order Taylor approximation of a function, a powerful tool for analyzing and optimizing functions in machine learning.