Hessian and Convexity: An Introduction

1. What is a Convex Function?

Let's start with the visual intuition. In one dimension, a convex function is one that is "bowl-shaped" or has positive curvature everywhere. A more formal geometric definition is:

A function is convex if the line segment connecting any two points on its graph lies on or above the graph itself.

Conversely, a function is concave if the line segment lies on or below the graph (like an upside-down bowl). A function that is not consistently one or the other is non-convex.

Why is this property so important for your goal of studying deep learning?
Because for a convex function, any local minimum is also a global minimum. This completely changes the game for optimization. If you are trying to find the minimum of a convex function, you can use an algorithm that just goes "downhill" (like gradient descent), and you are guaranteed not to get stuck in a "valley" that isn't the absolute lowest point.

2. The Second-Order Condition for Convexity

In the last lesson, a positive definite Hessian at a critical point signaled a local minimum. Now, we'll elevate that idea to a global statement about the function. The connection is remarkably direct.

For a twice-differentiable function $f(\mathbf{x})$, its convexity is determined by its Hessian matrix, ${\nabla }^{2}f(\mathbf{x})$, over its entire domain.

As you just read, the rule is as follows:

• A function $f$ is convex if and only if its Hessian ${\nabla }^{2}f(\mathbf{x})$ is positive semi-definite for all $\mathbf{x}$ in its domain. (Eigenvalues ${\lambda }_i\ge 0$).
• If the Hessian ${\nabla }^{2}f(\mathbf{x})$ is positive definite for all $\mathbf{x}$, then $f$ is strictly convex. (Eigenvalues ${\lambda }_i>0$).

A strictly convex function has a bowl shape that is always curving up, which guarantees a unique global minimum. A convex (but not strictly convex) function might have "flat" regions or troughs, allowing for multiple points to share the same minimum value.

3. The Archetype: Quadratic Functions

The most important and intuitive examples of convex functions are quadratic functions. Let's consider the general quadratic form:

where $A$ is a symmetric matrix. From our work on matrix calculus, we know the Hessian of this function is simply:

The Hessian is constant—it's the same matrix $A$ everywhere! This means the convexity of the entire quadratic function is determined solely by the properties of the matrix $A$.

The following reading provides an excellent and intuitive explanation of this exact connection, which is at the heart of many optimization problems.

The reading beautifully illustrates that if $A$ is positive definite, the quadratic function $f(\mathbf{x})$ is a convex, bowl-shaped surface whose unique minimum occurs precisely at the point ${\mathbf{x}}^{\ast }$ that solves the linear system $A\mathbf{x}=\mathbf{b}$.

To solidify this visual, the following short video clip shows how a positive definite matrix creates this characteristic "bowl" shape.

The article you read also contains excellent visualizations (in its section "Visualizing definiteness", Figure 9) showing what the quadratic surface looks like for different types of matrices:

• Positive definite A: A convex bowl (a paraboloid).
• Negative definite A: A concave, upside-down bowl.
• Indefinite A: A Pringles-like saddle shape.
• Positive semi-definite A (but singular): A convex "trough" or "channel".

Conclusion

In this lesson, we have solidified a crucial link between the algebraic properties of the Hessian matrix and the geometric shape of a function.

Key Takeaways:

• A convex function is "bowl-shaped," meaning any local minimum you find is guaranteed to be a global minimum.
• The second-order condition for convexity provides a definitive test: a function is convex if and only if its Hessian matrix is positive semi-definite everywhere in its domain.
• A function is strictly convex if its Hessian is positive definite everywhere, which guarantees a unique global minimum.
• Quadratic functions of the form $f(\mathbf{x})=\frac{1}{2}{\mathbf{x}}^{T}A\mathbf{x}-{\mathbf{b}}^T\mathbf{x}+c$ are the quintessential example. Their convexity is determined entirely by the definiteness of the matrix $A$.

Preview of the Next Lesson:

We now have a deep understanding of the "landscape" of a function—its direction of steepest ascent (the gradient) and its curvature (the Hessian and convexity). The next logical step is to develop an algorithm to navigate this landscape to find a minimum. In our next lesson, we will introduce the most fundamental optimization algorithm of all: gradient descent. We will see how it uses the gradient to take iterative steps "downhill," and how the concept of convexity assures us that this simple process can lead to a global optimum.