Understanding Hessians and Approximations in Optimization

1. The Hessian and Second-Order Optimization

For a scalar-valued function like a loss $\mathcal{L}(\text{vtheta})$ that depends on a parameter vector $\text{vtheta}\in {\mathbb{R}}^n$, the gradient $\nabla \mathcal{L}$ is a vector of first partial derivatives. The Hessian matrix, denoted $\mathbf{H}$, is the $n\times n$ matrix of all second partial derivatives:

Just as the gradient defines the best linear approximation to the loss function at a point, the Hessian is the key component of the best quadratic approximation. This is formalized by the second-order Taylor series expansion around a point ${\text{vtheta}}_0$:

This quadratic model of the loss landscape is much richer than the linear one. It doesn't just tell us which way is "down"; it describes the shape of the valley (or hill, or saddle) we are in.

Newton's Method

If we can build a quadratic approximation of our loss function, why not just jump to its minimum? This is precisely the idea behind Newton's method. By setting the gradient of the Taylor approximation above to zero, we can solve for the optimal step $\Delta \text{vtheta}=\text{vtheta}-{\text{vtheta}}_0$, which yields an elegant update rule.

The resulting update rule is:

Here, the inverse Hessian ${\mathbf{H}}^{-1}$ acts as a preconditioner. It automatically adapts the step size for each parameter direction, taking larger steps in flat directions (low curvature) and smaller steps in sharp directions (high curvature), effectively "warping" the landscape to look more like a perfect bowl.

However, Newton's method is rarely used in its pure form for deep learning due to two major challenges:

1. Computational Cost: For a network with $n$ parameters (often in the millions or billions), the Hessian is an $n\times n$ matrix. Storing it requires $O(n^2)$ memory, and inverting it costs $O(n^3)$ computation, both of which are prohibitive.
2. Non-Convexity: The derivation assumes we are minimizing a convex quadratic. In deep learning, the loss landscape is highly non-convex. The Hessian may not be positive definite, meaning it can have negative or zero eigenvalues. At such points (e.g., saddle points), the Newton step can point uphill or be undefined.

To stabilize the update, a common technique is damping, which involves adding a scaled identity matrix to the Hessian: $(\mathbf{H}+\lambda \mathbf{I}{)}^{-1}$. This ensures the matrix is invertible and positive definite, but it doesn't solve the fundamental computational problem. This is where approximations become essential.

2. The Gauss-Newton Approximation

The first major approximation we'll study is the Gauss-Newton matrix (also called the Generalized Gauss-Newton or GGN matrix). The key insight is that for many common loss functions, the Hessian can be decomposed into two terms, one of which is often small or difficult to compute and can be dropped.

Let our model be a composition of the network $f(\mathbf{x};\text{vtheta})$ producing logits, and a loss function $\ell (\cdot )$. The total loss is $\mathcal{L}(\text{vtheta})=\ell (f(\mathbf{x};\text{vtheta}))$. The Hessian of this composite function can be written as:

Here, ${\mathbf{J}}_f$ is the Jacobian of the network output with respect to $\text{vtheta}$, ${\mathbf{H}}_{\ell }$ is the Hessian of the loss with respect to the network's output, and ${\mathbf{H}}_{f_i}$ is the Hessian of the $i$-th network output with respect to $\text{vtheta}$.

The Gauss-Newton approximation ($\mathbf{G}$) is formed by dropping the second term:

Let's explore the derivation and rationale for this approximation.

As the text highlights, the GGN matrix has two significant advantages:

1. Guaranteed Positive Semi-Definiteness: As long as the loss function $\ell$ is convex (which is true for standard choices like MSE and cross-entropy), its Hessian ${\mathbf{H}}_{\ell }$ is positive semi-definite. This property is transferred to $\mathbf{G}$, ensuring that the optimization step moves in a descent direction.
2. Avoids Network Second Derivatives: The GGN matrix depends only on the Jacobian of the network $f$ and the Hessian of the loss function $\ell$. It completely bypasses the need to compute the network's own Hessian ${\mathbf{H}}_{f_i}$, which is the most computationally demanding and problematic part. This makes the GGN applicable to networks with non-twice-differentiable activation functions like ReLU.

Another powerful way to interpret the GGN is that it's the exact Hessian of the loss function applied to a first-order Taylor approximation (linearization) of the network $f$.

3. The Fisher Information Matrix (FIM)

Now we switch to a probabilistic lens. We can view the output of our network as defining the parameters of a probability distribution $p(y|\mathbf{x};\text{vtheta})$. For classification, the softmax outputs parameterize a Categorical distribution. For regression, the network output might be the mean of a Gaussian. The training process of minimizing the loss (e.g., cross-entropy) is equivalent to maximizing the log-likelihood $\log p(y|\mathbf{x};\text{vtheta})$.

The Fisher Information Matrix ($\mathbf{F}$) measures the amount of information that the data $y$ carries about the parameters $\text{vtheta}$. Intuitively, it quantifies how much the log-likelihood function curves at the location of the true parameters. A sharply curved log-likelihood (high Fisher information) means the parameters are well-determined by the data, while a flat log-likelihood (low Fisher information) means they are not.

The following video provides an excellent visual and intuitive build-up to the concept.

As the video explains, the Fisher Information Matrix has two equivalent definitions:

1. As the covariance of the score function:
   $\mathbf{F}={\mathbb{E}}_{y\sim p(\cdot |\mathbf{x};\text{vtheta})}\left[({\nabla }_{\text{vtheta}}\log p(y|\mathbf{x};\text{vtheta}))({\nabla }_{\text{vtheta}}\log p(y|\mathbf{x};\text{vtheta}){)}^T\right]$
   The score, ${\nabla }_{\text{vtheta}}\log p$, tells us how to change $\text{vtheta}$ to increase the log-likelihood. The FIM measures how much this direction varies on average for different data samples $y$ drawn from the model's own distribution.

2. As the negative expected Hessian of the log-likelihood:
   $\mathbf{F}=-{\mathbb{E}}_{y\sim p(\cdot |\mathbf{x};\text{vtheta})}\left[{\nabla }_{\text{vtheta}}^2\log p(y|\mathbf{x};\text{vtheta})\right]$
   This second definition directly connects the Fisher Information to the curvature of the log-likelihood landscape.

The Connection Between GGN and FIM

At first glance, the GGN (from optimization theory) and the FIM (from information geometry) seem to come from different worlds. The remarkable result is that for most standard deep learning setups, they are identical.

When the loss function is the negative log-likelihood of a distribution from the exponential family (which includes the Gaussian and Categorical distributions underlying MSE and cross-entropy loss), the Gauss-Newton matrix equals the Fisher Information Matrix.

This equivalence is profound. It means that approximating the Hessian by linearizing the network (the GGN view) is the same as measuring the curvature of the manifold of probability distributions defined by the network (the FIM view).

This connection gives rise to Natural Gradient Descent, an optimization algorithm that uses the inverse Fisher matrix as a preconditioner:

This method is considered "natural" because it is invariant to the specific parameterization of the network. It takes steps of constant size in the space of distributions, not parameters, leading to more stable and often faster convergence.

Finally, it's crucial to distinguish the proper Fisher from the empirical Fisher. The proper Fisher, which we've discussed, involves an expectation over the model's distribution $y\sim p(y|\mathbf{x})$. The empirical Fisher instead takes the expectation over the data distribution $y\sim q(y|\mathbf{x})$ (i.e., using the ground-truth labels). Optimizers like Adam use a diagonal version of the empirical Fisher. While the empirical Fisher is often easier to compute, only the proper Fisher enjoys all the theoretical guarantees of natural gradient descent.

Conclusion

In this lesson, we moved beyond first-order gradients to explore the role of second-order information in optimization. We saw that the Hessian matrix provides a full picture of local curvature but is computationally intractable for large models. This motivated the development of powerful approximations.

Key Takeaways:

• The Hessian matrix $\mathbf{H}$ contains all second-order partial derivatives of the loss function and describes the local curvature of the loss landscape.
• Newton's method uses the inverse Hessian to pre-condition the gradient, leading to faster convergence but suffering from high computational cost and instability in non-convex regions.
• The Gauss-Newton (GGN) matrix $\mathbf{G}$ is a practical approximation of the Hessian that avoids computing second derivatives of the network itself. It is guaranteed to be positive semi-definite for common convex loss functions.
• The Fisher Information Matrix (FIM) $\mathbf{F}$ arises from information geometry and measures the sensitivity of the model's output distribution to changes in its parameters.
• For standard deep learning loss functions (MSE, cross-entropy), the GGN and FIM are identical, providing a deep link between optimization and information geometry.
• These approximations form the basis for advanced optimizers like Natural Gradient Descent, which uses ${\mathbf{F}}^{-1}$ to achieve parameterization-invariant updates.

In our next lesson, we will delve deeper into the property of convexity. We'll see how the Hessian, particularly its eigenvalues, can be used to formally test for convexity, a critical concept for understanding why some optimization problems are fundamentally easier to solve than others.