Eigen-intuition: Decomposing Geometry and Loss Landscape Curvature

1. The Geometry of Eigenvectors and Eigenvalues

At its core, eigendecomposition asks a simple question about a linear transformation: are there any vectors whose direction is unchanged by the transformation? For a transformation represented by a square matrix $A$, an eigenvector $v$ is a non-zero vector that, when transformed by $A$, only gets scaled. The scaling factor is its corresponding eigenvalue $\lambda$. This relationship is captured by the deceptively simple equation:

This means that eigenvectors represent the stable axes of a transformation—the directions that are only stretched, squished, or flipped, but not rotated.

The 3Blue1Brown video, "Eigenvectors and eigenvalues," provides an outstanding visual intuition for this concept. It masterfully builds up from the geometric idea to the computational method for finding these special vectors and scalars.

The key takeaway from the video is that eigendecomposition re-expresses a transformation in its most natural coordinate system: the one defined by its own invariant directions. A transformation that might seem complex (e.g., a combination of stretching and shearing) can become a simple scaling operation along each axis in the eigenbasis. This is incredibly powerful and, as we'll see, is precisely what we need to understand the complex geometry of a neural network's loss function.

2. The Hessian Matrix and Directional Curvature

Now, let's pivot from pure linear algebra to multivariable calculus, the language of optimization. The loss function of a neural network, $L(\theta )$, is a high-dimensional scalar field where $\theta$ represents the model's millions or billions of parameters. During training, we use the gradient, $\nabla L(\theta )$, to find the direction of steepest descent and update the parameters. The gradient is the first-order information.

To understand the local geometry or curvature of the loss surface, we need to go to the second order. This is where the Hessian matrix comes in. The Hessian, $H$, is the matrix of all second partial derivatives of the loss function with respect to the parameters:

Just as the gradient vector allows us to compute the first derivative in any direction (the directional derivative), the Hessian matrix allows us to compute the second derivative in any direction.

The following video provides an excellent and intuitive primer on how the gradient and Hessian relate to directional derivatives.

3. Eigenvalues of the Hessian: Quantifying Curvature

We now have the two pieces of our puzzle:

1. Eigendecomposition tells us the natural scaling directions (eigenvectors) and scaling factors (eigenvalues) of a square matrix.
2. The Hessian matrix lets us calculate the curvature (directional second derivative) of our loss function via the expression $v^{T}Hv$.

Let's combine them. The Hessian matrix of a loss function is a real, symmetric matrix (assuming the loss is twice-continuously differentiable, which we take for granted in most deep learning contexts). Real symmetric matrices have a wonderful property, formalized by the Spectral Theorem (which we'll cover in the next lesson): they are always diagonalizable and their eigenvectors corresponding to distinct eigenvalues are orthogonal. We can always find an orthonormal basis of eigenvectors.

Let's consider this eigenbasis for the Hessian $H$. Let $v_i$ be an eigenvector of $H$ with eigenvalue ${\lambda }_i$. If we are at a point ${\theta }_0$ in the loss landscape and we move in the direction of this eigenvector $v_i$, what is the curvature?

Assuming $v_i$ is a unit vector, the curvature is simply the eigenvalue ${\lambda }_i$.

This gives us a profound geometric interpretation:

• The eigenvectors of the Hessian are the principal axes of curvature of the loss landscape. They point in the directions of maximal, minimal, and stationary curvature.
• The eigenvalues of the Hessian are the values of the curvature along these principal axes.

We can classify the local geometry based on the signs of the Hessian's eigenvalues:

• All ${\lambda }_i>0$: The surface curves upwards in all directions. We are at a local minimum. If the ${\lambda }_i$ are large, it's a "sharp" minimum. If they are small, it's a "flat" minimum.
• All ${\lambda }_i<0$: The surface curves downwards in all directions. We are at a local maximum.
• Some ${\lambda }_i>0$ and some ${\lambda }_i<0$: The surface curves up in some directions and down in others. We are at a saddle point.
• Some ${\lambda }_i=0$: There is at least one direction of zero curvature. The landscape is "flat" in this direction.

4. The Loss Landscape in Deep Learning

This analysis isn't just a theoretical exercise; it provides deep insights into the optimization challenges in deep learning. Computing the full Hessian is computationally infeasible for modern networks, but we can probe its properties using matrix-free methods (like power iteration to find the top eigenvalues) which rely only on Hessian-vector products. These studies have revealed a consistent structure in the Hessian spectrum.

The findings discussed in the paper by Sagun et al. are central to our modern understanding of deep learning optimization:

• The Hessian Spectrum: The spectrum is typically composed of two parts:

  1. A few large positive eigenvalues (the "outliers"). These directions correspond to high curvature and are learned quickly. Their number often correlates with the structure of the data (e.g., number of classes).
  2. A massive number of eigenvalues that are very close to zero (the "bulk"). This corresponds to the extreme over-parameterization of neural networks. These directions are flat, and moving along them barely changes the loss.

• Sharp vs. Flat Minima and Generalization: It is widely believed that solutions found in "flat" regions of the loss landscape (where local curvature/eigenvalues are small) generalize better than solutions in "sharp" regions (large eigenvalues). The intuition is that a flat minimum is less sensitive to small shifts between the training and test data distributions.

• The Plague of Saddle Points: In low-dimensional spaces, we picture getting stuck in poor local minima. However, in the very high-dimensional space of neural network parameters, true local minima are rare. Most critical points (where the gradient is zero) are saddle points. Optimization algorithms must be able to navigate these landscapes and escape the saddle points by finding directions of negative curvature (given by eigenvectors with negative eigenvalues).

Conclusion

In this lesson, we have connected the abstract linear algebra concept of eigendecomposition to the very practical problem of understanding and optimizing deep neural networks.

Key Takeaways:

• Eigendecomposition ($A=PD{P}^{-1}$) breaks down a square matrix $A$ into its fundamental scaling directions (eigenvectors) and scaling factors (eigenvalues).
• The Hessian matrix $H$ contains the second-order information about a loss function. The curvature in any direction $v$ is given by $v^{T}Hv$.
• The eigenvectors of the Hessian are the principal axes of curvature of the loss landscape.
• The eigenvalues of the Hessian are the curvature values along these axes. Their signs tell us whether we are at a minimum, maximum, or saddle point. Their magnitudes tell us if the curvature is sharp or flat.
• The Hessian spectrum of deep networks is characterized by a "bulk" of near-zero eigenvalues (due to over-parameterization) and a few "outlier" large eigenvalues (related to data structure), creating a landscape dominated by flat regions and saddle points.

In our next lesson, we will delve into the Spectral Theorem. This is the theorem that guarantees that symmetric matrices (like the Hessian, covariance matrices, and kernel matrices) have the nice properties we relied on today, such as having real eigenvalues and an orthonormal basis of eigenvectors. This will be a key building block for understanding Principal Component Analysis (PCA).