Symmetric Matrices and Quadratic Forms

1. What is a Quadratic Form?

A quadratic form is a special type of function that contains only second-degree terms. For any symmetric matrix $A$, the function $Q(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}$ is a quadratic form.

For example, if $A=\left(\begin{array}{cc}3& 4\\ 4& 6\end{array}\right)$ and $\mathbf{x}=\left(\begin{array}{c}{x}_1\\ x_2\end{array}\right)$, the quadratic form is:

The "definiteness" of the matrix $A$ is defined by the behavior of this function. The following video provides an excellent introduction to classifying quadratic forms and visualizing their shapes.

As the video explains, we classify the quadratic form $Q(\mathbf{x})$ as:

• Positive Definite: if $Q(\mathbf{x})>0$ for all non-zero vectors $\mathbf{x}$.
• Positive Semi-definite: if $Q(\mathbf{x})\ge 0$ for all $\mathbf{x}$.
• Negative Definite: if $Q(\mathbf{x})<0$ for all non-zero vectors $\mathbf{x}$.
• Negative Semi-definite: if $Q(\mathbf{x})\le 0$ for all $\mathbf{x}$.
• Indefinite: if $Q(\mathbf{x})$ takes on both positive and negative values.

These classifications correspond directly to the shapes you saw in the last lesson.

2. The Principal Axis Theorem: Why Eigenvalues Dictate the Shape

So, why does the eigenvalue test from our last lesson work? The connection is revealed through the Principal Axis Theorem, which is a direct application of the Spectral Theorem. It states that we can always find a change of coordinates that simplifies the quadratic form by removing the cross-product terms (like the $8x_1{x}_2$ term in our earlier example).

Here's the logic:

1. Since $A$ is symmetric, we can diagonalize it as $A=PD{P}^T$, where $D$ is a diagonal matrix of eigenvalues and $P$ is an orthogonal matrix whose columns are the corresponding eigenvectors.
2. We substitute this into the quadratic form:
   $Q(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}={\mathbf{x}}^T(PD{P}^T)\mathbf{x}=({\mathbf{x}}^{T}P)D(P^T\mathbf{x})$
3. Because $P$ is orthogonal, $P^T=P^{-1}$. The operation $P^T\mathbf{x}$ can be seen as a rotation of the coordinate system. Let's define a new set of coordinates $\mathbf{y}=P^T\mathbf{x}$.
4. The quadratic form in the new coordinates becomes much simpler:
   $Q(\mathbf{y})={\mathbf{y}}^{T}D\mathbf{y}={\lambda }_1{y}_1^2+{\lambda }_2{y}_2^2+\cdots +{\lambda }_n{y}_n^2$

This final expression has no cross-product terms! It's just a weighted sum of squares, where the weights are the eigenvalues of $A$.

The following resource walks through this exact derivation with an example.

From the simplified form $Q(\mathbf{y})=\sum {\lambda }_i{y}_i^2$, the connection is clear:

• If all ${\lambda }_i>0$, then $Q$ is a sum of positive terms and must be positive. The form is positive definite.
• If all ${\lambda }_i\ge 0$, then $Q$ can never be negative. The form is positive semi-definite.
• If there's a mix of positive and negative ${\lambda }_i$, we can choose $\mathbf{y}$ to isolate either a positive or negative term, making $Q$ positive or negative. The form is indefinite.

3. Application: Convexity, Loss Functions, and Optimization

This relationship between definiteness and the shape of a quadratic function is not just a geometric curiosity; it's fundamental to optimization, especially in machine learning.

Consider a more general quadratic function, which includes linear and constant terms:
$f(\mathbf{x})=\frac{1}{2}{\mathbf{x}}^{T}A\mathbf{x}-{\mathbf{b}}^T\mathbf{x}+c$

This is a canonical form for many problems. For example, in deep learning, a core task is to minimize a loss function that measures how poorly a model is performing. Near a minimum, many loss functions can be approximated by a quadratic function like this one.

Since you've studied multivariable calculus, you know about using the gradient and the Hessian matrix to analyze functions. For our function $f(\mathbf{x})$:

• The gradient is $\nabla f(\mathbf{x})=A\mathbf{x}-\mathbf{b}$.
• The Hessian matrix (the matrix of second partial derivatives) is simply ${\nabla }^{2}f(\mathbf{x})=A$.

Recall the second derivative test: at a critical point where $\nabla f=0$, the point is a local minimum if the Hessian is positive definite. Here, the Hessian is the constant matrix $A$. This leads to a powerful conclusion:

If the symmetric matrix $A$ is positive definite, the quadratic function $f(\mathbf{x})$ is convex (it's a "bowl" shape everywhere). This means it has a unique global minimum, which occurs at the point ${\mathbf{x}}^{\ast }$ where the gradient is zero, i.e., the solution to $A\mathbf{x}=\mathbf{b}$.

The following reading explains this connection beautifully.

Professor Strang also emphasizes this link to deep learning. This short clip will reinforce the idea.

Conclusion

Today, we've closed the loop on symmetric matrices and their properties. We moved from the algebraic definition of definiteness (${\mathbf{x}}^{T}A\mathbf{x}>0$) to the practical eigenvalue test, and have now connected it back to the geometric shape of the function that the matrix defines.

Key Takeaways:

• A quadratic form is a function $Q(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}$. Its definiteness classification (e.g., positive definite, indefinite) depends on the sign of its output.
• The Principal Axis Theorem allows us to simplify any quadratic form into a sum of squares, $\sum {\lambda }_i{y}_i^2$, by rotating to the eigenvector coordinate system.
• This simplified form makes it obvious that the signs of the eigenvalues of $A$ determine the definiteness of the quadratic form.
• A positive definite matrix $A$ defines a quadratic form that is shaped like a convex "bowl," with a unique minimum at the origin.
• This property is crucial in optimization: a quadratic objective function $f(\mathbf{x})=\frac{1}{2}{\mathbf{x}}^{T}A\mathbf{x}-{\mathbf{b}}^T\mathbf{x}+c$ is convex and has a unique global minimum if and only if $A$ is positive definite.

Preview of the Next Lesson:

Our study so far has heavily relied on symmetric matrices and their special properties, like having real eigenvalues and being diagonalizable. But what about matrices that are rectangular or not symmetric? In our next module, we will introduce the Singular Value Decomposition (SVD). The SVD is a powerful generalization of diagonalization that applies to any matrix and is one of the most important and widely used matrix decompositions in modern data analysis and deep learning.