Positive Definite and Semi-definite Matrices

Hello! Welcome back to our course on linear algebra.

In our last lesson, we explored the Spectral Theorem, which revealed the elegant structure of symmetric and Hermitian matrices. We learned that they are always diagonalizable with real eigenvalues and an orthonormal basis of eigenvectors. This property is not just a mathematical curiosity; it's the foundation for many applications.

Today, we'll use this foundation to classify these matrices further. We'll introduce a concept that parallels the idea of a number being positive, negative, or zero. Our learning outcome for this lesson is to define positive definite and semi-definite matrices. This classification is fundamental for your future studies in both deep learning (for optimization) and quantum mechanics (for describing physical observables).

1. Generalizing "Positivity" to Matrices

For a single number $a$ , we intuitively know what it means to be positive. One property of a positive number $a$ is that when you multiply another non-zero number $x$ by it, the result $a x$ stays on the same side of the origin as $x$ . The angle doesn't flip by 180 degrees.

How can we extend this idea to an $n \times n$ matrix $A$ ? It's not as simple as checking if all entries are positive. Instead, we look at what the matrix does to a vector. We want the output vector $A x$ to stay on the "same side" of the origin as the input vector $x$ .

The following video provides an excellent visual intuition for this concept. It argues that the right way to define positivity for a symmetric matrix is by checking if the dot product of $x$ and $A x$ is positive.

What Does It Mean For a Matrix to be POSITIVE? The Practical Guide to Semidefinite Programming(1/4)

Watch this segment from 'Visually Explained' to build an intuitive understanding of why the quadratic form is the right way to generalize positivity to matrices.

Watch the video from 2:49 to 4:43. Focus on the core idea: a matrix is considered 'positive' if, for any non-zero input vector x, the output vector Ax is on the same side of the origin. This 'sameness' is measured by the inner product xᵀ(Ax).

As the video explained, the key expression is $x^{T} A x$ . This is the dot product between the input vector $x$ and the output vector $A x$ . If this value is positive, it means the angle between $x$ and $A x$ is acute (less than 90 degrees), so they point in generally the same direction.

2. Formal Definitions

This brings us to the formal definitions. We will focus on symmetric real matrices, as this is where these concepts are most powerful and widely used.

Understanding Positive Definite Matrices

This article from Gregory Gundersen's blog provides clear, formal definitions and reinforces the geometric intuition we just discussed. Please read the introduction and the first part building intuition.

Read the sections 'Understanding Positive Definite Matrices' (which gives the formal definitions) and the following un-titled section that builds intuition with diagrams (starts with 'To start, let’s put aside the definition...'). Focus on the formal definitions (Eq. 1 and 2) and how they connect to the cosine angle between x and Ax.

Let's summarize the definitions from the reading. For a real symmetric matrix $A$ :

$A$ is positive definite (PD) if $x^{T} A x > 0$ for all non-zero vectors $x \in R^{n}$ .
$A$ is positive semi-definite (PSD) if $x^{T} A x \geq 0$ for all vectors $x \in R^{n}$ .
$A$ is negative definite (ND) if $x^{T} A x < 0$ for all non-zero vectors $x \in R^{n}$ .
$A$ is negative semi-definite (NSD) if $x^{T} A x \leq 0$ for all vectors $x \in R^{n}$ .
$A$ is indefinite if $x^{T} A x$ takes on both positive and negative values.

The expression $x^{T} A x$ is called a quadratic form. It's a function that maps a vector to a scalar. Given your background in Calculus 3, you can think of it as a function $f (x) = x^{T} A x$ whose graph is a surface in $(n + 1)$ -dimensional space.

3. The "Energy" View and Visualizing Quadratic Forms

Professor Gilbert Strang calls the quadratic form $x^{T} A x$ the "energy" of the system. Let's see how to compute it and what it looks like.

5. Positive Definite and Semidefinite Matrices

In this classic lecture, Professor Strang explains how to compute the quadratic form and visualizes its graph for a positive definite matrix.

Watch the segment from 9:12 to 14:21. Pay attention to how the entries of the matrix A contribute to the quadratic function f(x, y). Notice his key insight: for a positive definite matrix, the graph of the energy is a 'bowl' shape, always above zero except at the origin.

As Professor Strang showed for a 2x2 matrix, the graph of $f (x, y) = x^{T} A x$ for a positive definite matrix is a paraboloid that opens upward. Its minimum value is 0 at $x = 0$ . For an indefinite matrix, the graph is a saddle shape, with values going both above and below zero.

Visualizing Quadratic Forms: Positive Definite and Indefinite — A visual representation of quadratic forms. On the left, a positive definite form, which is always positive except at the origin, creating a 'bowl' shape. On the right, an indefinite form, which takes both positive and negative values, creating a 'saddle' shape.

This connection to the shape of multivariable functions is critical in optimization. In machine learning, you often want to minimize a cost function. If the Hessian matrix (the matrix of second derivatives) of that function is positive definite, it guarantees that a critical point is a local minimum—you've found the bottom of a "bowl".

Examples

Let's test the definition on a few simple matrices.

Identity Matrix: $A = I = (1001)$ .
The quadratic form is $x^{T} I x = x^{T} x = x_{1}^{2} + x_{2}^{2}$ . This is the squared length of the vector $x$ , which is always > 0 for any non-zero $x$ . So, the identity matrix is positive definite.
A Singular Matrix: $A = (1111)$ .
The quadratic form is:
$(x_{1} x_{2}) (1111) (x_{1} x_{2}) = (x_{1} + x_{2} x_{1} + x_{2}) (x_{1} x_{2}) = (x_{1} + x_{2}) x_{1} + (x_{1} + x_{2}) x_{2} = (x_{1} + x_{2})^{2}$
Since $(x_{1} + x_{2})^{2} \geq 0$ for any $x_{1}, x_{2}$ , the matrix is positive semi-definite. Note that if we choose $x = (1 - 1)$ , the energy is 0. This is the "borderline" case that Professor Strang mentions.

Test your understanding!

Is the matrix $A = (20 0 - 3)$ positive definite, negative definite, or indefinite? Use the quadratic form definition.

Show answer

Let's compute the quadratic form $x^{T} A x$ :
$x^{T} A x = (x_{1} x_{2}) (20 0 - 3) (x_{1} x_{2}) = 2 x_{1}^{2} - 3 x_{2}^{2}$
This expression can be positive or negative.

If we pick $x = (10)$ , the result is $2 (1)^{2} - 3 (0)^{2} = 2 > 0$ .
If we pick $x = (01)$ , the result is $2 (0)^{2} - 3 (1)^{2} = - 3 < 0$ .
Since the quadratic form takes on both positive and negative values, the matrix is indefinite.

4. Equivalent Conditions for Definiteness

While the quadratic form $x^{T} A x$ is the fundamental definition, checking it for all possible vectors $x$ is impractical. Fortunately, there are several equivalent conditions that are much easier to test.

Positive Semi-Definite Matrices

This short document from the University of British Columbia neatly summarizes the three equivalent ways to define positive definite and semi-definite matrices. This is a crucial summary that connects several ideas we've been building.

Read sections 1 ('Positive semi-definite matrices') and 2 ('Strictly positive definite matrices'). Pay close attention to the three equivalent conditions listed for each. We have focused on condition #2 today.

Let's summarize these powerful equivalences for a symmetric matrix $A$ :

A is Positive Semi-Definite (PSD) if:

All eigenvalues of $A$ are non-negative ( $λ_{i} \geq 0$ ).
$x^{T} A x \geq 0$ for all $x$ .
$A$ can be factored as $A = B^{T} B$ for some matrix $B$ .

A is Positive Definite (PD) if:

All eigenvalues of $A$ are strictly positive ( $λ_{i} > 0$ ).
$x^{T} A x > 0$ for all non-zero $x$ .
$A$ can be factored as $A = B^{T} B$ for an invertible (full-rank) matrix $B$ .

Today, we've focused on establishing the definition via condition #2. In our next lesson, we will see that condition #1 (checking the sign of the eigenvalues) is the most practical and powerful way to classify a matrix. The fact that the Spectral Theorem guarantees real eigenvalues for symmetric matrices makes this check possible.

Condition #3 is also very important in data science. A covariance matrix, which measures how different variables in a dataset vary together, is always positive semi-definite because it can always be constructed in the form $X^{T} X$ .

Conclusion

Today we've moved from the general properties of symmetric matrices to a specific and useful classification. By extending the concept of "positivity" from numbers to matrices, we have laid the groundwork for powerful applications.

Key Takeaways:

A symmetric matrix $A$ is defined by the behavior of its quadratic form, $x^{T} A x$ .
Positive definite (PD) matrices have $x^{T} A x > 0$ for all non-zero $x$ . Geometrically, $A$ keeps every vector $x$ on the same "side" of the origin. The graph of the quadratic form is a "bowl" opening upwards.
Positive semi-definite (PSD) matrices have $x^{T} A x \geq 0$ , representing the borderline case where some non-zero vectors can result in an energy of zero.
There are equivalent definitions for definiteness based on eigenvalues and matrix factorization ( $A = B^{T} B$ ), which provide practical tests and deeper connections to applications.

Preview of the Next Lesson:

We've introduced the definitions of positive definite and semi-definite matrices. In the next lesson, we will fully exploit the connection to eigenvalues. You will learn how to classify a symmetric matrix as positive definite, semi-definite, or indefinite simply by computing and inspecting the sign of its eigenvalues. This will give you a concrete, algorithmic tool to determine the definiteness of any symmetric matrix.

Recommended resources