Inner Products: Norms and Distances

1. Defining the Norm of a Vector

You're familiar with the length of a vector $\mathbf{v}=(v_1,v_2)$ in ${\mathbb{R}}^2$ from the Pythagorean theorem: $\Vert \mathbf{v}\Vert =\sqrt{v_1^2+v_2^2}$. Notice that the term inside the square root, $v_1^2+v_2^2$, is precisely the dot product $\mathbf{v}\cdot \mathbf{v}$.

This insight is the basis for generalizing the concept of length to any inner product space.

The norm of a vector $\mathbf{v}$ in an inner product space is defined as:

Thanks to the positive-definiteness axiom, we know $⟨\mathbf{v},\mathbf{v}⟩\ge 0$, so we are always taking the square root of a non-negative real number. The result, $\Vert \mathbf{v}\Vert$, is the generalized "length" of the vector.

The following video provides a clear introduction to this definition and how it relates to our geometric intuition.

2. Computing Norms in Real and Complex Spaces

Let's see how this definition works in practice for the standard inner products we discussed.

The Standard Norm in ${\mathbb{R}}^n$ (Euclidean Norm)

For a real vector $\mathbf{v}=(v_1,v_2,\dots ,v_n)$, the standard inner product is the dot product, $⟨\mathbf{v},\mathbf{v}⟩={\sum }_{i=1}^n{v}_i^2$. The induced norm is:

This is the familiar Euclidean length. For example, if $\mathbf{v}=(1,-2,4)$ in ${\mathbb{R}}^3$, its norm is:

The Standard Norm in ${\mathbb{C}}^n$

This is particularly important for your goal of studying quantum mechanics. For a complex vector $\mathbf{v}=(v_1,v_2,\dots ,v_n)$, the standard inner product is $⟨\mathbf{v},\mathbf{v}⟩={\sum }_{i=1}^n{\bar{v}}_i{v}_i={\sum }_{i=1}^n|v_i{|}^2$. The induced norm is:

Let's compute the norm for $\mathbf{v}=(3-i,2i)$ in ${\mathbb{C}}^2$.
First, we find the squared moduli of the components:

• $|3-i{|}^2=3^2+(-1{)}^2=9+1=10$
• $|2i{|}^2=|0+2i{|}^2=0^2+2^2=4$

Now, we compute the norm:

The following text provides a concise summary and examples for computing these standard norms.

3. Defining Distance Between Vectors

With a norm in hand, defining the distance between two vectors is straightforward. Geometrically, the distance between two points (at the tips of vectors $\mathbf{u}$ and $\mathbf{v}$) is the length of the vector connecting them. This connecting vector is simply their difference, $\mathbf{u}-\mathbf{v}$.

The distance between two vectors $\mathbf{u}$ and $\mathbf{v}$, denoted $d(\mathbf{u},\mathbf{v})$, is defined as the norm of their difference:

Since $\Vert \mathbf{u}-\mathbf{v}\Vert =\sqrt{⟨\mathbf{u}-\mathbf{v},\mathbf{u}-\mathbf{v}⟩}$, the distance is also fundamentally derived from the inner product.

Let's watch a video that illustrates this and works through an example.

4. Norms and Distances from Other Inner Products

The true power of these definitions becomes apparent when we use inner products that aren't the standard dot product. The geometry of the space—what we consider to be length and distance—changes with the inner product.

Example: A Weighted Inner Product

In the last lesson, we saw that $⟨\mathbf{u},\mathbf{v}⟩=2u_1{v}_1+3u_2{v}_2$ defines a valid inner product on ${\mathbb{R}}^2$. Let's calculate a norm and distance in this "weighted" space.

Let $\mathbf{u}=(1,1)$.
Its Euclidean norm would be $\sqrt{1^2+1^2}=\sqrt{2}$.
But its norm induced by our weighted inner product is:

The "length" of the vector is different in this new geometry!

Example: An Inner Product on a Function Space

For your background in calculus, this should be particularly interesting. Consider the space $C[0,1]$ of continuous functions on $[0,1]$, with the inner product $⟨f,g⟩={\int }_0^{1}f(x)g(x)dx$. Let's find the "length" of the function $f(x)=\sqrt{x}$.

So, the norm is $\Vert f(x)\Vert =\sqrt{1/2}=1/\sqrt{2}$. We have successfully calculated the length of a function!

The next video demonstrates exactly these kinds of calculations, showing why the general definition $\Vert \mathbf{v}\Vert =\sqrt{⟨\mathbf{v},\mathbf{v}⟩}$ is so essential.

5. A Broader View: The Axioms of a Norm

So far, we've only talked about norms that are induced by an inner product. However, the concept of a norm is more general. A function $\Vert \cdot \Vert$ is a norm if it satisfies three axioms, regardless of whether it comes from an inner product.

1. Definiteness: $\Vert \mathbf{v}\Vert \ge 0$, and $\Vert \mathbf{v}\Vert =0$ if and only if $\mathbf{v}=0$.
2. Absolute Homogeneity: $\Vert c\mathbf{v}\Vert =|c|\Vert \mathbf{v}\Vert$ for any scalar $c$.
3. Triangle Inequality: $\Vert \mathbf{u}+\mathbf{v}\Vert \le \Vert \mathbf{u}\Vert +\Vert \mathbf{v}\Vert$.

Any norm induced by an inner product automatically satisfies these properties. (Proving the triangle inequality requires the famous Cauchy-Schwarz inequality, which we will encounter soon).

However, some norms used in machine learning do not come from an inner product. For example:

• 1-norm (Manhattan norm): $\Vert \mathbf{v}{\Vert }_1={\sum }_i|v_i|$
• $\infty$-norm (Max norm): $\Vert \mathbf{v}{\Vert }_{\infty }={\max }_i|v_i|$

The standard Euclidean norm is also called the 2-norm, $\Vert \mathbf{v}{\Vert }_2$. These different norms are crucial for techniques like regularization in deep learning. The image below shows what the "unit circle" (the set of all vectors with norm 1) looks like for the 2-norm and 1-norm.

While these other norms are important, our primary focus in this module will remain on the geometry generated by inner products.

Conclusion

In this lesson, we made the crucial connection between the algebraic definition of an inner product and the geometric concepts of length and distance.

Key Takeaways:

• The norm (length) of a vector $\mathbf{v}$ in an inner product space is defined as $\Vert \mathbf{v}\Vert =\sqrt{⟨\mathbf{v},\mathbf{v}⟩}$.
• The distance between two vectors $\mathbf{u}$ and $\mathbf{v}$ is the norm of their difference: $d(\mathbf{u},\mathbf{v})=\Vert \mathbf{u}-\mathbf{v}\Vert$.
• These definitions hold for any inner product, whether it's the standard dot product, a weighted product, or one defined by an integral. The specific inner product defines the geometry of the space.
• The norm induced by an inner product is just one type of norm; other norms (like the 1-norm) are also useful, especially in machine learning.

Preview of the Next Lesson:

The inner product gives us one more fundamental geometric tool: a way to measure angles, which leads to the concept of orthogonality (perpendicularity). In the next lesson, we will define orthogonality in general inner product spaces and see why it is one of the most powerful ideas in all of linear algebra.