NumPy Norms: Vectors and Matrices

1. What is a Norm and Why is it Important?

At its core, a norm is a function that assigns a strictly positive length or size to each vector in a vector space (except for the zero vector, which has a length of zero). You can think of it as a generalized concept of distance.

In machine learning, we use norms constantly. For instance:

• When a model makes a prediction, we can represent the errors for all data points as a vector. The norm of this error vector gives us a single number that quantifies the total error of the model.
• In regularization, we add a term to our model's loss function that is based on the norm of the model's weight vector. This encourages the model to learn smaller, less complex weights, which can help it generalize better to new data.

To get a quick overview of the role of norms in machine learning, please watch the following short video clip.

2. Vector Norms

Most common vector norms are part of a family called Lp norms. The general formula for the Lp norm of a vector $\mathbf{x}$ is:

By choosing different values for $p$, we get different types of norms with unique properties. Let's explore the most important ones for machine learning.

The L2 Norm (Euclidean Norm)

This is the most common norm you'll encounter. It corresponds to setting $p=2$ in the general formula and represents the familiar concept of Euclidean distance—a straight line from the origin to the point defined by the vector.

Formula:

In NumPy, we can compute this using the numpy.linalg.norm function. If you don't specify the type of norm, NumPy calculates the L2 norm by default for vectors.

import numpy as np

# A vector representing a point in 3D space
x = np.array([3, 0, 4])

# Calculate the L2 norm (Euclidean distance from origin)
l2_norm = np.linalg.norm(x)

# The result is sqrt(3^2 + 0^2 + 4^2) = sqrt(9 + 16) = sqrt(25) = 5.0
print(f"Vector x: {x}")
print(f"L2 Norm of x: {l2_norm}")

# You can also be explicit by setting the 'ord' parameter to 2
l2_norm_explicit = np.linalg.norm(x, ord=2)
print(f"L2 Norm of x (explicit): {l2_norm_explicit}")

The L1 Norm (Manhattan Norm)

The L1 norm is calculated by setting $p=1$. It's the sum of the absolute values of the vector's components. It's often called the "Manhattan norm" because it represents the distance you would travel between two points in a city grid, where you can only move along horizontal or vertical streets.

Formula:

In machine learning, the L1 norm is particularly interesting because it tends to produce sparse results (many values being exactly zero). This makes it very useful for feature selection, as it can effectively "turn off" the influence of less important features by setting their weights to zero.

To compute the L1 norm in NumPy, we set ord=1.

# The same vector x
x = np.array([3, 0, -4])

# Calculate the L1 norm
l1_norm = np.linalg.norm(x, ord=1)

# The result is |3| + |0| + |-4| = 3 + 0 + 4 = 7.0
print(f"Vector x: {x}")
print(f"L1 Norm of x: {l1_norm}")

Geometric Intuition: L2 vs. L1

The different properties of L1 and L2 norms lead to different geometric shapes when you plot all the vectors that have a norm equal to 1.

The L-infinity Norm (Max Norm)

As $p$ approaches infinity, the Lp norm simplifies to become the absolute value of the largest element in the vector.

Formula:

To compute the max norm in NumPy, we use np.inf for the ord parameter.

x = np.array([3, 0, -4])

# Calculate the L-infinity norm
inf_norm = np.linalg.norm(x, ord=np.inf)

# The result is max(|3|, |0|, |-4|) = 4.0
print(f"Vector x: {x}")
print(f"L-infinity Norm of x: {inf_norm}")

3. Matrix Norms

Just as we can measure the size of vectors, we can also measure the size of matrices. While several types of matrix norms exist, the most common one in machine learning is the Frobenius norm.

The Frobenius Norm

The Frobenius norm is analogous to the vector L2 norm. To calculate it, you simply take all the elements of the matrix, square them, sum them all up, and take the square root of the total.

Formula: For an $m\times n$ matrix A:

This is a very natural way to extend the concept of Euclidean length to a matrix. In NumPy's linalg.norm() function, the Frobenius norm is the default for 2D arrays (matrices).

A = np.array([[1, 2], 
              [3, 4]])

# Calculate the Frobenius norm (default for matrices)
frob_norm = np.linalg.norm(A)

# Result: sqrt(1^2 + 2^2 + 3^2 + 4^2) = sqrt(1 + 4 + 9 + 16) = sqrt(30) ≈ 5.477
print(f"Matrix A:\n{A}")
print(f"Frobenius Norm of A: {frob_norm}")

# You can also be explicit by setting ord='fro'
frob_norm_explicit = np.linalg.norm(A, ord='fro')
print(f"Frobenius Norm of A (explicit): {frob_norm_explicit}")

4. A Practical Reference for Calculation

To consolidate your understanding of the formulas and see more examples of their implementation in Python, the following article is an excellent reference.

This article also briefly mentions calculating norms along a specific axis of a matrix, like summing up columns or rows. This is a powerful feature of NumPy that combines what you've learned about norms and array axes.

Conclusion

Excellent work! You've now learned how to quantify the magnitude of vectors and matrices, a fundamental skill for understanding and implementing machine learning algorithms.

Key Takeaways:

• A norm measures the "size" or "length" of a vector or matrix.
• The L2 norm (Euclidean) is the standard measure of distance and is the default for vectors in NumPy.
• The L1 norm (Manhattan) sums absolute values and is useful for promoting sparsity (zero-valued weights) in models.
• The Frobenius norm is the L2 equivalent for matrices and is the default for 2D arrays in NumPy.
• All these norms can be computed with the np.linalg.norm() function, using the ord parameter (1, 2, np.inf, 'fro') to specify the type.

Preview of the Next Lesson:

Now that we can create, operate on, and measure vectors and matrices, we'll dive into two more essential matrix operations in the next lesson: matrix multiplication and transposition. We will explore their properties and see why they are at the heart of how linear models and neural networks process information.