Vector Norms: Magnitude Measures

In our previous lesson, we began our exploration of vectors and established the dot product as a fundamental tool for measuring the relationship between them. We saw that the dot product $v \cdot v$ gives the square of a vector's length, $∥ v ∥^{2}$ . Today, we will formalize and expand upon this concept of "length" by delving into vector norms.

This lesson focuses on the two most important norms in machine learning: the L1 and L2 norms. You will learn not only how to compute them but, more crucially, how to interpret the different stories they tell about a vector's magnitude. This distinction is not just a theoretical curiosity; it has profound practical implications in areas like model regularization (Lasso vs. Ridge) and the design of loss functions, concepts central to the mechanistic interpretability work you're aiming for.

What is a Norm?

While we have an intuitive sense of length, mathematics requires a more rigorous definition. A norm is a function that assigns a strictly positive length or size to each vector in a vector space. For a function to be considered a norm, it must satisfy a few specific properties.

To understand these foundational rules, let's turn to a clear, concise guide.

Vector Norms: A Quick Guide | Built In

This article from Built In introduces the concept of a norm and lays out its defining properties. This will provide the rigorous foundation that you appreciate.

Start by reading the introductory sections "What Is a Norm?" to get a high-level overview. Then, carefully study the four key properties in the section "Properties of a Norm": Non-negativity: The length can't be negative. Definiteness: Only the zero vector has a length of zero. Homogeneity: Scaling the vector scales its length proportionally. Triangle Inequality: The length of a sum of two vectors is no more than the sum of their individual lengths (the shortest path between two points is a straight line).

As you've just read, any function that satisfies these four rules can be called a norm. We denote the norm of a vector $v$ as $∥ v ∥$ . Now, let's explore the most common ways to define this function.

The L2 Norm: Euclidean Distance

The first norm we'll discuss is the one you are already familiar with from the previous lesson. The L2 norm, also known as the Euclidean norm, is the standard measure of straight-line distance. For a vector $v = v_{1} v_{2} ⋮ v_{n}$ in $R^{n}$ , its L2 norm is defined as:

∥ v ∥_{2} = v_{1}^{2} + v_{2}^{2} + \dots + v_{n}^{2} = i = 1 \sum n v_{i}^{2}

Notice that this is precisely the square root of the dot product of the vector with itself: $∥ v ∥_{2} = v \cdot v$ . This is the "length" you know from geometry and physics—the length of the hypotenuse in a right-angled triangle, generalized to $n$ dimensions via the Pythagorean theorem.

The L1 Norm: Manhattan Distance

While the L2 norm is intuitive, it's not the only way to measure vector magnitude. An equally important measure, especially in data science and optimization, is the L1 norm.

Imagine navigating a city grid like Manhattan, where you can only travel along streets and avenues, not through buildings. The total distance you travel is not the straight-line "as the crow flies" distance, but the sum of the horizontal and vertical distances. This is the intuition behind the L1 norm.

For the same vector $v$ , its L1 norm is defined as the sum of the absolute values of its components:

∥ v ∥_{1} = ∣ v_{1} ∣ + ∣ v_{2} ∣ + \dots + ∣ v_{n} ∣ = i = 1 \sum n ∣ v_{i} ∣

Let's look at a simple example. For the vector $V_{1} = [34]$ :

The L2 norm is $∥ V_{1} ∥_{2} = 3^{2} + 4^{2} = 9 + 16 = 25 = 5$ . This is the length of the vector's arrow.
The L1 norm is $∥ V_{1} ∥_{1} = ∣3∣ + ∣4∣ = 7$ . This is the distance you would travel from the origin to the point (3, 4) if you could only move along the grid lines.

This image shows the calculation of the L1 norm for the vector (3, 4). The total "taxicab" path length is 3 + 4 = 7.

Interpretation: Total Change vs. Net Displacement

So, we have two different numbers, 5 and 7, for the "magnitude" of the same vector. This isn't a contradiction; it reflects that the two norms measure different concepts. The choice of which norm to use depends on what question you are trying to answer.

The following reading provides an excellent conceptual framework for this distinction.

L1-Norm: Principles, Sparsity, and Applications | Bohrium

This article from Bohrium gives a powerful intuition for when and why you'd choose one norm over the other.

First, read the section "Measuring the World" to solidify the contrast between the L2 Euclidean distance and the L1 Manhattan distance. Then, pay close attention to the analogy in the section "Total Change vs. Net Displacement". This explanation is key to building a deep interpretation of what each norm represents.

As the article explains:

The L2 norm measures net displacement. Because it squares components, it gives disproportionately more weight to large values. It is sensitive to outliers and tells you the "straight-line" distance between the start and end points in the vector space.
The L1 norm measures total change. It treats every component's contribution equally (no squaring). It's like an accountant's ledger of all activity, regardless of direction. It measures the total volume of movement along each axis.

This difference is why L1 and L2 regularization (Lasso and Ridge, respectively) have such different effects in machine learning. L1 regularization can force feature weights to exactly zero (promoting sparsity), while L2 regularization shrinks them towards zero but rarely makes them exactly zero. We'll see the geometric reason for this next.

The Geometry of Norms

The difference between L1 and L2 norms becomes strikingly clear when we visualize all the vectors that have a norm equal to 1. This set of vectors forms a shape called the "unit ball" for that norm.

The following video provides a great visual for this.

Vector Norms

This short video from the channel ritvikmath will help you visualize the geometric difference between the L1 and L2 norms.

Watch the segment from the geometric example. It shows what the set of all 2D vectors with a norm of 2 looks like for both L2 and L1.

As the video illustrates, and as shown in the image below:

The L2 unit ball (all vectors where $x^{2} + y^{2} = 1$ ) is a circle.
The L1 unit ball (all vectors where $∣ x ∣ + ∣ y ∣ = 1$ ) is a diamond (a square rotated by 45 degrees).

This image compares the unit balls for the L1 norm (diamond), L2 norm (circle), and L-infinity norm (square). The shape of the unit ball is a fundamental geometric property of each norm.

The "pointy" corners of the L1 diamond lie on the axes. In optimization problems, when a cost function contour expands and touches this shape, it is more likely to make contact at one of these corners. This means one of the vector components becomes zero, which is the geometric basis for the sparsity-inducing property of L1 regularization. The L2 circle, being perfectly round, has no such preference for the axes.

Practice Problem

Let's solidify this with a quick exercise. Consider the vector $v$ in $R^{3}$ :

v = 3 - 4 12

Calculate its L2 norm, $∥ v ∥_{2}$ .
Calculate its L1 norm, $∥ v ∥_{1}$ .
Briefly describe what each of these numbers represents.

Solution:

$∥ v ∥_{2} = 3^{2} + (- 4)^{2} + 1 2^{2} = 9 + 16 + 144 = 169 = 13$ .
$∥ v ∥_{1} = ∣3∣ + ∣ - 4∣ + ∣12∣ = 3 + 4 + 12 = 19$ .
The L2 norm, 13, represents the direct, straight-line distance from the origin (0,0,0) to the point (3, -4, 12) in 3D space. The L1 norm, 19, represents the total distance traveled from the origin to that point if you were constrained to move only parallel to the x, y, and z axes.

Conclusion

In this lesson, we formalized the concept of vector magnitude by introducing norms and focused on the two most prevalent ones in your field of interest.

Here are the key takeaways:

A norm is a function that measures a vector's magnitude and must satisfy four properties: non-negativity, definiteness, homogeneity, and the triangle inequality.
The L2 norm ( $∥ v ∥_{2} = \sum v_{i}^{2}$ ) corresponds to our intuitive notion of Euclidean distance and measures the vector's "net displacement."
The L1 norm ( $∥ v ∥_{1} = \sum ∣ v_{i} ∣$ ) corresponds to the "Manhattan distance" and measures the "total change" or activity along each axis.
The geometric difference is captured by their unit balls: a circle for L2 and a diamond for L1, which has important implications for optimization in machine learning.

We now have a solid understanding of individual vectors—their direction, their similarity (dot product), and their magnitude (norms). The next logical step is to analyze collections of vectors. In our next lesson, we will explore the crucial concept of linear independence, which helps us understand when a set of vectors provides genuinely new information and when it is redundant.

Recommended resources