In our previous lesson, we explored the geometric meaning of the dot product, discovering that it's deeply connected to the angle between vectors and the concept of projection. A key part of that geometric formula, , was the magnitude (or length) of the vectors, denoted as . We used this concept intuitively, but now it's time to formalize it.
This lesson focuses on how to precisely measure a vector's magnitude. You'll learn that there isn't just one way to do this. We will explore mathematical tools called norms, which are functions that take a vector and output a non-negative scalar value representing its size. For your goal of learning math for AI, understanding norms is essential as they are fundamental to how machine learning models measure error and regularize parameters. We will concentrate on the two most important norms in this field: the L1 norm and the L2 norm.
By the end of this lesson, you will be able to calculate both the L1 and L2 norms for any given vector.
What is a Vector Norm?
Think of a norm as a function that quantifies the "length" or "size" of a vector. You give it a vector, and it returns a single positive number. Different norm functions can give you different notions of length for the same vector, each with its own properties and uses.
Most of the norms you will encounter in machine learning belong to a family called p-norms. This family is governed by a single, general equation.
Vector Norms in Machine Learning
This article from Towards Data Science introduces the concept of vector norms and the general p-norm equation that unifies them.
First, read the section under the heading "Vector Norms" for a clear definition. Then, continue to the next section, "The Standard Norm Equation", to familiarize yourself with the general p-norm formula. This formula is the template from which we'll derive the L1 and L2 norms.
As you just read, the p-norm formula is:
Here, is a real number greater than or equal to 1 that defines the specific norm. By choosing different values for , we get different ways of measuring vector length. Let's explore the two most important cases for and .
The L2 Norm: Our Familiar Euclidean Distance
The most common way to measure a vector's length is the L2 norm, which is what we get when we set in the general formula.
This should look very familiar! It's simply the formula for the straight-line distance from the origin to a point in space, derived from the Pythagorean theorem. Because it represents the standard distance in Euclidean space, the L2 norm is also called the Euclidean norm. This is the measure of magnitude we were implicitly using in the last lesson.
Let's calculate the L2 norm for the vector from the introductory image.
This confirms the result in the diagram.
The L2 norm is so common that if a norm is written without a subscript, like , you should assume it's the L2 norm. The following video provides a great introduction to the L2 norm and contrasts it with the L1 norm, which we'll discuss next.
This video from the ritvikmath channel gives a clear, visual explanation of the L2 and L1 norms and how to generalize them.
First, watch the segment that introduces the L2 norm, showing how it relates to Cartesian distance. Then, skip ahead to the end of the video to see how the formula is generalized to n-dimensions.
The L1 Norm: The "Manhattan" Distance
Now let's see what happens when we set in the p-norm formula. This gives us the L1 norm.
The formula simplifies to just the sum of the absolute values of the vector's components. This norm is also known as the Manhattan norm or taxicab distance. Imagine navigating a city grid where you can only travel along the blocks, not cut diagonally through them. The L1 norm represents the total distance you'd have to travel.
Let's calculate the L1 norm for our vector .
Again, this matches the result from our initial diagram.
Now, let's watch the next part of the same video to build our intuition for the L1 norm.
This segment explains the L1 norm and its geometric interpretation.
Watch the portion of the video that defines the L1 norm. Pay close attention to the geometric analogy of adding the "legs" of the triangle instead of calculating the hypotenuse.
Why Have Different Norms? A Visual Intuition
You might be wondering why we need different ways to measure length. The choice of norm has significant implications in machine learning. The L2 norm is sensitive to large values because it squares them, making it suitable for penalizing large errors. The L1 norm is less sensitive to outliers and, due to its geometric properties, tends to produce sparse solutions (i.e., models where many parameters are zero), which is useful for feature selection.
To understand this intuitively, let's look at the set of all 2D vectors that have a norm of 1.
- For the L2 norm, the equation is , which is the equation of a circle.
- For the L1 norm, the equation is , which describes a diamond (or a square rotated by 45 degrees).
This fundamental difference in shape is the key to their different behaviors in machine learning algorithms.
This final clip from the ritvikmath video provides the most important visual for understanding the difference between L1 and L2 norms.
Watch the segment that visualizes the shapes created by setting the L2 norm and L1 norm to a constant value. The contrast between the circle (L2) and the diamond (L1) is a crucial concept.
This "diamond" shape of the L1 norm has sharp corners that lie on the axes. When used in machine learning optimization (a topic we'll touch on much later), solutions are often pushed towards these corners, which correspond to a component being exactly zero. This is why L1 regularization is said to promote sparsity.
To solidify this connection, read the following short text.
Learn Vector Norms and Magnitude | Section
This resource from Codefinity provides concise definitions and highlights the different applications of L1 and L2 norms in data science.
Read the introductory text that defines the L2 norm and the L1 norm. Then, read the paragraph that explains their different properties and uses, starting from this point.
Conclusion
In this lesson, we formalized the concept of vector magnitude by introducing norms. We focused on the two most critical norms for AI: L1 and L2.
Here are the key takeaways:
- A norm is a function that measures the magnitude or length of a vector.
- The L2 norm (Euclidean norm) calculates the straight-line distance from the origin. It is calculated as .
- The L1 norm (Manhattan norm) calculates the "taxicab" distance. It is calculated as .
- The choice of norm has practical consequences in machine learning, with L2 being a general-purpose error measure and L1 being useful for encouraging sparsity.
You now possess all the tools needed to tackle our next topic. We have covered the dot product and, in this lesson, the norms used to calculate vector magnitude. In the next lesson, we will combine these concepts to calculate cosine similarity, a vital metric used to measure the orientation or "similarity" between two vectors, with applications ranging from document analysis to recommendation engines.