Dot Product: Angle and Projection

In our previous lesson, we established the "how" of the dot product—the algebraic process of multiplying corresponding vector components and summing them to get a single scalar value. As promised, this lesson shifts our focus to the "why." We will explore the rich geometric meaning behind that scalar, uncovering what it tells us about the relationship between two vectors.

Your goal is to apply mathematics to AI, and understanding the geometry of vector operations is crucial. The dot product is not just a calculation; it's a powerful tool for measuring alignment and similarity, concepts that are at the heart of many machine learning models. By the end of this lesson, you will be able to interpret the dot product in terms of the angle between two vectors and the projection of one vector onto another.

The Dot Product and the Angle Between Vectors

In the last lesson, we defined the dot product of $a$ and $b$ as $a \cdot b = a_{1} b_{1} + a_{2} b_{2} + \dots + a_{n} b_{n}$ . It turns out this algebraic calculation is equivalent to a purely geometric one:

a \cdot b = ∣ a ∣∣ b ∣ cos θ

Here, $∣ a ∣$ and $∣ b ∣$ are the magnitudes (lengths) of the vectors, and $θ$ is the angle between them.

This image illustrates the geometric formula for the dot product, showing vectors $\vec{a}$ and $\vec{b}$, their magnitudes, and the angle $\theta$ between them.

This single equation provides a profound link between algebra and geometry. The article below provides an excellent, intuitive derivation of this formula, starting with the simplest case of unit vectors (vectors of length 1) and building up from there.

The Geometry Behind the Dot Product: Unit Vectors, Projections, and Intuition | Towards Data Science

This article from Towards Data Science builds the geometric intuition for the dot product from the ground up.

First, read the section "Unit Vector". This part shows why for two unit vectors, their dot product is simply \cos\theta. Then, skip to the section "The Dot Product Geometric Interpretation", which extends this logic to vectors of any length, arriving at the full geometric formula.

Interpreting the Sign of the Dot Product

The formula $a \cdot b = ∣ a ∣∣ b ∣ cos θ$ gives us a powerful way to understand the relationship between two vectors just by looking at the sign of their dot product. Since magnitudes $∣ a ∣$ and $∣ b ∣$ are always non-negative, the sign is determined entirely by $cos θ$ .

Positive Dot Product ( $a \cdot b > 0$ ): This means $cos θ > 0$ , which occurs when the angle $θ$ is acute ( $0^{\circ} \leq θ < 9 0^{\circ}$ ). The vectors are pointing in a generally similar direction. The more aligned they are, the larger the dot product.
Zero Dot Product ( $a \cdot b = 0$ ): This means $cos θ = 0$ , which occurs when $θ = 9 0^{\circ}$ . The vectors are orthogonal (perpendicular). This is a vital concept in AI, as it can imply that two features are independent or uncorrelated.
Negative Dot Product ( $a \cdot b < 0$ ): This means $cos θ < 0$ , which occurs when $θ$ is obtuse ( $9 0^{\circ} < θ \leq 18 0^{\circ}$ ). The vectors are pointing in generally opposite directions.

The following reading provides clear visual examples for each of these three cases.

Linear Algebra 101 for AI/ML – Part 2 - Backprop

This article from Backprop is written specifically for an AI/ML audience and explains the geometric interpretation of the dot product.

Start by reading the brief introduction under the "Visual Perspective" heading to see the cosine formula again. Then, scroll down past the interactive playground and examine the series of images and explanations starting from this point. These illustrations clearly show the relationship between vector alignment and the sign of the dot product.

The Dot Product as Projection

Another powerful interpretation is that the dot product relates to projection. Think of projection as the "shadow" that one vector casts onto another. The scalar projection of $a$ onto $b$ answers the question: "How much of vector $a$ is pointing in the direction of vector $b$ ?"

This image shows the projection of vector $\vec{a}$ onto vector $\vec{b}$. The length of this projection is given by $|\vec{a}|\cos\theta$, which can be calculated using the dot product.

From basic trigonometry, the length of this shadow is $∣ a ∣ cos θ$ . We can find this value directly from our dot product formula:

a \cdot b = ∣ a ∣∣ b ∣ cos θ

If we divide by the magnitude of $b$ , we get:

\frac{a \cdot b}{∣ b ∣} = ∣ a ∣ cos θ

This reveals that the scalar projection of $a$ onto $b$ is calculated by $\frac{a \cdot b}{∣ b ∣}$ . It is the dot product of $a$ with the unit vector of $b$ .

This gives us two ways to think about the geometric meaning of $a \cdot b$ :

The length of $a$ multiplied by the length of $b$ 's projection onto $a$ .
The length of $b$ multiplied by the length of $a$ 's projection onto $b$ .

The following video from 3Blue1Brown provides a brilliant visualization of this concept and explains why these two seemingly different perspectives always yield the same result.

Dot products and duality | Chapter 9, Essence of linear algebra

The "Essence of linear algebra" series by 3Blue1Brown is famous for its deep visual intuition. This clip explains the projection-based view of the dot product.

Watch from the beginning of the geometric interpretation at this segment to understand how projection defines the dot product. Then, continue watching the next part, which addresses why the order doesn't matter, providing intuition for the symmetry of the operation.

The concept of projection is everywhere in machine learning. For example, when an algorithm tries to find the best-fit line through a set of data points, it is essentially projecting the data onto a line to minimize the error.

For another excellent, intuitive take on this, the video from Looking Glass Universe builds the concept from the question "How much do two vectors point in the same direction?".

The meaning of the dot product | Linear algebra makes sense

This video offers a slightly different, highly intuitive way to arrive at the same geometric understanding of the dot product.

Watch the first few minutes, from the start up to where the narrator introduces the idea of 'X', a number that quantifies alignment. Then watch the next segment where he shows that this number X is cos(theta). This provides a complementary way to think about the concepts we've discussed.

Conclusion

In this lesson, we moved beyond mere calculation to understand the geometric essence of the dot product. This is a significant step in developing the intuition needed for advanced AI topics.

Here are the key takeaways:

The dot product has an equivalent geometric definition: $a \cdot b = ∣ a ∣∣ b ∣ cos θ$ .
The sign of the dot product tells you about the angle between vectors: positive for acute angles (< 90°), zero for orthogonal vectors (90°), and negative for obtuse angles (> 90°).
The dot product measures projection. The term $∣ a ∣ cos θ$ represents the scalar projection of $a$ onto $b$ . The dot product can be seen as multiplying this projection length by the length of $b$ .

We've now seen that the vector magnitude, or length, is a key component of the geometric interpretation. In our next lesson, we will formalize the concept of vector magnitude by learning how to calculate the L1 and L2 norms of a vector. This will equip us with all the necessary tools to later define and use cosine similarity, a critical metric for comparing vectors in AI applications.

Recommended resources

The Dot Product and the Angle Between Vectors

Interpreting the Sign of the Dot Product

The Dot Product as Projection

Conclusion