Understanding and Calculating Cosine Similarity

Welcome to the final lesson of our first module on vectors. In our journey so far, we have covered vector operations, the geometric meaning of the dot product, and most recently, how to measure a vector's magnitude using norms. We've seen that the dot product formula, $a \cdot b = ∣∣ a ∣∣ ∣∣ b ∣∣ cos θ$ , elegantly connects algebra with geometry, linking vector components to the angle between them.

In this lesson, we will put all these pieces together to explore cosine similarity, a powerful metric for comparing vectors. By simply rearranging the dot product formula, we can isolate the cosine of the angle, $cos θ$ , giving us a pure measure of orientation, independent of the vectors' lengths. This concept is fundamental in many AI applications, from comparing the meaning of texts to building recommendation engines.

By the end of this lesson, you will be able to define cosine similarity and calculate it for any pair of vectors, completing your foundational toolkit for working with vector data.

Defining Cosine Similarity

Let's begin by revisiting the geometric definition of the dot product you learned a few lessons ago:

a \cdot b = ∣∣ a ∣ ∣_{2} ∣∣ b ∣ ∣_{2} cos θ

Here, $∣∣ a ∣ ∣_{2}$ and $∣∣ b ∣ ∣_{2}$ are the L2 norms (or magnitudes) of the vectors, and $θ$ is the angle between them.

If our goal is to measure how "similar" two vectors are based on their direction, the angle $θ$ is the perfect tool. A small angle means they point in a similar direction, while a large angle means they point in different directions. To get a numerical value for this similarity, we can simply rearrange the equation to solve for $cos θ$ .

This gives us the formula for cosine similarity:

Similarity (a, b) = cos θ = \frac{a \cdot b}{∣∣ a ∣ ∣ _{2} ∣∣ b ∣ ∣ _{2}}

The formula for cosine similarity calculates the cosine of the angle θ between two vectors, A and B. In AI, this can be used to measure the similarity between different data points, such as an apple and an orange represented as vectors.

The formula beautifully combines the two main concepts from our recent lessons: the dot product in the numerator and the product of the vector magnitudes (L2 norms) in the denominator. To solidify this definition, let's turn to a concise resource.

Understanding the Cosine Similarity Formula

This article from PingCap provides a clear and formal definition of cosine similarity and its geometric meaning.

Please read the sections "Definition and Basic Concept" and "Geometric Interpretation". Focus on how the formula relates the dot product and vector magnitudes to the angle between vectors.

Interpreting the Cosine Similarity Score

As you've just read, the value of cosine similarity ranges from -1 to 1, each value having a distinct geometric meaning:

Cosine Similarity = 1: This means $cos θ = 1$ , so the angle $θ$ is 0°. The vectors point in the exact same direction. They might have different lengths, but their orientation is identical.
Cosine Similarity = 0: This means $cos θ = 0$ , so the angle $θ$ is 90°. The vectors are orthogonal. In the context of similarity, this means they have nothing in common or are independent.
Cosine Similarity = -1: This means $cos θ = - 1$ , so the angle $θ$ is 180°. The vectors point in diametrically opposite directions.

The image below provides a great visual summary of these three cases.

This image illustrates the relationship between the angle θ between two vectors and their cosine similarity. A small angle gives a similarity score close to 1, a 90-degree angle gives a score of 0, and an angle approaching 180 degrees gives a score close to -1.

In many AI applications, such as text analysis, vector components represent counts or frequencies, which are always non-negative. In such cases, the vectors will lie in the first quadrant, and their cosine similarity will range from 0 (orthogonal) to 1 (identical direction).

How to Calculate Cosine Similarity

Calculating the cosine similarity between two vectors, $a$ and $b$ , is a straightforward three-step process that uses skills you've already learned:

Calculate the Numerator: Find the dot product of the two vectors, $a \cdot b$ .
Calculate the Denominator: Find the L2 norm (magnitude) of each vector, $∣∣ a ∣ ∣_{2}$ and $∣∣ b ∣ ∣_{2}$ , and multiply them together.
Divide: Divide the dot product by the product of the magnitudes.

Let's watch a video that walks through this entire process with a clear numerical example.

2.5 DS: Cosine similarity between two rows in a data table. Explains dot product and norm.

This video from the "Computing For All" channel provides a methodical breakdown of a cosine similarity calculation, reinforcing concepts like the dot product and L2 norm along the way.

Watch the following segments: Introduction: The video presents the main formula. Dot Product: A review of how to calculate the numerator. L2 Norm: A review of how to calculate the magnitude of a vector. Final Calculation: The video puts all the pieces together to get the final cosine similarity score.

Why Cosine Similarity is Crucial for AI

You might wonder, why not just use the Euclidean distance (the straight-line distance between the vector endpoints) to measure similarity? The key advantage of cosine similarity is its magnitude invariance. It focuses solely on the direction of the vectors, not their length.

This property is invaluable in fields like Natural Language Processing (NLP). Imagine you want to compare the topics of two documents:

Document A: "Math for AI is great."
Document B: "Math for AI is great. Learning math for AI is a great journey."

If we represent these documents as vectors of word counts, the vector for Document B will have a much larger magnitude because it's longer and repeats words. If we used Euclidean distance, they would seem very far apart. However, both documents are fundamentally about the same topic. Cosine similarity captures this perfectly. Because the vectors representing them point in a very similar direction in multi-dimensional space, their cosine similarity score will be close to 1, correctly identifying their thematic likeness regardless of document length.

The following video from StatQuest provides an excellent and intuitive explanation of this concept.

Cosine Similarity, Clearly Explained!!!

Josh Starmer clearly explains the intuition behind cosine similarity, why it's useful, and how it works in high-dimensional spaces common in AI.

First, watch the simple example of comparing "hello world" to "hello". This establishes the core idea. Next, see why magnitude doesn't matter by watching the explanation for "hello hello hello". This is the most crucial part for understanding magnitude invariance. Then, watch the summary of what the similarity values mean (0, 1, and in between). Finally, see how the formula allows us to perform these calculations in high-dimensional spaces, which is impossible to do visually but essential for real-world AI problems.

Conclusion

In this lesson, we unified our knowledge of dot products and vector norms to define and calculate cosine similarity. This metric is a cornerstone of many AI systems, allowing them to understand relationships in data by comparing the orientation of vectors.

Here are the main takeaways:

Definition: Cosine similarity is the cosine of the angle between two vectors, calculated as $cos θ = \frac{a \cdot b}{∣∣ a ∣ ∣ _{2} ∣∣ b ∣ ∣ _{2}}$ .
Interpretation: It provides a score from -1 to 1, where 1 means identical direction, 0 means they are orthogonal (unrelated), and -1 means they are opposite.
Key Property: Its most important feature is magnitude invariance, making it ideal for comparing data where magnitude (like document length or user activity level) can be misleading, such as in text analysis and recommendation systems.

This lesson concludes our module on vectors. You now have a solid understanding of how vectors are used to represent data and how fundamental operations like the dot product, norms, and cosine similarity allow us to manipulate and compare that data.

In our next module, we will move from individual vectors to collections of them. We'll see how to represent entire datasets as matrices and learn the essential operations for transforming data and solving systems of linear equations—skills that form the bedrock of machine learning algorithms.

Recommended resources

Defining Cosine Similarity

Interpreting the Cosine Similarity Score

How to Calculate Cosine Similarity

Why Cosine Similarity is Crucial for AI

Conclusion