Vectors in R^n: Dot Product and Geometric Interpretation

What is a Vector?

To start, it's essential to have a solid dual intuition for what a vector is. On one hand, a vector is simply a list of numbers. In machine learning, these can represent anything from the features of a house to a word embedding in a high-dimensional space. On the other hand, a vector is an arrow in a coordinate system, possessing both length and direction. Grasping both perspectives is crucial for building the geometric intuition that underpins much of linear algebra.

The following video from the "Essence of linear algebra" series by 3Blue1Brown provides an excellent introduction to this dual-faceted nature of vectors.

As the video explains, a vector in an $n$-dimensional space, denoted as ${\mathbb{R}}^n$, can be formally written as an ordered list (or n-tuple) of $n$ real numbers. For example, a vector $\mathbf{v}$ in ${\mathbb{R}}^3$ is written as:

where $v_1,v_2,v_3$ are real numbers corresponding to the vector's coordinates along the x, y, and z axes, respectively. This representation is what allows us to perform algebraic operations on geometric objects.

The Dot Product: Computation and Geometry

Now that we have a working definition of a vector, we can explore how to combine two of them. The most fundamental operation for this is the dot product. It takes two vectors and produces a single number—a scalar.

Algebraic Definition

The dot product of two vectors $\mathbf{v}$ and $\mathbf{w}$ in ${\mathbb{R}}^n$ is defined as the sum of the products of their corresponding components:

For example, if $\mathbf{v}=\left[\begin{array}{c}1\\ 2\end{array}\right]$ and $\mathbf{w}=\left[\begin{array}{c}3\\ 4\end{array}\right]$, their dot product is $\mathbf{v}\cdot \mathbf{w}=(1)(3)+(2)(4)=3+8=11$.

The following resource provides a formal definition and a rigorous derivation of the geometric meaning of the dot product.

Geometric Interpretation

As you saw in the reading, the simple algebraic operation of the dot product has a profound geometric meaning. The two key relationships are:

1. Length: The dot product of a vector with itself gives the square of its length (also called its magnitude or norm, denoted $\Vert \mathbf{v}\Vert$). $$\mathbf{v}\cdot \mathbf{v}=v_1^2+v_2^2+\cdots +v_n^2=\Vert \mathbf{v}{\Vert }^2$$
2. Angle: The dot product of two different vectors is related to the angle $\theta$ between them. $$\mathbf{v}\cdot \mathbf{w}=\Vert \mathbf{v}\Vert \Vert \mathbf{w}\Vert \cos \theta$$

This second formula is the cornerstone of the dot product's utility. It tells us that the dot product is fundamentally a measure of how much two vectors "agree" or point in the same direction.

This image summarizes the two ways to compute the dot product: the algebraic component form and the geometric formula involving the angle between the vectors. It also visualizes the projection of vector A onto vector B, which is related to the dot product.

The geometric formula also reveals that the dot product $\mathbf{v}\cdot \mathbf{w}$ can be interpreted as the length of the projection of $\mathbf{w}$ onto $\mathbf{v}$, multiplied by the length of $\mathbf{v}$. The next video provides a superb visual intuition for this idea of projection.

The Dot Product as a Measure of Similarity

Your work in data engineering and future work in MI will often involve measuring the "similarity" between high-dimensional vectors (like word embeddings or neuron activation patterns). The dot product provides a direct way to do this.

By rearranging the geometric formula, we get an expression for the cosine of the angle between the vectors:

This quantity is known as cosine similarity. It ranges from -1 to 1 and tells us about the orientation of the vectors, independent of their magnitudes.

• If $\mathbf{v}\cdot \mathbf{w}>0$, then $\cos \theta >0$, meaning $0\le \theta <90^{\circ }$. The vectors point in generally the same direction. A larger positive value implies a smaller angle and thus higher similarity.
• If $\mathbf{v}\cdot \mathbf{w}=0$, then $\cos \theta =0$, meaning $\theta =90^{\circ }$. The vectors are orthogonal (perpendicular). This is a crucial concept we will return to throughout this course. It signifies a complete lack of directional alignment.
• If $\mathbf{v}\cdot \mathbf{w}<0$, then $\cos \theta <0$, meaning $90^{\circ }<\theta \le 180^{\circ }$. The vectors point in generally opposite directions (dissimilar).

This diagram shows how the dot product and cosine similarity between a reference vector (P1) and several other vectors correlate with our visual intuition of similarity. P1 and P3 are highly similar (high positive dot product), while P1 and P4 are dissimilar (negative dot product).

This concept is not just an abstract geometric idea. As noted in the supplementary PDF "Dot products and geometry in Rn" from Michigan State University, the correlation coefficient used in statistics is precisely the cosine of the angle between two mean-centered data vectors. This provides a direct link between linear algebra and the data analysis you're familiar with.

Practice Problem

Let's put this into practice. Consider two vectors in ${\mathbb{R}}^4$:

1. Calculate the dot product $\mathbf{v}\cdot \mathbf{w}$.
2. Calculate the lengths of each vector, $\Vert \mathbf{v}\Vert$ and $\Vert \mathbf{w}\Vert$.
3. Calculate the angle $\theta$ between them in degrees. Are they more similar or dissimilar?

Solution:

1. $\mathbf{v}\cdot \mathbf{w}=(2)(1)+(1)(4)+(-1)(2)+(3)(-1)=2+4-2-3=1$
2. $\Vert \mathbf{v}\Vert =\sqrt{2^2+1^2+(-1{)}^2+3^2}=\sqrt{4+1+1+9}=\sqrt{15}$
   $\Vert \mathbf{w}\Vert =\sqrt{1^2+4^2+2^2+(-1{)}^2}=\sqrt{1+16+4+1}=\sqrt{22}$
3. $\cos \theta =\frac{\mathbf{v}\cdot \mathbf{w}}{\Vert \mathbf{v}\Vert \Vert \mathbf{w}\Vert }=\frac{1}{\sqrt{15}\sqrt{22}}=\frac{1}{\sqrt{330}}\approx 0.055$
   $\theta =\arccos (0.055)\approx 86.8^{\circ }$

Since the dot product is positive and the angle is close to $90^{\circ }$, the vectors are nearly orthogonal but point in a slightly similar direction.

Conclusion

In this lesson, we established the foundational concept of vectors in ${\mathbb{R}}^n$ and explored the dot product, a simple yet powerful tool that connects algebra to geometry.

Here are the key takeaways:

• Vectors can be understood both as lists of numbers (a CS perspective) and as arrows in space (a geometric perspective).
• The dot product is computed as $\mathbf{v}\cdot \mathbf{w}=\sum v_i{w}_i$.
• Geometrically, the dot product relates to vector lengths and the angle between them via the formula $\mathbf{v}\cdot \mathbf{w}=\Vert \mathbf{v}\Vert \Vert \mathbf{w}\Vert \cos \theta$.
• The sign of the dot product tells us if vectors point in similar ($>0$), orthogonal ($=0$), or dissimilar ($<0$) directions, making it a cornerstone for concepts like cosine similarity.

In our next lesson, we will formalize the concept of vector length by studying norms. We'll see how different ways of measuring length (like the L1 and L2 norms) can have different properties and applications, particularly in the context of machine learning optimization and regularization.