Dot Products and Vector Alignment

Welcome back. Last lesson, you used shapes to see that a $1\times n$ row vector multiplied by an $n\times 1$ column vector produces one number. Today we will calculate that number—the dot product—and see why it tells us something about how two vectors point relative to each other.

This is a foundation for reading Attention Is All You Need: attention uses dot products to score pairs of learned vectors. For now, the goal is to understand the calculation and its geometric meaning, before adding the rest of the attention mechanism. Allow about 35 minutes, including the short videos.

Calculating a dot product

Two vectors must have the same number of components to take their dot product. Pair components in the same positions, multiply each pair, and add the products. For vectors with $n$ components,

The result is a scalar—one number—not another vector. For example,

Watch Professor Dave Explains calculate a dot product from components. Pay particular attention to the negative component: it contributes a negative product to the sum.

This is also the single-value operation you encountered in the previous lesson. If $a$ and $b$ are both written as $n\times 1$ columns, we turn $a$ into a $1\times n$ row to form $a^{\mathsf{T}}b$. Its shape is $1\times 1$, and its value is $a\cdot b$. The same calculation occurs inside matrix multiplication: each output entry pairs one row of the first matrix with one column of the second.

A useful arithmetic check is to write all the paired products before adding them. In a vector with many components, it is easy to overlook a negative sign or pair values from different positions.

Why the answer says something about direction

Imagine two arrows drawn from the same starting point. When they point in similar directions, one has a substantial component along the other. When they are perpendicular, neither points along the other. When they point against each other, that component is negative. The dot product captures this signed alignment.

For nonzero vectors $a$ and $b$, the relationship is

where $\Vert a\Vert$ and $\Vert b\Vert$ are their lengths, and $\theta$ is the angle between them. The cosine is positive for an angle smaller than $90^{\circ }$, zero at $90^{\circ }$, and negative for an angle larger than $90^{\circ }$. Because the lengths are positive, they do not change the sign:

• Positive dot product: the vectors point partly in the same direction.
• Zero dot product: the vectors are perpendicular, also called orthogonal.
• Negative dot product: the vectors point partly in opposing directions.

For a visual explanation, watch Looking Glass Universe build this idea first with unit-length vectors—vectors whose lengths are one—and then connect it to vectors of other lengths.

We can check the geometry using components. Keep $a=(2,1)$ fixed and compare three vectors:

The first pair has positive alignment. The second is perpendicular—even though neither vector is zero. The third points in exactly opposite directions, since $(-2,-1)$ is $-1$ times $a$.

There is one important exception to the angle interpretation: a zero vector has dot product zero with every vector, but it has no direction. So a zero dot product means “perpendicular” only when both vectors are nonzero.

Direction is not the whole value

The dot product measures alignment and depends on vector lengths. To see the difference, compare $q=(1,0)$ with two vectors:

The first dot product is larger, yet $(1,0)$ points exactly in $q$’s direction, while $(10,10)$ points at an angle. The first vector’s greater length accounts for its larger dot product. Multiplying either input vector by a positive number multiplies the dot product by that number without changing the angle.

If you want to compare direction alone, divide out the lengths:

This quantity, often called cosine similarity, lies between $-1$ and $1$ for nonzero vectors. In the example above, the cosine similarities are $1/\sqrt{2}$ for $(10,10)$ and $1$ for $(1,0)$. That correctly identifies the second vector as more closely aligned in direction.

You can calculate a vector’s length from its components:

This also gives a useful connection between the two ideas. A vector dotted with itself has no opposing components:

For example, $(2,1)\cdot (2,1)=5$, so its length is $\sqrt{5}$. You do not need to calculate lengths or angles to compute an ordinary dot product; component-wise multiplication and addition are enough. The lengths explain how to interpret its magnitude.

The connection to attention

Later in this course, an attention operation will compute dot products between vectors called queries and keys. At this stage, you can read such a score as the result of multiplying corresponding learned components and summing them. A larger raw score can come from stronger directional alignment, larger vector lengths, or both; it is not automatically a pure measure of angle, nor proof that two words mean the same thing.

That distinction is why it helps to keep the two views together: the component formula tells you how to calculate the dot product, while the angle formula tells you what influences its value.

Key takeaways

A dot product pairs equal-length vectors’ components, multiplies each pair, and sums to produce one number. For nonzero vectors, its sign distinguishes broadly similar, perpendicular, and opposing directions. Its size also depends on both vector lengths; dividing by those lengths isolates directional alignment as cosine similarity.

Next, we will use scalar numbers as weights to combine several vectors into a single weighted sum—another operation that will appear when we build attention.