Vector Operations: Dot, Cross, and Projections

1. What is a Vector?

Before we can operate on vectors, let's clarify what they are. You've already worked with them implicitly as one-dimensional NumPy arrays. Formally, a vector is a mathematical object that has both magnitude (or length) and direction. This is in contrast to a scalar, which has only magnitude (like temperature or mass).

Vectors can be represented in a few ways:

• Geometrically: As an arrow in a 2D or 3D space, where the arrow's length is the magnitude and where it points is the direction.
• Analytically: As an ordered list of numbers called components, like [x, y] in 2D or [x, y, z] in 3D.

In data science, a single data point is often represented as a vector. For instance, a house could be represented by a vector [area, num_bedrooms, age], where each component is a feature.

To get a more formal introduction, please read the first part of the following tutorial.

2. The Dot Product and Vector Norm

One of the most fundamental operations is the dot product. It takes two vectors and returns a single scalar number.

For two vectors $\vec{u}=[u_1,u_2,...,u_n]$ and $\vec{v}=[v_1,v_2,...,v_n]$, their dot product (written as $\vec{u}\cdot \vec{v}$) is the sum of the products of their corresponding components:

The dot product is not just a calculation; it tells us about the geometric relationship between the two vectors. This relationship is defined by an alternative formula for the dot product:

Here:

• $||\vec{u}||$ is the norm or magnitude (length) of vector $\vec{u}$. For a vector $\vec{u}=[u_1,u_2,...,u_n]$, the most common norm (the L2 norm) is calculated as $||\vec{u}||=\sqrt{u_1^2+u_2^2+...+u_n^2}$. This is essentially an application of the Pythagorean theorem.
• $\theta$ is the angle between the two vectors.

Let's watch a short video that explains these concepts and shows how to compute them in Python using NumPy.

Orthogonality

A critical concept derived from the dot product is orthogonality. Look at the geometric formula: if the angle $\theta$ between two non-zero vectors is 90° ($\pi /2$ radians), then $\cos (90^{\circ })=0$. This means their dot product will be zero.

Therefore, two vectors $\vec{u}$ and $\vec{v}$ are orthogonal (perpendicular) if and only if $\vec{u}\cdot \vec{v}=0$. This is a simple and powerful way to check if vectors are at a right angle to each other.

To dive deeper into the properties and applications of the dot product, please review the following resource.

3. Vector Projection

The concept of projection involves figuring out how much of one vector "points" in the direction of another. Imagine the sun is directly above vector $\vec{u}$. The shadow that vector $\vec{v}$ casts onto the line containing $\vec{u}$ is the projection of $\vec{v}$ onto $\vec{u}$.

This "shadow" is itself a vector. Its formula uses the dot product:

Let's break this down:

• $\frac{\vec{v}\cdot \vec{u}}{||\vec{u}|{|}^2}$ is a scalar value. It determines how long the projection vector is and in which direction it points (same as $\vec{u}$ if positive, opposite if negative).
• We then multiply this scalar by the vector $\vec{u}$ to create the new vector along the direction of $\vec{u}$.

The following image provides an excellent visual and formulaic summary.

The formula can be implemented in NumPy by combining the np.dot() and np.linalg.norm() functions you've already seen. For a good reference, check out the "Projection" section in the article Exercises in NumPy Vector Algebra by George Marklow.

4. The Cross Product

While the dot product results in a scalar, the cross product is an operation between two vectors that results in a new vector. This operation is defined only for vectors in 3D space.

Given $\vec{u}=[u_1,u_2,u_3]$ and $\vec{v}=[v_1,v_2,v_3]$, their cross product $\vec{u}\times \vec{v}$ is a new vector $\vec{w}$ with two very important properties:

1. Direction: The resulting vector $\vec{w}$ is orthogonal to both $\vec{u}$ and $\vec{v}$.
2. Magnitude: The magnitude of the resulting vector, $||\vec{w}||$, is equal to the area of the parallelogram formed by $\vec{u}$ and $\vec{v}$.

The calculation and properties of the cross product are best explained visually.

In Python, the cross product is easily calculated with NumPy's np.cross() function.

5. Applying Vector Operations in Python

Let's bring all these concepts together in a single Python example using NumPy. This will demonstrate how the mathematical formulas translate directly into code.

First, make sure you have NumPy imported:

import numpy as np

Now, let's define two 3D vectors:

u = np.array([2, 1, 0])
v = np.array([1, -1, 3])

1. Dot Product and Angle:
We can find the angle between u and v using the dot product formula $\cos (\theta )=\frac{\vec{u}\cdot \vec{v}}{||\vec{u}||||\vec{v}||}$.

# Calculate the dot product
dot_product = np.dot(u, v)

# Calculate the norms (magnitudes)
norm_u = np.linalg.norm(u)
norm_v = np.linalg.norm(v)

# Calculate the cosine of the angle
cos_theta = dot_product / (norm_u * norm_v)

# Calculate the angle in radians, then convert to degrees
angle_rad = np.arccos(cos_theta)
angle_deg = np.degrees(angle_rad)

print(f"Dot Product: {dot_product}")
print(f"Angle between vectors: {angle_deg:.2f} degrees")

2. Vector Projection:
Let's calculate the projection of v onto u.

# proj_u(v) = ( (v . u) / ||u||^2 ) * u
projection_vector = (dot_product / (norm_u**2)) * u

print(f"Projection of v onto u: {projection_vector}")

3. Cross Product:
Finally, let's compute the cross product and verify its orthogonality.

# Calculate the cross product
cross_product_vec = np.cross(u, v)
print(f"Cross Product u x v: {cross_product_vec}")

# Verify orthogonality: dot product of the result with the original vectors should be 0
dot_with_u = np.dot(cross_product_vec, u)
dot_with_v = np.dot(cross_product_vec, v)

# We use np.isclose to handle potential floating-point inaccuracies
print(f"Is cross product orthogonal to u? {np.isclose(dot_with_u, 0)}")
print(f"Is cross product orthogonal to v? {np.isclose(dot_with_v, 0)}")

For more examples of how to implement these and other vector algebra concepts in NumPy, the following article is a useful reference.

Conclusion

In this lesson, you have taken your first step into the world of linear algebra. You've moved from seeing vectors as just NumPy arrays to understanding their geometric meaning and the powerful operations that define their relationships.

Key Takeaways:

• A vector has both magnitude (norm) and direction, while a scalar has only magnitude.
• The dot product of two vectors results in a scalar. It is used to find the angle between them and to determine if they are orthogonal (dot product is 0).
• Vector projection uses the dot product to find the "shadow" of one vector onto another.
• The cross product (for 3D vectors) results in a new vector that is orthogonal to both original vectors. Its magnitude represents the area of the parallelogram they form.
• All these operations are efficiently implemented in Python using the NumPy library.

In our next lesson, we will expand from individual vectors to matrices. You will learn how to perform matrix operations like addition and multiplication. This is a crucial step, as matrices are the primary way we represent and manipulate entire datasets in machine learning.