Vector Operations: Addition, Scalar Multiplication, and Dot Products

1. Understanding Vectors: The Building Blocks of Data

Before we can operate on vectors, we need a clear understanding of what they are. In the context of machine learning, a vector can be seen in two complementary ways: as a geometric object with direction and length, and as a list of numbers representing a point in space.

To build this dual intuition, let's start with a video that beautifully illustrates these perspectives.

As the video explains, a vector is fundamentally a way to store a piece of data that has multiple attributes. For instance:

• A 2D vector [x, y] could represent a point on a map.
• A 3D vector [r, g, b] could represent a color.
• A vector with hundreds of dimensions could represent a single user's preferences on a streaming service, with each dimension corresponding to a different movie genre.

In machine learning, we almost always work with the numerical representation, but the geometric intuition is crucial for understanding what our operations are actually doing to the data.

2. Combining and Scaling Vectors

The two most basic things we can do with vectors are add them together and scale them. These operations form the foundation of everything else in linear algebra.

Vector Addition

Geometrically, adding two vectors, $\mathbf{v}$ and $\mathbf{w}$, means placing the tail of $\mathbf{w}$ at the tip of $\mathbf{v}$. The sum, $\mathbf{v}+\mathbf{w}$, is the new vector drawn from the origin to the tip of $\mathbf{w}$. You can think of this as chaining two movements together.

Numerically, the process is simpler: we just add the corresponding components.

If $\mathbf{v}=\left[\begin{array}{c}{v}_1\\ v_2\end{array}\right]$ and $\mathbf{w}=\left[\begin{array}{c}{w}_1\\ w_2\end{array}\right]$, then:

Scalar Multiplication

A scalar is just a regular number (not a vector). Multiplying a vector by a scalar "scales" it.

• Multiplying by a scalar $c>1$ stretches the vector.
• Multiplying by a scalar $0<c<1$ shrinks it.
• Multiplying by a negative scalar reverses its direction and scales it.

Numerically, you multiply every component of the vector by the scalar.

If $\alpha$ is a scalar and $\mathbf{v}=\left[\begin{array}{c}{v}_1\\ v_2\end{array}\right]$, then:

Let's watch the next segment of the 3Blue1Brown video to see these two operations in action.

From Theory to Practice: NumPy

Your goal is to build intelligent systems, which means you'll be implementing these concepts in code. Python's NumPy library is the standard tool for numerical computation. Let's see how these operations look in practice.

3. Measuring Relationships: The Dot Product

We can now add and scale vectors. But how do we measure the relationship between two vectors? For instance, how "similar" are they? Do they point in the same direction? Are they perpendicular? This is the role of the dot product (also called the scalar product or inner product).

The dot product of two vectors, $\mathbf{a}$ and $\mathbf{b}$, produces a single number (a scalar). It has a geometric and an algebraic definition that are equivalent.

1. Algebraic Definition: Multiply the corresponding components and sum the results.

   $$\mathbf{a}\cdot \mathbf{b}=a_1{b}_1+a_2{b}_2+\cdots +a_n{b}_n=\sum _{i=1}^n{a}_i{b}_i$$

   This is how you'll usually compute it.

2. Geometric Definition: Multiply the lengths of the two vectors and the cosine of the angle between them.

   $$\mathbf{a}\cdot \mathbf{b}=\Vert \mathbf{a}\Vert \Vert \mathbf{b}\Vert \cos (\theta )$$

   Where $\Vert \mathbf{a}\Vert$ denotes the length (or magnitude) of vector $\mathbf{a}$, and $\theta$ is the angle between $\mathbf{a}$ and $\mathbf{b}$. This definition gives us the intuition.

Let's watch a video that focuses on the intuition behind the dot product.

The sign of the dot product tells us about the angle between the vectors:

• Positive ($\mathbf{a}\cdot \mathbf{b}>0$): The angle $\theta$ is less than 90°. The vectors point in generally the same direction.
• Zero ($\mathbf{a}\cdot \mathbf{b}=0$): The angle $\theta$ is exactly 90°. The vectors are orthogonal (perpendicular). This is an extremely important property used throughout machine learning, for example, to find independent features in data.
• Negative ($\mathbf{a}\cdot \mathbf{b}<0$): The angle $\theta$ is greater than 90°. The vectors point in generally opposite directions.

Now, let's look at how to compute this in NumPy.

Conclusion

In this lesson, you've taken your first steps into the mathematical world that underpins modern AI. You moved from the discrete rules of logic to the continuous space of data represented by vectors.

Key Takeaways:

• Vectors as Data: Vectors are ordered lists of numbers that represent data points in a high-dimensional space.
• Vector Addition: Combines vectors by adding their corresponding components. Geometrically, it's like following one vector's path and then the other's.
• Scalar Multiplication: Scales a vector's magnitude and/or reverses its direction by multiplying every component by a single number.
• Dot Product: A crucial operation that results in a scalar. It measures the alignment between two vectors, with its sign indicating whether they point in similar, perpendicular, or opposite directions. A dot product of zero means the vectors are orthogonal.

These three operations—addition, scalar multiplication, and the dot product—are the foundational tools you will use repeatedly. They are the simple arithmetic from which complex models are built.

Preview of the Next Lesson:
Now that you can operate on individual vectors, the next logical step is to operate on entire sets of vectors at once. In the next lesson, you will learn to multiply matrices and vectors. This will introduce you to matrices as objects that can transform vectors or entire spaces of data—scaling, rotating, and shearing them. This concept of transformation is central to how linear regression models and neural networks process and learn from data.