Vector Operations

Welcome to your first lesson! We'll begin our journey into the mathematics of machine learning by building a solid foundation in linear algebra. This lesson focuses on the most fundamental building block: the vector. Our goal is to refresh your understanding of what vectors are and how to perform basic operations with them, such as addition, subtraction, scaling, and measuring their length. These operations are not just abstract math; they are the very tools used to manipulate data and update models in almost every machine learning algorithm.

1. What is a Vector?

To start, let's build a strong intuition for what vectors represent. They can be understood from a few different perspectives—as arrows in space, as lists of numbers, or as abstract mathematical objects. For our purposes in machine learning, the most useful approach is to combine the first two.

A vector can represent a data point or a set of features. For example, in a simple housing price model, a vector $h = area (sq. ft.) number of bedrooms distance to city center (km)$ could represent a single house. Each number in the vector is a feature, and the vector as a whole is a single data point in a multi-dimensional "feature space."

To get a dynamic and intuitive introduction to this concept, let's watch the beginning of a video from the excellent "Essence of Linear Algebra" series by 3Blue1Brown.

Vectors | Chapter 1, Essence of linear algebra

Watch the first part of this video, "Vectors | Chapter 1, Essence of linear algebra."

Focus on the connection between the geometric view (an arrow with its tail at the origin) and the computer science view (a list of numbers, or coordinates). Pay attention to how a pair or triplet of numbers provides instructions to get from the origin to the vector's tip. Watch up to the end of the introduction.

As the video explains, we will primarily think of vectors in two ways that are two sides of the same coin:

Geometrically: An arrow of a specific length and direction, rooted at the origin of a coordinate system (like the x-y plane).
Numerically: An ordered list of numbers, written as a column, where each number corresponds to a coordinate along an axis.

For a vector $v$ in a 2D plane, we write:

v = [v_{1} v_{2}]

Here, $v_{1}$ is the component along the x-axis and $v_{2}$ is the component along the y-axis. This duality is key: the geometry gives us intuition, and the numbers allow us to compute things precisely.

2. Basic Vector Operations

The two fundamental operations in linear algebra are vector addition and scalar multiplication. Subtraction is just a variation of addition.

Vector Addition and Subtraction

Geometrically, adding two vectors, say $v$ and $w$ , follows the "tip-to-tail" rule: you move the tail of $w$ to the tip of $v$ . The new vector that goes from the origin to the tip of the moved $w$ is their sum, $v + w$ .

Numerically, the process is even simpler: you just add the corresponding components.

v + w = [v_{1} v_{2}] + [w_{1} w_{2}] = [v_{1} + w_{1} v_{2} + w_{2}]

Vector subtraction, $v - w$ , is defined as adding the negative of $w$ . The vector $- w$ has the same length as $w$ but points in the opposite direction. Numerically, this means subtracting the components:

v - w = [v_{1} v_{2}] - [w_{1} w_{2}] = [v_{1} - w_{1} v_{2} - w_{2}]

The following images provide a clear static visual for these operations.

This image shows how vector addition corresponds to component-wise addition and is visualized by the parallelogram rule (a result of the tip-to-tail method). It also illustrates scalar multiplication.

This image illustrates the geometric result of subtracting one vector from another. Notice that $\vec{v}_1 - \vec{v}_2$ is the vector that points from the tip of $\vec{v}_2$ to the tip of $\vec{v}_1$.

This image illustrates the geometric result of subtracting one vector from another. Notice that $v_{1} - v_{2}$ is the vector that points from the tip of $v_{2}$ to the tip of $v_{1}$ .

Scalar Multiplication

Multiplying a vector $v$ by a number, or a scalar $c$ , scales the vector.

If $c > 1$ , the vector is stretched.
If $0 < c < 1$ , the vector is shrunk.
If $c < 0$ , the vector is flipped in direction and then scaled.

Numerically, you multiply every component of the vector by the scalar:

c v = c [v_{1} v_{2}] = [c v_{1} c v_{2}]

To see these operations animated and explained with the same excellent intuition, let's return to the 3Blue1Brown video. Then, we'll watch a clip from an Imperial College London video that offers a slightly different perspective and formally introduces vector subtraction.

Vectors | Chapter 1, Essence of linear algebra

Watch the next two segments of "Vectors | Chapter 1, Essence of linear algebra."

First, watch the section on vector addition to see the tip-to-tail method and its link to component-wise addition. Next, watch the explanation of scalar multiplication to understand how numbers scale vectors.

M4ML - Linear Algebra - 1.4 Operations with vectors

Now, watch these clips from "M4ML - Linear Algebra - 1.4 Operations with vectors" by Imperial College London.

This video reinforces the concepts from a slightly more formal, lecture-style perspective. Watch from the beginning of the explanation to see the geometric rules for addition and how this translates to adding components. Then, watch the specific segment on vector subtraction to see how it's defined as the addition of a negative vector.

3. Measuring Vector Length: The Norm

A critical property of a vector is its length, also known as its magnitude or norm. In machine learning, we often need to measure the "size" of vectors—for instance, to calculate the distance between two data points or to measure the size of an error vector.

The most common way to measure length is the Euclidean norm, or L2 norm. For a 2D vector $v = [v_{1} v_{2}]$ , its L2 norm is written as $∥ v ∥_{2}$ . You can see from the geometry that this is a direct application of the Pythagorean theorem:

∥ v ∥_{2} = v_{1}^{2} + v_{2}^{2}

This generalizes to any number of dimensions ( $n$ ). For a vector $v = v_{1} v_{2} ⋮ v_{n}$ , the formula is:

∥ v ∥_{2} = v_{1}^{2} + v_{2}^{2} + \dots + v_{n}^{2} = i = 1 \sum n v_{i}^{2}

While the L2 norm is the most common, it's not the only way to measure a vector's length. Two other norms that are important in machine learning are the L1 norm and the L-infinity norm. You'll encounter these later in the context of regularization, a technique to prevent model overfitting.

The following reading provides a concise summary of these norms.

Introduction to Linear Algebra for Applied Machine Learning ...

This article provides a clear definition of different vector norms.

Read the section titled Vector norms. Start from the beginning of that section and read through the subsections on the Euclidean norm, Manhattan norm, and Max norm. Focus on understanding the formula for each and the conceptual difference between them. Don't worry about the formal properties (homogeneity, triangle inequality, etc.) for now; focus on the definitions.

4. Practice Problems

Let's put these concepts into practice. Given the following two vectors:
$a = [4 - 2], b = [- 1 5]$
Calculate the following:

$a + b$
$a - b$
$3 a$
$∥ b ∥_{2}$ (the L2 norm of $b$ )
$∥ a ∥_{1}$ (the L1 norm of $a$ )

Take a moment to solve these on your own before checking the solutions below.

Solutions

Vector Addition:
$a + b = [4 + (- 1) - 2 + 5] = [33]$
Vector Subtraction:
$a - b = [4 - (- 1) - 2 - 5] = [5 - 7]$
Scalar Multiplication:
$3 a = 3 [4 - 2] = [3 \times 4 3 \times (- 2)] = [12 - 6]$
L2 Norm:
$∥ b ∥_{2} = (- 1)^{2} + 5^{2} = 1 + 25 = 26 \approx 5.1$
L1 Norm:
$∥ a ∥_{1} = ∣4∣ + ∣ - 2∣ = 4 + 2 = 6$

Conclusion

In this lesson, we have refreshed the essential concepts of vectors and their fundamental operations.

Key Takeaways:

Vectors can be viewed both as arrows in space (giving geometric intuition) and as lists of numbers (enabling computation).
In machine learning, vectors are used to represent data points, features, and model parameters.
Vector addition, subtraction, and scalar multiplication are performed component-wise.
The norm of a vector measures its length or magnitude. The most common is the L2 norm (Euclidean norm), which is calculated using the Pythagorean theorem.
Other norms like L1 and L-infinity also exist and have important applications in machine learning.

These basic operations are the absolute bedrock of linear algebra. In the next lesson, we will build directly on this foundation to explore the dot product, another crucial operation that allows us to understand the relationship between two vectors, including the angle between them.

Recommended resources