Vector Operations: Addition and Scalar Multiplication

Vector Addition

Just as you can add two numbers, you can add two vectors. This operation combines the information from two data points into a new, single data point. The process is straightforward and works identically from both the numerical and geometric perspectives.

The Numerical View: Element-wise Addition

From the "list of numbers" perspective, adding two vectors is simple: you add their corresponding elements. For this to work, the vectors must have the same number of dimensions (i.e., the same number of elements).

For example, let's take two 2-dimensional vectors:

Their sum, $\vec{v}+\vec{w}$, is calculated as:

Each element of the new vector is the sum of the elements at the same position in the original vectors.

The following reading provides a concise definition of this process.

The Geometric View: The Tip-to-Tail Method

Geometrically, adding vectors follows a "tip-to-tail" rule. To visualize the sum $\vec{v}+\vec{w}$, you can imagine taking the vector $\vec{w}$ and moving it without changing its length or direction, so that its tail is at the tip (or head) of $\vec{v}$. The resulting vector sum, $\vec{v}+\vec{w}$, is a new vector that starts at the origin (the tail of $\vec{v}$) and ends at the tip of the repositioned $\vec{w}$.

The order of addition doesn't matter; you get the same result if you place the tail of $\vec{v}$ at the tip of $\vec{w}$. This property is called commutativity.

The excellent video series "Essence of linear algebra" provides a fantastic visualization that connects the geometric tip-to-tail method with the numerical component-wise addition we just discussed.

Scalar Multiplication

The second fundamental operation is scalar multiplication. As we touched on in the last lesson, a scalar is simply a number. Scalar multiplication is the operation of multiplying a vector by a scalar, which has the effect of scaling—stretching, shrinking, or reversing—the vector.

The Numerical View: Element-wise Multiplication

Numerically, you perform scalar multiplication by multiplying every element of the vector by the scalar.

For instance, if we take our vector $\vec{v}=\left[\begin{array}{c}2\\ 1\end{array}\right]$ and multiply it by the scalar 3:

This is a simple, element-wise multiplication.

The Geometric View: Scaling the Vector

Geometrically, scalar multiplication changes the length (magnitude) and potentially the direction of the vector, but it always remains on the same line through the origin.

• Multiplying by a scalar > 1 stretches the vector. (e.g., 2 * v doubles its length).
• Multiplying by a scalar between 0 and 1 shrinks the vector. (e.g., 0.5 * v halves its length).
• Multiplying by a negative scalar reverses the vector's direction and then scales it. (e.g., -1 * v points in the exact opposite direction).

The following reading and video provide further detail and visualization for this concept.

Why These Operations Matter for AI

Vector addition and scalar multiplication are not just abstract mathematical exercises; they are at the heart of how machine learning models work.

• Averaging and Combining Features: Imagine you have two vectors representing two different houses. To find the "average" house, you would add the two vectors and then multiply the result by the scalar 0.5. This creates a new vector representing a house with the average features of the original two.
• Model Updates during Learning: Many machine learning algorithms, like the ones used to train neural networks, learn by adjusting their internal parameters. This adjustment is often done using an update rule that looks something like this: new_parameters = old_parameters - learning_rate * gradient. Here, learning_rate is a scalar and gradient is a vector. The operation involves scalar multiplication (learning_rate * gradient) and vector subtraction (which is just addition of a negative vector), exactly the operations we've learned today.

These two simple operations allow us to navigate the "feature space" we discussed in the last lesson, moving from one point to another in a controlled way to combine data or find optimal model parameters.

Conclusion

In this lesson, we moved from defining vectors to manipulating them. You learned how to perform the two most basic vector operations and what they represent.

Here are the key takeaways:

• Vector Addition: Is performed element-wise. Geometrically, it corresponds to the "tip-to-tail" method, creating a resultant vector that represents a combined displacement.
• Scalar Multiplication: Involves multiplying every element of a vector by a single number (a scalar). Geometrically, this scales the vector, stretching, shrinking, and/or reversing it along its original line.
• Foundation for AI: These operations are fundamental tools for manipulating data represented as vectors, enabling tasks like averaging data points and updating model parameters during training.

Now that you can add and scale vectors, the next logical step is to understand how to multiply two vectors together. In our next lesson, we will explore the dot product, a different type of multiplication that reveals crucial information about the relationship between two vectors, such as the angle between them.