Introduction to Vector Spaces

What is a Vector? The Power of Abstraction

You might be used to thinking of vectors as arrows in space or as lists of numbers (e.g., a column in a dataset). While these are essential examples, the concept is far more general and powerful. Mathematicians and scientists found that many different types of objects—like polynomials, matrices, and functions—could be "added" together and "scaled" in ways that followed the same underlying rules.

Instead of developing separate theories for each of these object types, it is far more efficient to create a single, unified theory. This is the purpose of defining an abstract vector space.

To get a better sense of this motivation, let's start with a short, conceptual reading.

To build on this idea, the following video provides a brilliant visual intuition for why objects like functions can be treated as vectors.

The Formal Definition of a Vector Space

As you saw in the video, if we want to create a general theory, we need a formal definition. A vector space is a collection of objects (called vectors) for which two operations—addition and scalar multiplication—are defined and satisfy a specific set of rules, or axioms.

These axioms are not arbitrary; they are the minimum set of properties required for our standard algebraic manipulations to work consistently.

Let's look at the formal definition.

A structure $(V,F,+,\cdot )$ is a vector space if it satisfies these ten axioms, where $V$ is a set of vectors, $F$ is a field of scalars (for us, this will typically be the real numbers $\mathbb{R}$ or complex numbers $\mathbb{C}$), $+$ is vector addition, and $\cdot$ is scalar multiplication.

Verification in Action: Proving a Set is a Vector Space

Now comes the practical part: how do we use this definition? To show that a set $V$ with given operations is a vector space, you must systematically verify that all ten axioms hold.

Example 1: The Canonical Vector Space ${\mathbb{R}}^n$

The most familiar vector space is ${\mathbb{R}}^n$, the set of all $n$-tuples of real numbers, which we often write as column vectors. Let's walk through the formal proof that ${\mathbb{R}}^n$ is indeed a vector space over the field of real numbers $\mathbb{R}$ with the standard component-wise addition and scalar multiplication.

Example 2: The Vector Space of Matrices

In machine learning, you'll constantly work with matrices. The set of all $m\times n$ matrices with real entries, denoted $M_{m\times n}(\mathbb{R})$, also forms a vector space with the usual matrix addition and scalar multiplication.

The proof is very similar to the one for ${\mathbb{R}}^n$, as matrix operations are also defined element-wise.

Example 3: The Vector Space of Functions

As hinted at in the 3Blue1Brown video, the set of all real-valued functions defined on an interval also forms a vector space. This is a very important example in many areas of mathematics and physics, including functional analysis, which has deep connections to ML theory (e.g., kernel methods).

Properties Derived from the Axioms

The power of an axiomatic system is that we can derive all other properties of a vector space from just those ten axioms. This is the essence of mathematical proof and rigor. As an example, let's prove the uniqueness of the zero vector.

Theorem: In any vector space $V$, the zero vector is unique.

Proof:
Suppose there are two zero vectors, $0_1$ and $0_2$, in $V$.

1. By the definition of a zero vector (Axiom A5), for any vector $\mathbf{v}\in V$, $\mathbf{v}+0=\mathbf{v}$.
2. Let's use $0_1$ as our "vector" and $0_2$ as our "zero vector". From A5, we get:
   $0_1+0_2=0_1$
3. Now, let's use $0_2$ as our "vector" and $0_1$ as our "zero vector". From A5, we get:
   $0_2+0_1=0_2$
4. By the commutativity of addition (Axiom A3), we know that $0_1+0_2=0_2+0_1$.
5. Therefore, by combining our results from steps 2 and 3, we must have $0_1=0_2$.
   This proves that there can only be one zero vector. Q.E.D.

This is a classic example of a proof built directly from axioms. Several other fundamental properties can be proven this way, such as the uniqueness of the additive inverse and the fact that multiplying any vector by the scalar 0 yields the zero vector.

You can find proofs for these additional properties in Theorem 4.2.6 (pages 246-247) of the Purdue PDF (LINK) we've been using. A quick look at these proofs will further solidify your understanding of axiomatic reasoning.

Conclusion

In this lesson, we have laid the formal groundwork for our study of linear algebra. Here are the key takeaways:

• A vector space is an abstract structure containing a set of "vectors" and a set of "scalars" that are related by two operations: vector addition and scalar multiplication.
• These operations must satisfy ten axioms that guarantee they behave in a consistent and predictable manner.
• The abstract definition is powerful because it allows us to apply the same linear algebra tools to diverse objects, including $n$-tuples (${\mathbb{R}}^n$), matrices, and functions.
• To verify if a set is a vector space, one must rigorously check that all ten axioms hold true for the defined operations.

This abstract foundation is what makes linear algebra such a versatile and universal tool.

Preview of the next lesson:
Now that we have defined what a vector space is, we can explore vector spaces that exist inside of other vector spaces. These are called subspaces. In the next lesson, we will define what a subspace is and learn a much more efficient method to check for them—a shortcut that saves us from re-verifying all ten axioms every time.