Vector Spaces and Subspaces: Definitions and Properties

Hello! Welcome to the first lesson of our course. Your goal is to build a deep, theoretical understanding of AI, and we'll start that journey today by establishing the mathematical bedrock upon which much of machine learning is built: linear algebra.

This initial module focuses on the foundational concepts of linear algebra. In this lesson, we will address the first learning outcome: to define vector spaces and subspaces, and verify their axiomatic properties. We'll explore the abstract rules that govern vectors and see how these rules provide a powerful framework for everything from data representation to optimization algorithms.

Let's begin by defining what a "vector space" truly is.

1. What is a Vector Space?

In your experience with projects like house price prediction, you've likely represented data as vectors—arrays of numbers. For example, a house might be represented by a vector [size, num_bedrooms, age]. The concept of a vector space formalizes the environment where such objects live. It's an abstract definition for a collection of objects (which we call vectors) that can be added together and scaled by numbers, in a way that is consistent and predictable.

This abstraction is powerful because any set that satisfies these rules—be it arrows in 3D space, matrices, polynomials, or even audio signals—is a vector space and inherits a rich set of properties and tools.

The definition of a vector space hinges on a set of rules, or axioms. The image below provides a quick overview of these properties, which we are about to study in detail.

To get a clear and intuitive introduction to these axioms, please watch the following video from the "Oxford Linear Algebra" series by Dr. Tom Crawford.

As you saw in the video, a vector space over a field of scalars $F$ (for our purposes, $F$ will typically be the real numbers, $\mathbb{R}$) is a non-empty set $V$ of objects called vectors, equipped with two operations:

1. Vector Addition: + : V × V → V (adding two vectors in $V$ produces another vector in $V$).
2. Scalar Multiplication: · : F × V → V (multiplying a vector in $V$ by a scalar in $F$ produces another vector in $V$).

These operations must satisfy the following eight axioms for all vectors $\mathbf{u},\mathbf{v},\mathbf{w}\in V$ and all scalars $a,b\in F$:

Axioms for Vector Addition:

1. Commutativity: $\mathbf{u}+\mathbf{v}=\mathbf{v}+\mathbf{u}$
2. Associativity: $(\mathbf{u}+\mathbf{v})+\mathbf{w}=\mathbf{u}+(\mathbf{v}+\mathbf{w})$
3. Additive Identity: There exists a zero vector $0\in V$ such that $\mathbf{v}+0=\mathbf{v}$ for all $\mathbf{v}\in V$.
4. Additive Inverse: For every $\mathbf{v}\in V$, there exists an inverse vector $-\mathbf{v}\in V$ such that $\mathbf{v}+(-\mathbf{v})=0$.

Axioms for Scalar Multiplication:
5. Multiplicative Identity: $1\cdot \mathbf{v}=\mathbf{v}$ (where 1 is the multiplicative identity in $F$).
6. Associativity: $(ab)\cdot \mathbf{v}=a\cdot (b\cdot \mathbf{v})$
7. Distributivity (vectors): $a\cdot (\mathbf{u}+\mathbf{v})=a\cdot \mathbf{u}+a\cdot \mathbf{v}$
8. Distributivity (scalars): $(a+b)\cdot \mathbf{v}=a\cdot \mathbf{v}+b\cdot \mathbf{v}$

Note that the first two properties required for the operations, closure under addition and scalar multiplication, are sometimes listed as axioms themselves, bringing the total count to ten.

2. Verifying the Axioms: An Example

Knowing the axioms is one thing; seeing them in action is another. To prove that a set is a vector space, one must rigorously verify that all the axioms hold true.

Continue with the same video to see a detailed verification that the familiar 3D coordinate system, ${\mathbb{R}}^3$, is indeed a vector space.

The process you just watched is the blueprint for verifying any potential vector space. Other common examples of vector spaces you'll encounter in machine learning include:

• The set of all $m\times n$ matrices, $M_{m\times n}(\mathbb{R})$.
• The set of all polynomials of degree less than or equal to $n$, $P_n(\mathbb{R})$.
• The set of all continuous functions $f:\mathbb{R}\to \mathbb{R}$.

Test your understanding!

Consider the vector space of all $2\times 2$ matrices with real entries, $M_{2\times 2}(\mathbb{R})$, with the standard matrix addition and scalar multiplication. What is the zero vector $0$ in this space? For a generic matrix $\mathbf{A}=\left(\begin{array}{cc}a& b\\ c& d\end{array}\right)$, what is its additive inverse $-\mathbf{A}$?

Show answer

• The zero vector is the $2\times 2$ zero matrix: $0=\left(\begin{array}{cc}0& 0\\ 0& 0\end{array}\right)$. Adding this matrix to any other $2\times 2$ matrix $\mathbf{A}$ leaves $\mathbf{A}$ unchanged.
• The additive inverse of $\mathbf{A}$ is $-\mathbf{A}=\left(\begin{array}{cc}-a& -b\\ -c& -d\end{array}\right)$. Adding $\mathbf{A}$ and $-\mathbf{A}$ results in the zero matrix.

3. Proving Properties from Axioms

As you requested a high degree of mathematical rigor, it's important to see how new mathematical truths can be built directly from a small set of axioms. For example, the axioms state that a zero vector and an additive inverse exist, but they don't state that they are unique. We can prove this.

This next segment of the video demonstrates how to construct a formal proof for the uniqueness of the additive identity (the zero vector). This is a classic example of axiomatic reasoning.

The technique used—assuming two such entities exist and then proving they must be identical—is a standard method for proving uniqueness in mathematics. Similar proofs can be constructed for the uniqueness of the additive inverse and other fundamental properties, all deriving from the same eight axioms.

4. Subspaces

Often, we are interested in a set of vectors that lies within a larger, known vector space. For example, in ${\mathbb{R}}^3$, consider the set of all vectors that lie on a plane passing through the origin. This set is a subspace.

A subspace $W$ of a vector space $V$ is a subset of $V$ that is itself a vector space, using the same addition and scalar multiplication operations from $V$.

Checking all eight axioms to verify a subspace would be repetitive, since most are inherited from the parent space $V$. Fortunately, there's a much more efficient method called the Subspace Test.

To learn about this test and see it applied, please read the following excerpt from a set of linear algebra lecture notes.

As the notes detail, a non-empty subset $W$ of a vector space $V$ is a subspace of $V$ if and only if the following three conditions hold:

1. Zero Vector: The zero vector of $V$ is in $W$ ($0\in W$).
2. Closure under Addition: For any two vectors ${\mathbf{w}}_1,{\mathbf{w}}_2\in W$, their sum ${\mathbf{w}}_1+{\mathbf{w}}_2$ is also in $W$.
3. Closure under Scalar Multiplication: For any vector $\mathbf{w}\in W$ and any scalar $c\in F$, the product $c\cdot \mathbf{w}$ is also in $W$.

If these three conditions are met, all other vector space axioms are automatically satisfied.

Test your understanding!

Let $V={\mathbb{R}}^3$. Use the three-part Subspace Test to determine if the set $W$ of all vectors $(x_1,x_2,x_3)$ such that $x_1+x_2+x_3=0$ is a subspace of ${\mathbb{R}}^3$.

Show answer

Let's check the three conditions:

1. Is the zero vector in W? The zero vector in ${\mathbb{R}}^3$ is $(0,0,0)$. Since $0+0+0=0$, the condition is met. Yes, $0\in W$.

2. Is W closed under addition? Let $\mathbf{u}=(u_1,u_2,u_3)$ and $\mathbf{v}=(v_1,v_2,v_3)$ be two arbitrary vectors in $W$. By definition of $W$, we know $u_1+u_2+u_3=0$ and $v_1+v_2+v_3=0$. Their sum is $\mathbf{u}+\mathbf{v}=(u_1+v_1,u_2+v_2,u_3+v_3)$. To check if this sum is in $W$, we add its components:
   $(u_1+v_1)+(u_2+v_2)+(u_3+v_3)=(u_1+u_2+u_3)+(v_1+v_2+v_3)=0+0=0$
   Since the sum of the components is 0, the vector $\mathbf{u}+\mathbf{v}$ is in $W$. Yes, $W$ is closed under addition.

3. Is W closed under scalar multiplication? Let $\mathbf{u}=(u_1,u_2,u_3)$ be a vector in $W$ and $c$ be any real scalar. We know $u_1+u_2+u_3=0$. The scalar multiple is $c\mathbf{u}=(c{u}_1,c{u}_2,c{u}_3)$. Let's check the sum of its components:
   $c{u}_1+c{u}_2+c{u}_3=c(u_1+u_2+u_3)=c(0)=0$
   Since the sum of the components is 0, the vector $c\mathbf{u}$ is in $W$. Yes, $W$ is closed under scalar multiplication.

Since all three conditions of the Subspace Test are satisfied, $W$ is a subspace of ${\mathbb{R}}^3$. Geometrically, this is the plane passing through the origin with the normal vector $(1,1,1)$.

---

Conclusion

In this lesson, we have laid the first crucial stone in your foundation of mathematical AI. We moved from the intuitive idea of a vector to the formal, axiomatic definition of a vector space.

Here are the key takeaways:

• A vector space is a set of objects (vectors) with two operations (addition and scalar multiplication) that satisfy eight specific axioms. This abstract framework unifies many different mathematical objects under one theory.
• To verify that a set forms a vector space, one must demonstrate that it is closed under the two operations and that all eight axioms hold true.
• The axioms are not just a checklist; they are foundational rules from which we can prove further properties, such as the uniqueness of the zero vector.
• A subspace is a vector space contained within another. We can efficiently check if a subset is a subspace using the three-part Subspace Test: verify the presence of the zero vector and closure under both addition and scalar multiplication.

In our next lesson, we will build upon this foundation. Now that we have defined the "space" where vectors exist, we will begin to explore the relationships between vectors within that space. We will cover the concepts of linear independence and the span of a set of vectors, which are essential for understanding how to construct vector spaces and represent data efficiently.