Subspaces and Spanning Sets

Hello! Welcome to your second lesson.

In our last session, we established the abstract definition of a vector space by exploring its ten axioms. We saw that this rigorous definition allows us to apply the tools of linear algebra to a wide range of objects, from the familiar arrows in $R^{n}$ to more abstract entities like functions, which are crucial in both deep learning and quantum mechanics.

Today, we will build directly on that foundation to explore vector spaces that exist inside of other vector spaces. This brings us to this lesson's learning outcome: to identify subspaces and construct the span of a set of vectors. We will start by learning how to "build" new spaces from a handful of vectors using an operation called span. Then, we'll define the properties of a subspace and see how these two powerful concepts are fundamentally linked.

These ideas are essential. In deep learning, understanding the span of a layer's weight vectors can give you insight into the expressive power of a model. In quantum mechanics, the set of possible states a system can occupy often forms a subspace of a larger state space.

1. Building with Vectors: Linear Combinations and Span

Let's begin with a fundamental question: if you are given a few vectors, what new vectors can you create using only the two operations we have in a vector space—vector addition and scalar multiplication?

The 3Blue1Brown video below provides a beautiful geometric intuition for this process. It introduces the concepts of a linear combination and the span of a set of vectors.

Linear combinations, span, and basis vectors | Chapter 2, Essence of linear algebra

This video will help you visualize what it means to combine vectors and see the entire set of possibilities, known as the 'span'.

Please watch from the start until 8:21. Pay close attention to how the concepts of linear combination and span are visualized in 2D and 3D. Notice what happens to the span when vectors are aligned versus when they point in different directions.

As the video illustrated, combining vectors this way generates geometric objects like lines and planes. Let's formalize these ideas.

A linear combination of a set of vectors ${v_{1}, v_{2}, \dots, v_{p}}$ is any vector $v$ that can be written in the form:
$v = c_{1} v_{1} + c_{2} v_{2} + \dots + c_{p} v_{p}$
where $c_{1}, c_{2}, \dots, c_{p}$ are scalars.
The span of the set of vectors ${v_{1}, v_{2}, \dots, v_{p}}$ , denoted as $Span {v_{1}, \dots, v_{p}}$ , is the set of all possible linear combinations of these vectors.

A common and important task is to determine if a specific vector lies within the span of a given set of vectors. This question is equivalent to asking: "Can we find the right scalars to build this vector?" This problem elegantly transforms into solving a system of linear equations.

Span and subspace

The following reading from a textbook by Roger Holmes demonstrates how to computationally check if a vector is in the span of another set of vectors.

Read the first two pages, up to and including 'Example 4.2.1'. Focus on how the question 'Is vector b in the span of x1 and x2?' is turned into a system of linear equations represented by an augmented matrix. This should feel familiar from your previous experience with matrices.

As you saw, asking if $b$ is in $Span {v_{1}, \dots, v_{p}}$ is the same as asking if the vector equation $c_{1} v_{1} + \dots + c_{p} v_{p} = b$ has a solution for the scalars $c_{i}$ . This is precisely the kind of problem you can solve by setting up an augmented matrix $[v_{1} \dots v_{p} ∣ b]$ and checking for consistency.

Test your understanding!

Let $v_{1} = 10 - 2$ and $v_{2} = - 3 18$ . Is the vector $w = - 1 14$ in $Span {v_{1}, v_{2}}$ ?

Show answer

We need to check if there are scalars $c_{1}$ and $c_{2}$ such that $c_{1} v_{1} + c_{2} v_{2} = w$ . This gives the system:

c_{1} - 3 c_{2} c_{2} - 2 c_{1} + 8 c_{2} = - 1 = 1 = 4

From the second equation, we immediately get $c_{2} = 1$ . Substituting this into the first equation gives $c_{1} - 3 (1) = - 1$ , which means $c_{1} = 2$ .

Now we must check if these values work for the third equation: $- 2 (2) + 8 (1) = - 4 + 8 = 4$ . It works!

Since we found a solution ( $c_{1} = 2, c_{2} = 1$ ), the vector $w$ is in the span. Specifically, $w = 2 v_{1} + 1 v_{2}$ .

2. Subspaces: Vector Spaces Within Vector Spaces

In the last lesson, we saw that a set must satisfy ten axioms to be a vector space. What if we have a subset of a known vector space, like $R^{n}$ ? Do we need to check all ten axioms again to see if this subset is also a vector space in its own right?

Fortunately, no. If a subset inherits its operations from the larger "parent" space, most of the axioms (like commutativity and associativity) are automatically satisfied. We only need to verify a much smaller, more manageable list of properties. This leads to the definition of a subspace.

The following video provides a clear definition of a subspace and excellent visual examples of what qualifies and what doesn't.

Subspaces are the Natural Subsets of Linear Algebra | Definition + First Examples

Dr. Trefor Bazett explains the three simple conditions a subset must meet to be considered a subspace.

Watch from 0:50 to 6:24. Focus on the 'Subspace Test', which consists of three crucial properties. Observe the geometric intuition for why a line not passing through the origin fails to be a subspace.

To summarize, a subset $H$ of a vector space $V$ is a subspace if it satisfies these three conditions:

Contains the Zero Vector: The zero vector $0$ of $V$ must be in $H$ .
Closed under Vector Addition: For any two vectors $u$ and $v$ in $H$ , their sum $u + v$ must also be in $H$ .
Closed under Scalar Multiplication: For any vector $u$ in $H$ and any scalar $c$ , the vector $c u$ must also be in $H$ .

These three conditions guarantee that the subset $H$ is a self-contained vector space. You can perform linear algebra operations on vectors within $H$ and you will never "leave" $H$ .

3D Subspace Illustration — A plane passing through the origin in $\mathbb{R}^3$ is a classic example of a subspace. If you take any two vectors in the plane (like V1 and V2), their sum will also lie in the plane. Scaling any vector in the plane will keep it in the plane. And, of course, the origin is in the plane.

Test your understanding!

Consider the following two subsets of $R^{3}$ . Which one is a subspace? Explain why the other is not.

$H_{1}$ : The set of all vectors of the form $a b a + b$ , where $a$ and $b$ are any real numbers.
$H_{2}$ : The set of all vectors of the form $a b 1$ , where $a$ and $b$ are any real numbers.

Show answer

$H_{1}$ is a subspace.
- Zero Vector: If $a = 0$ and $b = 0$ , we get the zero vector $000$ . So it's in $H_{1}$ .
- Closure under Addition: Let $u = a_{1} b_{1} a_{1} + b_{1}$ and $v = a_{2} b_{2} a_{2} + b_{2}$ . Their sum is $u + v = a_{1} + a_{2} b_{1} + b_{2} (a_{1} + a_{2}) + (b_{1} + b_{2})$ . This has the correct form to be in $H_{1}$ , with $a = a_{1} + a_{2}$ and $b = b_{1} + b_{2}$ .
- Closure under Scalar Multiplication: Let $c$ be a scalar. Then $c u = c a_{1} c b_{1} c (a_{1} + b_{1}) = c a_{1} c b_{1} c a_{1} + c b_{1}$ . This also has the correct form, with $a = c a_{1}$ and $b = c b_{1}$ .
$H_{2}$ is not a subspace. It fails the first condition. The zero vector $000$ cannot be created, because the third component must always be 1. It also fails the other two conditions, but failing one is enough.

3. The Grand Connection: Spans are Subspaces

Now we can connect our two main ideas. We have a tool for building sets of vectors (span) and a definition for a "vector space within a vector space" (subspace). The fundamental link is this: the span of any set of vectors is always a subspace.

This is an incredibly useful result. It means that whenever we construct a set of vectors using span, we are guaranteed to have a valid subspace where we can do linear algebra.

Let's look at the formal statement and proof.

Subspaces

This resource formally states and proves the crucial theorem that the span of a set of vectors is always a subspace. Understanding this proof is an excellent way to solidify the definitions of both span and subspace.

Find the subsection titled 'Common Types of Subspaces' and read the theorem 'Spans are Subspaces and Subspaces are Spans' along with its proof. The proof simply involves checking that the definition of a span satisfies the three subspace conditions.

The proof confirms what our intuition suggests:

The zero vector is in the span (just choose all scalars to be 0).
The sum of two linear combinations is just another linear combination.
A scalar multiple of a linear combination is also just another linear combination.

This connection gives us our primary method for describing subspaces. Two of the most important subspaces in linear algebra, especially for applications in data science and machine learning, are defined using spans of vectors from a matrix.

Column Space (Col A): The column space of a matrix $A$ is the span of its column vectors. By definition, it is a subspace. It represents the set of all possible outputs $b$ for the equation $A x = b$ .
Null Space (Nul A): The null space of a matrix $A$ is the set of all vectors $x$ that solve the homogeneous equation $A x = 0$ . It can be shown that the null space is also a subspace. While it is defined as a solution set, we can always find a set of vectors that span it.

The same resource you just read explains how to find a spanning set for the null space. This is a concrete example of "constructing the span" for a subspace that isn't initially defined as one.

Subspaces

Now let's see these ideas applied to matrices. This section defines the column space and null space, and importantly, gives you a recipe for constructing the span of a null space.

Read the rest of the 'Common Types of Subspaces' section, starting from the definition of Column Space and Null Space. Pay special attention to the 'Recipe: Compute a spanning set for a null space'. This involves finding the parametric vector form of the solution, a key computational skill.

Conclusion

In this lesson, we explored two cornerstone concepts of linear algebra. We moved from the abstract definition of a vector space to the tangible ways we can construct and identify them in practice.

Key Takeaways:

A linear combination is a weighted sum of vectors, $c_{1} v_{1} + \dots + c_{p} v_{p}$ .
The span of a set of vectors is the collection of all possible linear combinations. Geometrically, it's the line, plane, or higher-dimensional object "painted" by the vectors.
A subspace is a special subset of a vector space that is closed under addition and scalar multiplication and contains the zero vector. It's a vector space in its own right.
The most powerful link between these ideas is that the span of any set of vectors is always a subspace. This gives us a universal tool for building subspaces.
The column space and null space of a matrix are two fundamental examples of subspaces that are central to many applications.

Preview of Next Lesson:

We saw that the span of two non-collinear vectors in $R^{3}$ is a plane. What happens if we include a third vector that is already in that plane? The span doesn't change; the third vector is "redundant". This idea of redundancy is formalized by the concept of linear dependence. In our next lesson, we will investigate linear dependence and independence, which will allow us to find the most efficient set of vectors needed to span a subspace.

Recommended resources

1. Building with Vectors: Linear Combinations and Span

2. Subspaces: Vector Spaces Within Vector Spaces

3. The Grand Connection: Spans are Subspaces

Conclusion