Basis and Dimension of Vector Spaces

Welcome to our next lesson. In our previous sessions, we explored two powerful, complementary ideas: linear independence, which identifies sets of vectors without any redundancy, and span, which describes all the vectors you can possibly create from a given set. You've seen that a set can span a space but be redundant, or it can be non-redundant but fail to span the entire space.

This lesson brings these two concepts together to define what is arguably the most important idea in linear algebra: the basis of a vector space. A basis is a set of vectors that hits the "sweet spot"—it is both linearly independent and spans the space. It is the most efficient possible toolkit for building and describing a vector space. We will define what a basis is, and then use it to define the dimension of a space, a concept that gives a precise, numerical meaning to our intuitive notion of "size."

Understanding basis and dimension is crucial for your goal of tackling mechanistic interpretability. The high-dimensional spaces where neural network weights and activations live are vector spaces. A basis provides a coordinate system for these spaces, and its dimension tells us the number of independent directions or features the model can represent.

The Goldilocks Set: Defining a Basis

A basis is a set of vectors that is "just right." It contains no redundant vectors, yet it's powerful enough to build any other vector in the space. Let's formalize this.

Vector spaces and bases

These notes from Columbia University provide a concise and rigorous definition of a basis.

Please read the first paragraph of section 3, titled "Bases". It consists of two short, critical conditions. Pay close attention to how it explicitly combines the concepts of linear independence and span. Find the text starting from the definition of a basis.

As you just read, a set of vectors $B$ is a basis for a vector space $V$ if:

$B$ is linearly independent.
$B$ spans $V$ (i.e., $span (B) = V$ ).

The most intuitive example is the standard basis for $R^{n}$ . For $R^{3}$ , the standard basis is the set ${e_{1}, e_{2}, e_{3}}$ , where:

e_{1} = 100, e_{2} = 010, e_{3} = 001

These vectors are clearly linearly independent (you can't make one from the others), and they span all of $R^{3}$ (any vector $[x, y, z]^{T}$ can be written as $x e_{1} + y e_{2} + z e_{3}$ ).

A key consequence of these two conditions is that any vector in the space can be written as a unique linear combination of the basis vectors. The spanning property guarantees at least one way, and the linear independence property guarantees at most one way. This is why a basis is so powerful: it provides a stable and unambiguous coordinate system for the vector space.

Dimension: A Universal Constant for a Space

So, $R^{3}$ has a basis with three vectors. What about other bases for $R^{3}$ ? Could we find one with two vectors? Or four? The intuition from our 3D world suggests "no," and linear algebra proves this intuition correct. This leads to one of the most fundamental theorems of the subject.

The following video from MIT OpenCourseWare, featuring Professor Gilbert Strang, gives an excellent intuitive explanation for why the size of a basis must be a fixed number for any given space.

Independence, Basis, and Dimension

This video, "Independence, Basis, and Dimension," ties all these concepts together.

Please watch from this section, where Prof. Strang formally defines a basis. Then, continue from here as he introduces the concept of dimension and argues why it's a constant number. Finally, watch his illustration of a 2D subspace (a plane) in 3D space from this part.

Professor Strang's argument is that for a space like $R^{3}$ , you can't span it with fewer than 3 vectors, and any set of more than 3 vectors must be linearly dependent. This special number—the size of any basis for a space—is its dimension.

Let's see the formal statement of this idea.

Vector spaces and bases

These notes formalize the intuition from the video.

Read "Theorem 5 (Size-of-Basis Theorem)" and the short definition that follows it. The proof uses the "Exchange Theorem," which we haven't covered, but the logic is what's important: if you have two bases of different sizes, you can show a contradiction. The key result is in the definition of dimension that immediately follows the theorem.

The dimension of a vector space $V$ , written $dim (V)$ , is the number of vectors in any basis for $V$ .

This is a profound statement. It means that no matter how you choose to construct a coordinate system (a basis) for a space, the number of vectors you'll need is always the same.

A line through the origin is a subspace of dimension 1.
A plane through the origin is a subspace of dimension 2.
$R^{n}$ has dimension $n$ .

The image below shows two vectors that span a plane in $R^{3}$ . Since the two vectors are not collinear, they are linearly independent. Therefore, they form a basis for the plane they span. This plane is a 2-dimensional subspace of the 3-dimensional space $R^{3}$ .

Two linearly independent vectors in $\mathbb{R}^3$ form a basis for a 2-dimensional subspace (a plane). The dimension of this subspace is 2.

How to Find a Basis and Dimension

Now for the practical part. Given a subspace, how do we find a basis for it and determine its dimension? The method depends on how the subspace is described.

Case 1: Subspace defined by a spanning set

Let's say we have a subspace $W$ that is the span of a set of vectors ${v_{1}, v_{2}, \dots, v_{k}}$ . This set spans $W$ by definition, but it might be linearly dependent. To find a basis, we just need to weed out the redundant vectors.

The most systematic way to do this is:

Create a matrix $A$ with the vectors as its columns. The span of the vectors is the column space of $A$ .
Row reduce the matrix $A$ to its reduced row echelon form (RREF).
Identify the columns in the RREF that contain pivots.
The corresponding columns in the original matrix $A$ form a basis for the subspace.
The number of vectors in this basis (which is equal to the number of pivots) is the dimension of the subspace.

The following video provides a clear, step-by-step walkthrough of this exact procedure.

Linear Algebra 4.5 Basis and Dimension

This video by Kimberly Brehm demonstrates how to find the dimension of a subspace.

Watch the segment from this timestamp. The instructor works through a concrete example where a subspace of \mathbb{R}^3 is defined by four spanning vectors. She shows how to extract the vectors, set up a matrix, row reduce it, and identify the basis vectors from the pivot columns.

Case 2: Subspace defined by an equation

Sometimes a subspace is defined by a condition or an equation, not an explicit set of vectors. For example, consider the subspace $S$ of $R^{3}$ consisting of all vectors $x y z$ such that $x - 2 y + z = 0$ .

To find a basis here, we need to describe what a generic vector in this space looks like.

The equation $x - 2 y + z = 0$ has a leading variable ( $x$ ) and free variables ( $y$ and $z$ ).
Express the leading variable in terms of the free variables: $x = 2 y - z$ .
Write a generic vector from the subspace using only the free variables: $x y z = 2 y - z y z$
Decompose this vector into parts, one for each free variable: $2 y - z y z = 2 y y 0 + - z 0 z = y 210 + z - 1 01$

This shows that any vector in $S$ is a linear combination of $210$ and $- 1 01$ . These two vectors are linearly independent, so they form a basis for $S$ . The dimension of $S$ is 2, which makes sense—it's a plane in $R^{3}$ .

This technique is explained well in the following resource.

4.3. Vector Spaces, Basis, and Dimension - EECS 245 Course Notes

These course notes provide several excellent examples. The one on finding a basis for a subspace defined by a constraint is particularly relevant.

Read the section titled "Basis and Dimension". Pay special attention to the subsection "Finding a Basis for a Subspace" and the final worked example, "Example: Equal Second and Fourth Components," which uses this parameterization method.

Conclusion

In this lesson, we have unified the concepts of linear independence and span to define a basis and, with it, the dimension of a vector space.

Here are the essential takeaways:

A basis is a linearly independent set of vectors that spans a vector space. It is the minimal set required to generate the space.
Representation in terms of a basis is unique.
The dimension of a vector space is the number of vectors in any of its bases. This number is an intrinsic property of the space itself.
We can find a basis for a subspace by either pruning a spanning set using pivot columns or by parameterizing the solution to a defining equation.

We have now completed our foundational tour of vector spaces. We have the language to describe these fundamental mathematical structures precisely. In the next module, we will turn our attention to matrices, not just as containers for vectors, but as operators that perform linear transformations. You will see that the concepts of basis and dimension are critical for understanding what these transformations do, through the analysis of key subspaces associated with every matrix, such as its column space and null space.

Recommended resources