Linear Independence of Vectors

Linear Combinations and Span: The Building Blocks

The two fundamental operations we have are scaling a vector and adding vectors together. When we do both—scale several vectors and add them up—we create a linear combination. For a set of vectors $\{{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_n\}$, a linear combination takes the form:

where $c_1,c_2,\dots ,c_n$ are scalars. The set of all possible vectors you can create by varying these scalars is called the span of the vectors.

The 3Blue1Brown YouTube channel provides one of the best visual intuitions for these concepts.

As the video illustrated, the span of a set of vectors is the space they can "generate". If you have two vectors in 3D space that don't point along the same line, their span is a plane cutting through the origin. If you add a third vector that lies on that plane, you haven't gained anything; you're still confined to the same plane. But if that third vector points off the plane, you can now reach any point in 3D space. This idea of redundancy is the key to understanding linear dependence.

Defining Linear Independence

We can now define linear independence in two ways: one intuitive and one formal.

1. The Intuitive Definition: Redundancy and Span

A set of vectors is linearly dependent if at least one vector is redundant, meaning it's already in the span of the others. You could remove it without shrinking the span.

Conversely, a set of vectors is linearly independent if every vector is essential. Each one adds a new direction or dimension to the span that wasn't accessible before.

This intuitive definition is powerful but not always easy to test directly. How can we check if one vector is a combination of the others in a systematic way? This leads us to the formal, algebraic definition.

2. The Formal Definition: The Zero Vector Test

Let's rephrase the idea of dependence. If a vector ${\mathbf{v}}_k$ is a linear combination of others in the set $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_n\}$, we can write:

By simply rearranging this equation, we get:

This is a linear combination of all the vectors that equals the zero vector, where at least one coefficient (the $-1$ in front of ${\mathbf{v}}_k$) is not zero. This gives us a universal test.

A set of vectors $S=\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_n\}$ is linearly independent if and only if the only way to make their linear combination equal to the zero vector is by using all zero coefficients (the "trivial solution").

If there's any other solution where at least one $c_i$ is non-zero, the set is linearly dependent.

The following article provides a rigorous statement of this principle.

The image below gives a simple 2D visualization of this principle.

On the left, two vectors are linearly independent; only the trivial combination $0{\mathbf{v}}_1+0{\mathbf{v}}_2$ results in the zero vector. On the right, two vectors are linearly dependent (collinear); a non-trivial combination, such as $1{\mathbf{v}}_1+1{\mathbf{v}}_2$, can produce the zero vector.

How to Check for Linear Independence

This formal definition provides a direct computational method. The vector equation $c_1{\mathbf{v}}_1+\cdots +c_n{\mathbf{v}}_n=0$ is simply a system of linear equations.

Given a set of vectors in ${\mathbb{R}}^m$, $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_n\}$, we can perform the following steps:

1. Create a matrix $A$ where the columns are your vectors: $A=[{\mathbf{v}}_1|{\mathbf{v}}_2|\dots |{\mathbf{v}}_n]$.
2. The vector equation becomes the matrix equation $A\mathbf{c}=0$, where $\mathbf{c}$ is the column vector of unknown coefficients $[c_1,\dots ,c_n{]}^T$.
3. Solve this system for $\mathbf{c}$. The key question is whether only the trivial solution $\mathbf{c}=0$ exists.
4. From your knowledge of systems of equations, you know that this homogeneous system has only the trivial solution if and only if the matrix $A$, when row-reduced, has a pivot position in every column. If there is any column without a pivot (which corresponds to a free variable), then non-trivial solutions exist, and the vectors are linearly dependent.

This method transforms the abstract question of linear independence into a concrete algorithmic procedure.

Practice Problem

Let's apply this. Determine if the following set of vectors in ${\mathbb{R}}^3$ is linearly independent.

Your task:

1. Set up the matrix $A$.
2. Set up the equation $A\mathbf{c}=0$.
3. Without doing a full row-reduction, can you spot a relationship between the vectors? (Hint: Look at the columns).
4. What does this relationship tell you about the solution for $\mathbf{c}$? Are the vectors linearly independent or dependent?

Solution:

1. The matrix is $A=\left[\begin{array}{ccc}1& 0& 1\\ 0& 1& 1\\ 1& 1& 2\end{array}\right]$.
2. The equation is $\left[\begin{array}{ccc}1& 0& 1\\ 0& 1& 1\\ 1& 1& 2\end{array}\right]\left[\begin{array}{c}{c}_1\\ c_2\\ c_3\end{array}\right]=\left[\begin{array}{c}0\\ 0\\ 0\end{array}\right]$.
3. By inspection, we can see that the third column is the sum of the first two columns: ${\mathbf{v}}_3={\mathbf{v}}_1+{\mathbf{v}}_2$.
4. This means ${\mathbf{v}}_1+{\mathbf{v}}_2-{\mathbf{v}}_3=0$. This corresponds to a non-trivial solution where $c_1=1,c_2=1,c_3=-1$. Since a non-trivial solution exists, the set of vectors is linearly dependent. The vector ${\mathbf{v}}_3$ is redundant.

Key Properties and Rules of Thumb

Finally, here are a few useful properties that can serve as shortcuts:

• A set of two vectors is linearly dependent if and only if they are collinear (one is a scalar multiple of the other).
• Any set of vectors containing the zero vector is automatically linearly dependent. (Why? If ${\mathbf{v}}_1=0$, then $1{\mathbf{v}}_1+0{\mathbf{v}}_2+\cdots +0{\mathbf{v}}_n=0$ is a non-trivial solution.)
• If you have more vectors than dimensions (e.g., 3 vectors in ${\mathbb{R}}^2$, or 4 in ${\mathbb{R}}^3$), the set is always linearly dependent. A "wide" matrix $A$ (more columns than rows) cannot have a pivot in every column, guaranteeing the existence of free variables.

Conclusion

In this lesson, we established the crucial concept of linear independence as a measure of non-redundancy in a set of vectors.

Here are the main takeaways:

• Intuitive Meaning: A set of vectors is linearly independent if each vector adds a new dimension to the span. It's dependent if at least one vector is redundant (i.e., lies in the span of the others).
• Formal Test: A set $\{{\mathbf{v}}_i\}$ is linearly independent if the equation $\sum c_i{\mathbf{v}}_i=0$ is only true when all $c_i=0$.
• Computational Method: To test a set of vectors, form a matrix with the vectors as columns and row-reduce it. The vectors are linearly independent if and only if every column has a pivot.

We briefly introduced the idea of span today to define independence. In our next lesson, we will explore span in its own right, examining the geometry of the subspaces that vectors can generate. Afterward, we will combine the concepts of span and linear independence to define a basis—the most efficient possible set of building blocks for a vector space.