Linear Independence and Dependence of Vectors

Geometric Intuition: Redundancy in Vectors

Before diving into formal definitions, let's build a strong visual intuition. A set of vectors is linearly dependent if at least one vector is "redundant"—it doesn't add a new direction and already lies within the span of the other vectors.

• In ${\mathbb{R}}^2$, two vectors are linearly dependent if they are collinear (lie on the same line). One is just a scaled version of the other.
• In ${\mathbb{R}}^3$, three vectors are linearly dependent if they are coplanar (lie on the same plane). One of them can be written as a linear combination of the other two.

The following video provides an excellent visual demonstration of these ideas.

This image provides a static summary of the geometric interpretation of linear dependence and independence.

The key takeaway is that a set of vectors is linearly dependent if one vector is contained within the span of the others. This makes that vector redundant in terms of the total span of the set.

Formal Definitions of Linear Independence and Dependence

Now let's translate this intuition into a rigorous mathematical definition. The standard way to define linear independence is by considering a linear combination that equals the zero vector.

Consider a set of vectors $S=\{{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_p\}$. We are interested in the vector equation:

Notice there is always one obvious solution: setting all scalars $c_i$ to zero. This is called the trivial solution. The crucial question is whether any other solutions exist.

The following reading formally defines linear independence based on the answer to this question.

To summarize the formal definitions:

• A set of vectors $S=\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_p\}$ is linearly independent if the only solution to the equation $c_1{\mathbf{v}}_1+\cdots +c_p{\mathbf{v}}_p=0$ is the trivial solution, where $c_1=c_2=\cdots =c_p=0$.

• The set is linearly dependent if there exists a non-trivial solution, meaning at least one scalar $c_i$ is non-zero. Such a non-trivial solution is sometimes called a linear dependence relation.

Connecting the Definitions: A Proof

We now have two ways of thinking about linear dependence:

1. Geometric Intuition: At least one vector is in the span of the others.
2. Formal Definition: There's a non-trivial linear combination that equals the zero vector.

For a proof-based understanding, it is essential to show that these two ideas are equivalent.

Theorem: A set of vectors $S=\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_p\}$ (with $p\ge 2$) is linearly dependent if and only if at least one of the vectors in $S$ is a linear combination of the others.

The following resource provides a clear proof of this theorem. Given your interest in proofs, I encourage you to work through it carefully.

Let's quickly outline the logic of the proof:

• "If" part: Assume one vector (say ${\mathbf{v}}_p$) is a linear combination of the others. We can rearrange the equation to show a non-trivial linear combination that equals zero, proving dependence.
• "Only if" part: Assume the set is linearly dependent. This means a non-trivial combination equals zero. We can pick a term with a non-zero coefficient and solve for that vector in terms of the others.

This equivalence is powerful. It solidifies our intuition and gives us flexibility when constructing proofs.

An Algorithm for Testing Linear Independence

The formal definition provides a direct computational method to test for linear independence. The problem of finding whether a non-trivial solution exists for $c_1{\mathbf{v}}_1+\cdots +c_p{\mathbf{v}}_p=0$ is equivalent to solving a homogeneous system of linear equations.

Given a set of vectors $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_p\}$ in ${\mathbb{R}}^n$, we can construct a matrix $A$ where these vectors are the columns: $A=[{\mathbf{v}}_1{\mathbf{v}}_2\cdots {\mathbf{v}}_p]$. The vector equation becomes:

The set of vectors is linearly independent if and only if this system has only the trivial solution $\mathbf{x}=0$.

This transforms the abstract question of independence into a concrete algorithmic problem.

The algorithm is as follows:

1. Construct a matrix $A$ whose columns are the vectors in the set.
2. Use Gaussian elimination to reduce $A$ to its row echelon form.
3. Examine the pivot positions.
   • If every column has a pivot, there are no free variables. The only solution to $A\mathbf{x}=0$ is the trivial one ($\mathbf{x}=0$). The vectors are linearly independent.
   • If there is at least one non-pivot column, there is at least one free variable. This guarantees the existence of non-trivial solutions. The vectors are linearly dependent.

Let's see this in action with a worked example.

Two Useful Theorems

Finally, there are two straightforward but powerful theorems that can often serve as shortcuts to determine linear dependence without any computation.

1. Theorem: A set $S=\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_p\}$ in ${\mathbb{R}}^n$ that contains the zero vector is linearly dependent.

   • Proof Idea: If ${\mathbf{v}}_i=0$, then we can form the non-trivial combination $0{\mathbf{v}}_1+\cdots +1{\mathbf{v}}_i+\cdots +0{\mathbf{v}}_p=0$.

2. Theorem: If a set of vectors in ${\mathbb{R}}^n$ contains more vectors than the dimension of the space (i.e., $p>n$), then the set is linearly dependent.

   • Proof Idea: The matrix $A$ will be $n\times p$. If $p>n$, there are more columns (variables) than rows (equations). This guarantees there must be at least one free variable, which implies a non-trivial solution to $A\mathbf{x}=0$. For example, any three vectors in ${\mathbb{R}}^2$ must be linearly dependent.

Conclusion

In this lesson, we formalized the crucial concept of linear independence. This idea of a "minimal" or "non-redundant" set of vectors is fundamental to everything that follows.

Key Takeaways:

• Geometric View: Linearly dependent vectors are redundant. One lies in the span of the others (e.g., they are collinear or coplanar).
• Formal Definition: A set of vectors is linearly independent if the only linear combination that sums to the zero vector is the trivial one (all coefficients are zero).
• Equivalence: These two notions are provably equivalent.
• Computational Test: To test a set of vectors for linear independence, form a matrix with them as columns and row-reduce. The set is independent if and only if every column is a pivot column.
• Key Theorems: Any set containing the zero vector is dependent. Any set of $p$ vectors in ${\mathbb{R}}^n$ with $p>n$ is dependent.

Preview of the next lesson:
We have now studied two essential properties of a set of vectors:

1. Does it span the space? (Topic 3)
2. Is it linearly independent? (Topic 4)

When a set of vectors satisfies both of these properties, it is special. Such a set is called a basis for the vector space. A basis is an efficient spanning set—it's just large enough to span the entire space without any redundant vectors. We will explore this powerful concept in our next lesson.