Solving Linear Systems and Understanding Feature Redundancy

A linear system in matrix form

A linear equation contains unknowns only to the first power and does not multiply unknowns together. For example,

is linear; $x_1{x}_2=3$ and $x_1^2+x_2=3$ are not.

A system collects several linear constraints. Consider

We can write it compactly as

where

The roles are worth keeping distinct:

• Each row of $A$ encodes one equation or constraint.
• Each column of $A$ gives the coefficients of one unknown across all equations.
• $\mathbf{x}$ is the vector we want to find.
• $\mathbf{b}$ contains the known right-hand-side values.

In a predictive-model setting, $\mathbf{x}$ can represent unknown model coefficients. A system such as $A\mathbf{x}=\mathbf{b}$ asks: which coefficient vector exactly satisfies these constraints?

For now, treat an inverse as a useful idea rather than a computational method. If a square matrix $A$ has an inverse, then the system has one unique solution:

In practice, numerical software solves the system directly rather than explicitly calculating $A^{-1}$.

Solving a small system by Gaussian elimination

The practical hand method is Gaussian elimination. First, replace the system with its augmented matrix: write the coefficient matrix and append the right-hand-side vector after a vertical divider.

The divider has no mathematical effect; it is a bookkeeping device separating coefficients from constants.

Gaussian elimination uses row operations that preserve the system’s solution set:

1. Swap two rows.
2. Multiply a row by a nonzero constant.
3. Add a multiple of one row to another row.

The aim is to create zeros below successive leading entries, producing an upper-triangular form that can be solved from the bottom upward.

Now solve our system. Use the first row to eliminate the $x$-terms below it:

This gives

It is convenient to swap rows 2 and 3, putting the leading $1$ in the second row:

Next, eliminate the $-3$ below the second-row leading entry:

This corresponds to the simpler system

Now work upward.

From the last equation,

Substitute this into the second equation:

Finally, substitute $y=2$ and $z=3$ into the first equation:

Therefore,

A fast verification is to multiply the original matrix by the proposed solution:

The product matches $\mathbf{b}$, so the solution is correct.

Three possible outcomes

Elimination does more than calculate values. It reveals the structure of the system. A linear system can have only one of three outcomes:

The distinction between the last two outcomes matters for singular matrices.

Consider the coefficient matrix

Its second row is exactly twice its first row. Thus it does not provide two independent constraints.

First choose

The system is

Eliminate the second row:

The second equation adds no information. We have only

Let $x_2=t$, where $t$ can be any real number. Then

So there are infinitely many solutions:

Now keep the same $A$, but use a different target:

Elimination gives

The last row says

This is impossible, so the system has no solution.

A square matrix is singular when it has no inverse. The example matrix $A$ is singular because it collapses one independent direction: its two rows and its two columns contain redundant information.

The crucial conclusion is:

A singular coefficient matrix does not automatically mean “no solution.” It means there cannot be exactly one solution for every possible $\mathbf{b}$. Depending on $\mathbf{b}$, the system has either infinitely many solutions or no solution.

Redundant features create non-unique coefficients

Now connect singularity to a regression design matrix.

Suppose a data pipeline creates two numerical features for every trial workspace:

• $d_i$: active days during the first week;
• $p_i$: an “activity points” feature defined as exactly twice active days.

By construction,

A design matrix with an intercept and these two features might look like

The third column is exactly twice the second. It contributes no new independent information.

If a linear model is

then, since $p_i=2d_i$,

Only the combined quantity

affects predictions. There is no way for the data alone to decide how much of that combined effect belongs to ${\beta }_d$ and how much belongs to ${\beta }_p$.

More formally, for any number $t$, changing coefficients as follows leaves every prediction unchanged:

Indeed,

So the model has infinitely many coefficient vectors that make identical predictions. The coefficient-estimation problem is not uniquely solvable.

This is exact multicollinearity or feature redundancy. Common sources include:

• a duplicated feature column;
• one feature coded as an exact constant multiple of another;
• a total and all of its mutually exclusive parts included together;
• an intercept plus one indicator column for every category of a categorical variable.

For example, if each workspace belongs to exactly one plan type, then

If the model already has an intercept column of ones, including all three plan indicators creates an exact linear dependency. The usual solution is to keep the intercept and omit one category indicator as the reference category.

Do not confuse exact multicollinearity with two features being merely highly correlated:

• Exact redundancy: one column is an exact linear combination of others. The relevant matrix is structurally singular.
• High but imperfect correlation: columns are not identical combinations, so a unique solution may exist, but coefficient estimates can be unstable.

Neither condition is causal confounding. These are algebraic properties of the features you supplied to a model; causal confounding concerns how treatment, outcome, and other variables are generated and related.

Solving in NumPy

For a small square, nonsingular system, use np.linalg.solve. It solves $A\mathbf{x}=\mathbf{b}$ directly.

import numpy as np

A = np.array([
    [1.0, 1.0, 1.0],
    [2.0, -1.0, 1.0],
    [1.0, 2.0, -1.0],
])

b = np.array([6.0, 3.0, 2.0])

x = np.linalg.solve(A, b)

print(x)                    # [1. 2. 3.]
print(np.allclose(A @ x, b))  # True

Avoid the less direct pattern below:

# Avoid this as a default approach
x = np.linalg.inv(A) @ b

Computing an inverse is unnecessary when the task is to solve one system, and it is generally less numerically appropriate.

For an exactly singular matrix, np.linalg.solve raises a LinAlgError because there is no unique solution:

A_singular = np.array([
    [1.0, 1.0],
    [2.0, 2.0],
])

b = np.array([3.0, 6.0])

np.linalg.solve(A_singular, b)

That error should trigger investigation of the matrix construction and feature definitions. It should not automatically be “fixed” by selecting an arbitrary coefficient solution: the underlying question is whether the model specification contains redundant information.

Real datasets also bring floating-point rounding and near-redundant features, which are more subtle than the exact examples here. The next lesson introduces rank as the systematic language for diagnosing exact linear dependence.

Key takeaways

A system of linear equations can be written as

Gaussian elimination transforms its augmented matrix using solution-preserving row operations, then uses back-substitution to recover unknowns.

A system has exactly one outcome:

• one unique solution;
• infinitely many solutions;
• no solution.

For a square matrix, singular means “not invertible.” Singularity reflects lost independent information. A singular system can have infinitely many solutions when its constraints are compatible, or no solution when they are contradictory.

In a predictive model, exact redundant feature columns mean that multiple coefficient vectors produce exactly the same predictions. This makes separate coefficients non-identifiable, even when predictions themselves may be unchanged.

Next, you will use matrix rank to detect these missing independent directions directly in a design matrix.