Diagnosing Exact Multicollinearity Using Matrix Rank

Rank: counting independent information

A matrix’s rank is the number of linearly independent columns (equivalently, the number of linearly independent rows). For a design matrix $X$, focus on the columns, because each column corresponds to an intercept or feature.

A column is linearly dependent if it can be reconstructed exactly from other columns using weighted sums. For example, if

then the third feature contains no information beyond the second. It is a duplicate direction, merely rescaled.

Consider

Interpret the columns as:

• column 1: an intercept;
• column 2: active days in the first week;
• column 3: activity points, defined as exactly twice active days.

The third column is exactly twice the second:

There are three columns, but only two independent columns: the intercept and active days. Therefore,

The matrix is rank deficient because its rank is less than its number of columns.

A useful operational rule is:

where $p$ is the number of columns in $X$. When $X$ has full column rank, no feature column is an exact linear combination of the others.

The video’s left-to-right procedure is useful for intuition, but in practical work you will normally let numerical linear-algebra routines determine rank. Before doing that, it helps to see the same conclusion through elimination.

Rank from row reduction

Return to the activity-feature design matrix:

Use the first row to eliminate the intercept entries below it:

Then use the second row to eliminate the entries beneath its leading $1$:

There are two nonzero rows, or equivalently two pivot positions. Thus,

This procedure looks at rows, yet it detects dependence among columns because a fundamental result of linear algebra says row rank and column rank are always equal.

For a design matrix with many more observations than features, the practical interpretation is straightforward:

The gap

is called the rank deficiency. It counts the number of independent coefficient directions the data cannot distinguish.

Why rank deficiency means non-unique coefficients

Suppose a linear prediction model uses the matrix above:

with

Because points equal twice days, we can write

and therefore

Only the combined coefficient

affects predictions. The data cannot decide how to split that quantity between the two redundant columns.

The matrix statement of this fact is especially important:

The nonzero vector

is a direction in the null space of $X$. It represents a coefficient change that produces no prediction change. For any number $t$,

So if one coefficient vector fits the data, infinitely many do:

This is the exact algebra behind the statement “the coefficients are not identifiable.”

For ordinary least squares, the familiar coefficient expression is

Exact multicollinearity means $X$ is not full column rank, so $X^{\mathsf{T}}X$ is singular and its inverse does not exist.

Some software may still return a coefficient vector by using a pseudoinverse, regularization, or an internal convention that drops a column. That does not restore a unique, data-supported interpretation for the separate redundant coefficients. It only selects one among many equivalent representations.

A common source: the dummy-variable trap

Categorical features often create exact multicollinearity during preprocessing.

Suppose every workspace belongs to exactly one plan category:

• free;
• pro;
• enterprise.

Create one binary indicator per category, plus an intercept:

For every row,

Since the intercept column is also all ones,

Thus the columns obey the exact relation

With four columns, the rank can be at most three.

The standard remedy is to retain the intercept and omit one category, which becomes the reference category. For example, retaining free and pro indicators but omitting enterprise produces a full-rank encoding, provided no other dependency exists. The intercept then represents the enterprise baseline, and the two indicator coefficients represent deviations from it.

Other common causes of exact multicollinearity include:

• retaining a duplicated feature under a different name;
• creating both a raw feature and an exact constant multiple of it;
• including a total together with all mutually exclusive component counts;
• adding both a feature and an exact deterministic duplicate created in a join or aggregation step;
• combining manually created dummy variables with an automatically added intercept.

These are feature-construction issues, not evidence of causal confounding. Rank concerns whether the columns you supplied to a model are algebraically redundant.

Diagnose rank in NumPy

For a dense numeric design matrix, NumPy provides np.linalg.matrix_rank.

import numpy as np

active_days = np.array([1.0, 2.0, 3.0, 4.0])
activity_points = 2.0 * active_days

X = np.column_stack([
    np.ones(len(active_days)),  # intercept
    active_days,
    activity_points,
])

rank = np.linalg.matrix_rank(X)
n_rows, n_columns = X.shape

print(X)
print(f"Shape: {n_rows} rows, {n_columns} columns")
print(f"Rank: {rank}")
print(f"Rank deficiency: {n_columns - rank}")

The expected result is:

Shape: 4 rows, 3 columns
Rank: 2
Rank deficiency: 1

A compact diagnostic is:

def check_full_column_rank(X):
    n_rows, n_columns = X.shape
    rank = np.linalg.matrix_rank(X)

    if n_rows < n_columns:
        return (
            f"Rank is {rank} with {n_columns} columns. "
            "There are fewer rows than columns, so full column rank is impossible."
        )

    if rank < n_columns:
        return (
            f"Rank deficient: rank {rank}, columns {n_columns}. "
            "Investigate exact linear dependencies in feature construction."
        )

    return f"Full column rank: rank {rank}, columns {n_columns}."

print(check_full_column_rank(X))

Two cautions make this check more trustworthy in a real workflow.

First, rank detects that a dependency exists, but it does not automatically explain the business or pipeline cause. If rank is deficient, inspect feature definitions, encoding choices, joins, and derived totals. For the activity example, the cause is visible in the feature specification:

Second, computers use finite-precision arithmetic. np.linalg.matrix_rank determines numerical rank using a tolerance. It is reliable for clean, intentionally exact constructions like duplicate columns and dummy-variable traps, but a full-rank result does not prove that features are well behaved. Two features can be extremely close to redundant while still not being exactly dependent; that creates a different numerical and statistical issue, to be treated separately from exact multicollinearity.

For this lesson, use rank as a precise diagnostic question:

Does this design matrix contain an exact linear dependency among its columns?

If yes, remove or deliberately re-encode redundant information before treating individual linear-model coefficients as meaningful.

Key takeaways

Rank is the number of independent columns in a design matrix. For an $n\times p$ matrix with at least $p$ rows:

means the design matrix has full column rank, while

signals exact multicollinearity.

Rank deficiency means there is a nonzero vector $\mathbf{c}$ such that

Consequently, changing coefficients along $\mathbf{c}$ leaves predictions unchanged. The model may still make the same predictions, but separate coefficients for redundant features are not uniquely determined.

In practice, diagnose rank with np.linalg.matrix_rank, then trace any deficiency back to feature construction. Duplicate features, deterministic transformations, totals plus every component, and an intercept plus all dummy variables are frequent causes.

Next, you will shift from linear-algebra structure to the transformations used constantly in binary classification: probabilities, odds, log-odds, logarithms, and exponentials.