Solving Systems with Inverse Matrices

Introduction

Welcome to the next lesson in our study of Linear Algebra for Machine Learning. In our previous session, you learned how to compute the determinant and inverse of a matrix. We established that the determinant tells us if a transformation is reversible, and the inverse matrix, $A^{- 1}$ , represents that "undo" operation.

Today, we will directly apply these concepts to achieve a classic and powerful goal: solving systems of linear equations. This lesson corresponds to the learning outcome: Solve systems of linear equations using matrix inversion. We will transform a set of equations into the compact matrix form $A x = b$ and then use the inverse, $A^{- 1}$ , to elegantly solve for the unknown variables in the vector $x$ . This technique is a cornerstone of computational mathematics and provides the foundation for understanding more advanced methods used in AI.

1. The Core Idea: Reversing the Transformation

A system of linear equations can be viewed as a transformation problem. We have a matrix $A$ that transforms an unknown vector $x$ into a known vector $b$ . Our goal is to find the original vector $x$ . If the transformation is reversible (i.e., det(A) ≠ 0), we can solve this by applying the inverse transformation.

The following video provides the essential geometric intuition for this process.

Inverse matrices, column space and null space | Chapter 7, Essence of linear algebra

The video 'Inverse matrices, column space and null space' from 3Blue1Brown, which we touched on last lesson, beautifully illustrates how solving a system of equations is equivalent to finding which vector lands on another after a transformation.

Please watch the segment from 01:01 to 06:52. Focus on these two key ideas: From 01:01 to 03:25: How a system of linear equations is cleanly represented by the matrix equation A\mathbf{x} = \mathbf{v}. From 03:44 to 06:52: The conceptual leap that if A has a non-zero determinant, an inverse transformation A^{-1} exists. Applying this inverse to \mathbf{v} reveals the unique \mathbf{x} we are looking for.

The Formal Derivation

The video gives us the "why"; now let's formalize the "how". As you prefer a theory-first approach, the algebraic derivation is both simple and powerful.

Given a system of equations represented as:

A x = b

Here, $A$ is the square matrix of coefficients, $x$ is the column vector of unknown variables, and $b$ is the column vector of constants.

If $A$ is invertible, we can pre-multiply both sides of the equation by its inverse, $A^{- 1}$ :

A^{- 1} (A x) = A^{- 1} b

Because matrix multiplication is associative, we can regroup the left side:

(A^{- 1} A) x = A^{- 1} b

From the last lesson, we know that a matrix multiplied by its inverse yields the identity matrix, $I$ :

I x = A^{- 1} b

And since the identity matrix acts like the number 1 in matrix multiplication ( $I x = x$ ), we arrive at the solution:

x = A^{- 1} b

This proves that if the matrix $A$ is invertible, the system of equations has a unique solution given by the product of the inverse of the coefficient matrix and the constant vector.

To fully satisfy our theoretical grounding, let's watch a short, formal proof that this solution is not just a solution, but the only solution.

Solve Linear System with Inverse Matrix | Linear Algebra

The video 'Solve Linear System with Inverse Matrix' by Wrath of Math offers a concise proof of this uniqueness.

Watch the segment from 00:49 to 02:21. The video proves that any solution must be equal to A^{-1}\mathbf{b}, confirming it's unique.

2. The Method in Practice

We now have a complete method for solving a system of $n$ linear equations with $n$ variables:

Represent the System: Write the system in matrix form $A x = b$ .
Check Invertibility: Calculate the determinant of $A$ . If $det (A) = 0$ , the matrix is singular, and this method cannot be used. The system either has no solution or infinitely many solutions.
Find the Inverse: If $det (A) \neq = 0$ , calculate $A^{- 1}$ using the appropriate method (formula for 2x2, Gauss-Jordan for larger matrices, as learned in the previous lesson).
Calculate the Solution: Compute the final solution by multiplying the inverse matrix by the constant vector: $x = A^{- 1} b$ .

Let's see this process in action. The following resource provides clear, worked-out examples for both 2x2 and 3x3 systems.

7.8: Solving Systems with Inverses - Mathematics LibreTexts

The article 'Solving Systems with Inverses' from Mathematics LibreTexts walks through the entire process, from setting up the matrix equation to calculating the final variable values.

Please read the final section of the article, titled 'Solving a System of Linear Equations Using the Inverse of a Matrix'. Start by reading the explanation of how to set up the AX=B form. Pay close attention to Example 7, which solves a 2x2 system step-by-step. Then, follow Example 8, which demonstrates the same process for a more complex 3x3 system. Notice that finding the inverse of the 3x3 matrix is the most computationally intensive part, reinforcing what we learned in the last lesson.

Quick Practice

Let's apply the method to the 2x2 system from the reading:

3 x + 8 y 4 x + 11 y = 5 = 7

Matrix Form: $A = [34811]$ , $x = [x y]$ , $b = [57]$ .
Determinant: $det (A) = (3) (11) - (8) (4) = 33 - 32 = 1$ . Since it's not zero, an inverse exists.
Inverse (using 2x2 formula): $A^{- 1} = \frac{1}{1} [11 - 4 - 8 3] = [11 - 4 - 8 3]$ .
Solution:
$x = A^{- 1} b = [11 - 4 - 8 3] [57] = [(11) (5) + (- 8) (7) (- 4) (5) + (3) (7)] = [55 - 56 - 20 + 21] = [- 1 1]$
So, the solution is $x = - 1, y = 1$ , matching the example.

3. Theoretical Connections and Practical Limitations

For someone aiming to be a polymath, it's not enough to know the method; you must also understand its context and its boundaries.

The Invertible Matrix Theorem

The fact that an invertible matrix $A$ guarantees a unique solution to $A x = b$ is just one piece of a much larger puzzle. The Invertible Matrix Theorem is a powerful result that links a dozen different properties of a square matrix together. For an $n \times n$ matrix $A$ , if any one of these properties is true, all of them are true.

The Inverse of a Matrix — Linear Algebra, Geometry, and Computation

This section from 'The Inverse of a Matrix' provides a concise summary of this important theorem. It connects many concepts we've discussed, such as pivots, linear independence, and unique solutions.

Read the section titled 'The Invertible Matrix Theorem'. Don't worry about memorizing all ten points. Instead, focus on appreciating how invertibility is the central pin connecting all these ideas. For instance, notice the equivalence: 'A is an invertible matrix' \Leftrightarrow 'The equation A\mathbf{x} = \mathbf{b} has a unique solution for each \mathbf{b} in \mathbb{R}^n.' \Leftrightarrow 'The columns of A form a linearly independent set.'

This theorem acts as a powerful diagnostic tool. If you can prove just one of the conditions (e.g., that the columns are linearly independent), you automatically know the matrix is invertible and that a unique solution exists for any $b$ .

A Practical Warning: Ill-Conditioned Matrices

While $x = A^{- 1} b$ is theoretically perfect, in the world of computer calculations with finite precision, it has a weakness. Some invertible matrices are "nearly singular." A tiny change in the vector $b$ can cause a huge, disproportionate change in the solution $x$ . Such matrices are called ill-conditioned.

Using inversion to solve systems with ill-conditioned matrices can lead to wildly inaccurate results due to floating-point rounding errors.

The Inverse of a Matrix — Linear Algebra, Geometry, and Computation

The same document provides an excellent, non-technical explanation of this practical issue.

Read the section titled 'Ill-Conditioned Matrices'. Pay attention to the demonstration where a tiny change in \mathbf{b} leads to a massive change in \mathbf{x}. This highlights why, in many real-world AI applications, other methods (like LU decomposition) are often preferred for solving linear systems over direct matrix inversion.

This is a key insight for any aspiring engineer: theoretical elegance must always be balanced against practical stability. Knowing when not to use a tool is as important as knowing how to use it.

Conclusion

In this lesson, you've bridged the gap between the abstract concept of a matrix inverse and its powerful application in solving systems of linear equations.

Key Takeaways:

A system of $n$ linear equations in $n$ variables can be expressed as a single matrix equation, $A x = b$ .
If the coefficient matrix $A$ is invertible ( $det (A) \neq = 0$ ), the system has a unique solution given by $x = A^{- 1} b$ .
The Invertible Matrix Theorem provides a unified framework connecting a matrix's invertibility to many other fundamental properties, such as linear independence and the nature of its solutions.
In practice, solving systems via direct matrix inversion can be numerically unstable for ill-conditioned matrices, where small input errors can lead to large output errors.

Preview of the Next Lesson:
We have now used matrices to transform vectors and solve for them. The next step is to understand the deeper geometric structure of these transformations. We will ask: "In a given transformation, are there any vectors that only get stretched or shrunk but don't change their direction?" These special vectors are called eigenvectors, and the factor by which they stretch is their eigenvalue. This concept is absolutely central to many machine learning algorithms, particularly in dimensionality reduction (like PCA) and the analysis of complex systems.

Recommended resources