Evaluating Classification Models with Confusion Matrices

In our previous lesson, we established a toolkit for evaluating regression models by measuring their error with metrics like MSE, MAE, and R². We are now shifting our focus from predicting continuous values to assigning discrete labels—the task of classification. This change in task demands a completely new set of evaluation tools. Answering "how wrong is the price?" is fundamentally different from answering "did we correctly identify this as spam or not?"

This lesson will equip you with the fundamental tools for evaluating classification models. Our primary goal is to compute accuracy, precision, recall, and F1-score using a confusion matrix. You will learn not just the formulas for these metrics, but more importantly, the distinct questions each one answers about a model's performance. This understanding is crucial for moving beyond a superficial view of model evaluation and selecting the right metric for the right problem—a key skill you'll need, especially given your goal to teach this material yourself one day.

The Confusion Matrix: A Map of Your Model's Predictions

For any classification model, there are four possible outcomes for a single prediction. For a binary problem (e.g., predicting "spam" vs. "not spam"), these are:

True Positive (TP): The model correctly predicts the positive class. (e.g., predicts "spam" and it is spam).
True Negative (TN): The model correctly predicts the negative class. (e.g., predicts "not spam" and it is not spam).
False Positive (FP): The model incorrectly predicts the positive class. Also known as a Type I error. (e.g., predicts "spam" but it was a legitimate email).
False Negative (FN): The model incorrectly predicts the negative class. Also known as a Type II error. (e.g., predicts "not spam" but it was spam).

The confusion matrix is a simple but powerful table that tallies these four outcomes across an entire test set, giving us a complete picture of a model's performance.

To start, let's get a formal definition of these four terms from the V7 Labs article, "Confusion Matrix: How To Use It & Interpret Results."

Confusion Matrix: How To Use It & Interpret Results ...

This section provides clear, concise definitions for the four essential components of a binary confusion matrix.

Please read the section Confusion Matrix for Binary Classes. Focus on mastering the definitions of True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN).

The image below, from Analytics Vidhya, provides an excellent annotated example of a confusion matrix in a medical diagnosis context. Notice how it clearly maps each quadrant to a specific prediction outcome.

A confusion matrix for a medical diagnosis model. The matrix shows 30 True Positives (sick people correctly identified), 30 False Positives (healthy people incorrectly identified as sick), 10 False Negatives (sick people missed), and 930 True Negatives (healthy people correctly identified).

With this $2 \times 2$ matrix as our foundation, we can now derive the key metrics for classification.

Accuracy: The Simplest, but Deceiving, Metric

The most intuitive metric is accuracy, which answers the question: "What fraction of predictions did the model get right?"

Accuracy = \frac{Number of Correct Predictions}{Total Number of Predictions} = \frac{T P + T N}{T P + T N + F P + F N}

Using the numbers from the "Example Confusion matrix" above:

Accuracy = \frac{30 + 930}{30 + 930 + 30 + 10} = \frac{960}{1000} = 0.96 or 96%

A 96% accuracy seems fantastic. However, accuracy can be dangerously misleading, especially with imbalanced datasets. In our example, 960 out of 1000 people are healthy. A trivial model that always predicts "healthy" would still achieve an accuracy of $\frac{930 + 30}{1000} = 96%$ ! Yet, it would fail to identify a single sick person, making it useless for its intended purpose. This is why we need more nuanced metrics.

Precision and Recall: A Tale of Two Errors

To overcome the limitations of accuracy, we turn to two metrics that focus on the performance of the positive class: precision and recall. The distinction between them is subtle but critical. The codebasics channel on YouTube offers one of the best visual explanations of this difference.

Precision, Recall, F1 score, True Positive|Deep Learning Tutorial 19 (Tensorflow2.0, Keras & Python)

This video uses a simple "dog vs. not a dog" example to build an incredibly strong intuition for precision and recall. It masterfully clarifies the different "baselines" or denominators used for each calculation.

Watch the following segments: Introduction and setup. Explanation of TP, FP, TN, and FN using the visual example. Explanation of Precision, which focuses on the set of positive predictions. Explanation of Recall, which focuses on the set of positive truths.

As the video explains, the key difference lies in the question they answer:

Precision: How trustworthy are the positive predictions?

Precision focuses on the predictions your model made. It answers: Of all the instances the model predicted as positive, what fraction was actually positive?

Precision = \frac{T P}{T P + F P}

When it's important: Use precision when the cost of a False Positive is high. For example, in email spam detection, you don't want to mistakenly mark an important email (a legitimate message) as spam. High precision ensures that when the model says "spam," it's very likely to be correct.

Recall (or Sensitivity): How well did we find all the positives?

Recall focuses on the actual ground truth. It answers: Of all the actual positive instances in the data, what fraction did the model successfully identify?

Recall = \frac{T P}{T P + F N}

When it's important: Use recall when the cost of a False Negative is high. In medical diagnosis, failing to detect a disease (a false negative) is often far more dangerous than flagging a healthy person for more tests (a false positive). High recall ensures you find as many actual cases as possible.

Let's compute these for our medical example:

Precision = $\frac{30}{30 + 30} = 0.50$ . When the model predicts someone is sick, it's only correct 50% of the time.
Recall = $\frac{30}{30 + 10} = 0.75$ . The model successfully finds 75% of all sick people.

Notice how these numbers (50% and 75%) paint a much more detailed and sobering picture than the single 96% accuracy score.

The Trade-off and the F1-Score

You can't have your cake and eat it too. In most models, increasing precision tends to decrease recall, and vice-versa. This is known as the precision-recall trade-off.

Imagine you're tuning your cancer-detection model. To increase recall, you could lower the threshold for flagging a scan as "potentially cancerous." You'll catch more real cases (fewer false negatives), but you'll also inevitably flag more healthy scans (more false positives), which lowers your precision.

So, how do we find a balance? This is where the F1-Score comes in. The F1-score is the harmonic mean of precision and recall.

F1-Score = 2 \times \frac{Precision \times Recall}{Precision + Recall}

Since you have a strong math background, it's worth noting why we use the harmonic mean instead of a simple average. The harmonic mean heavily penalizes low values. A model with 1.0 precision and 0.1 recall would have an arithmetic mean of 0.55, but its F1-score would be only $\approx 0.18$ . The F1-score is high only when both precision and recall are high.

Let's get a clear explanation of this trade-off and the F1-score from the NeuralNine channel.

Evaluation Metrics For Classification - Full Overview

This video provides an excellent conceptual overview of the precision-recall trade-off and the F1-score's role in balancing them.

Please watch the segment explaining the recall/precision trade-off, followed by the section on the F1-score.

For our medical example, the F1-score would be:

F1-Score = $2 \times \frac{0.50 \times 0.75}{0.50 + 0.75} = 2 \times \frac{0.375}{1.25} = 0.60$

The image below summarizes the calculation for all three metrics on a different example.

This diagram shows a confusion matrix and the corresponding calculations for Precision (60%), Recall (75%), and F1-Score (67%). It visually links the TP, FP, and FN values from the matrix to the formulas.

A Practical Walkthrough

Now let's solidify this by walking through a complete example from an article on Towards Data Science. This resource shows how the metrics are calculated and, crucially, discusses how to interpret them in the context of a real-world problem.

Confusion Matrix Made Simple: Accuracy, Precision, Recall ...

This article provides a superb, practical walkthrough using a breast cancer dataset. It clearly connects the confusion matrix values to the formulas and discusses the real-world implications of choosing one metric over another.

First, read the section that breaks down a sample confusion matrix into TP, FP, FN, and TN. Next, study the section where the author calculates accuracy, precision, and recall using the values from the matrix. This directly applies the formulas we've just discussed. Finally, read the crucial discussion on choosing the right metric for a healthcare application. This highlights why maximizing recall is often the priority in such scenarios.

Conclusion

In this lesson, we transitioned from regression to classification evaluation. You now understand how to construct a confusion matrix and use it to compute the core metrics that reveal a model's true performance, especially when a simple accuracy score can be misleading.

Key Takeaways:

The Confusion Matrix is the foundational tool for classification evaluation, tabulating True Positives, True Negatives, False Positives, and False Negatives.
Accuracy measures overall correctness but is unreliable for imbalanced datasets.
Precision measures the reliability of positive predictions, crucial when False Positives are costly.
Recall measures the model's ability to find all actual positive cases, crucial when False Negatives are costly.
The F1-Score provides a single-number summary of the model's performance by calculating the harmonic mean of precision and recall, offering a balance between the two.

We have established the "what" and "how" of these metrics. In our next lesson, we will explore a powerful visualization tool that builds directly on these concepts: the Receiver Operating Characteristic (ROC) curve and its associated metric, the Area Under the Curve (AUC). This will allow us to evaluate a model's performance across all possible classification thresholds.

Recommended resources