Evaluating CNN Performance

Introduction

Welcome back! In our last lesson, you successfully trained your first Convolutional Neural Network and evaluated its performance using a single number: overall accuracy. While accuracy is a useful starting point, it often hides crucial details about a model's performance. For example, a model with 90% accuracy might be excellent at identifying 9 out of 10 classes but fail completely on the 10th one.

To become a proficient AI engineer, you need to move beyond a single score and learn to perform a more detailed diagnosis of your model. This lesson is dedicated to that skill. We will explore how to evaluate a CNN using an accuracy score, a confusion matrix, and per-class metrics like precision, recall, and F1-score.

By the end of this lesson, you will be able to answer not just "how accurate is my model?" but also "where is my model getting confused, and how well does it perform on each specific category?"

1. The Limits of Accuracy and the Power of the Confusion Matrix

Overall accuracy tells us the percentage of correct predictions across all classes. It's a great high-level metric, but it doesn't show us the types of errors the model is making. To see that, we need a confusion matrix.

A confusion matrix is a grid that compares the model's predicted labels against the actual (true) labels. This allows us to see, for instance, how many times the model mistook a "Shirt" for a "Pullover".

To understand how to interpret a confusion matrix in more detail, let's watch a short video.

Implementing a Confusion Matrix in PyTorch

Generating a confusion matrix involves three main steps:

1. Run the entire test dataset through your trained model to get predictions for every sample.
2. Use a library to compute the matrix from these predictions and the true labels.
3. Plot the matrix for visual interpretation.

A modern and clean way to do this is by using the torchmetrics library for computation and mlxtend for plotting.

This process gives you a powerful visual diagnostic tool to understand your model's behavior at a glance.

2. Quantifying Performance: Precision, Recall, and F1-Score

The confusion matrix is excellent for visualization, but we also need quantitative metrics to describe per-class performance. This is where Precision, Recall, and F1-Score come in.

Before defining them, we need to understand four basic terms for a given class (e.g., "Cat"):

• True Positive (TP): The model correctly predicts "Cat" when the image is a cat.
• False Positive (FP): The model incorrectly predicts "Cat" when the image is not a cat (e.g., a dog).
• True Negative (TN): The model correctly predicts "Not Cat" when the image is not a cat.
• False Negative (FN): The model incorrectly predicts "Not Cat" when the image is a cat.

With these terms, we can define our metrics:

Precision

Precision answers the question: "Of all the times the model predicted a certain class, how often was it correct?"
$\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}$
A high precision for the "Cat" class means that when the model says it's a cat, it's very likely to be a cat. This metric is important in scenarios where false positives are costly (e.g., a spam filter incorrectly marking an important email as spam).

Recall (or Sensitivity)

Recall answers the question: "Of all the actual instances of a class, how many did the model correctly identify?"
$\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}}$
A high recall for the "Cat" class means the model is good at finding all the cats in the dataset. This metric is critical where false negatives are costly (e.g., a medical test failing to detect a disease).

F1-Score

The F1-score is the harmonic mean of Precision and Recall. It provides a single score that balances both metrics.
$\text{F1-Score}=2\times \frac{\text{Precision}\times \text{Recall}}{\text{Precision}+\text{Recall}}$
The F1-score is useful when you need a balance between precision and recall, especially if you have an uneven class distribution.

The following video provides a clear, step-by-step walkthrough of how these metrics are calculated directly from a confusion matrix.

3. The Classification Report: All Metrics at a Glance

Manually calculating these metrics is insightful but impractical. Fortunately, scikit-learn provides a convenient function, classification_report, that does all the work for us.

To use it, you first gather your model's predictions on the test set (as you did for the confusion matrix), and then pass the true labels and predicted labels to the function.

The output of classification_report is a text table that gives you a comprehensive summary of your model's performance, including:

• Precision, recall, and F1-score for each class.
• Support: The number of true instances for each class.
• Macro Avg: The average of the metric across all classes (each class has equal weight).
• Weighted Avg: The average of the metric, weighted by the support of each class. This is more representative if you have class imbalance.

Conclusion

In this lesson, you've expanded your model evaluation toolkit far beyond a single accuracy score. You now have a framework for deeply analyzing your model's performance.

Key Takeaways:

• Overall accuracy provides a high-level summary but can be misleading, especially with imbalanced datasets.
• A confusion matrix is a powerful visual tool that shows exactly where your model is making errors by comparing predicted labels to true labels.
• Precision measures the reliability of your model's positive predictions (minimizing false positives).
• Recall measures your model's ability to find all actual positive instances (minimizing false negatives).
• The F1-score provides a single, balanced measure of both precision and recall.
• Libraries like torchmetrics, mlxtend, and scikit-learn make it easy to generate these visualizations and reports.

You've learned to identify what and where your model's weaknesses are. In our next lesson, we will start exploring techniques to address these weaknesses. We will begin with data augmentation, a powerful method for creating more training data and building more robust models that generalize better to unseen images.