Understanding Confusion Matrices and Performance Metrics

1. The Confusion Matrix: Organizing Predictions

After we fit a logistic regression model, it gives us a predicted probability for each observation (e.g., a 75% probability of having the disease). To make a concrete prediction, we apply a cut-off threshold (typically 0.5). If the probability is above the threshold, we predict "Yes" (positive); otherwise, we predict "No" (negative).

The confusion matrix is a simple but powerful table that summarizes how our predictions stack up against the actual, true outcomes. It shows us not just how many predictions were right or wrong, but also the types of errors the model made.

To get a clear, intuitive introduction to the confusion matrix, please watch the following video from StatQuest.

As the video explained, a 2x2 confusion matrix has four essential components. The layout is crucial: typically, predictions are on one axis (rows) and the actual truth is on the other (columns).

Let's formally define these terms, which are the bedrock of classification performance evaluation:

• True Positive (TP): The model correctly predicted the positive class. (e.g., a patient with the disease is correctly identified as having the disease).
• True Negative (TN): The model correctly predicted the negative class. (e.g., a healthy patient is correctly identified as healthy).
• False Positive (FP): The model incorrectly predicted the positive class. This is also known as a Type I Error. (e.g., a healthy patient is incorrectly told they have the disease).
• False Negative (FN): The model incorrectly predicted the negative class. This is also known as a Type II Error. (e.g., a patient with the disease is incorrectly told they are healthy).

The "positive" class is the outcome of interest. This could be the presence of a disease, a fraudulent transaction, or a customer who churns. The choice is context-dependent.

2. Sensitivity and Specificity: Beyond Raw Counts

The counts in the confusion matrix are insightful, but to compare models or understand performance in a standardized way, we need to convert them into rates or proportions. Two of the most important metrics derived from the confusion matrix are sensitivity and specificity.

Let's formalize those definitions:

Sensitivity (True Positive Rate or Recall)

Sensitivity measures how well the model identifies actual positive cases.

$\text{Sensitivity}=\frac{\text{TP}}{\text{TP}+\text{FN}}=\frac{\text{Number of Correctly Identified Positives}}{\text{Total Number of Actual Positives}}$

• Interpretation: "Of all the people who actually have the disease, what percentage did our model correctly flag?"
• A model with high sensitivity is good at finding the positive cases. It has a low false negative rate.

Specificity (True Negative Rate)

Specificity measures how well the model identifies actual negative cases.

$\text{Specificity}=\frac{\text{TN}}{\text{TN}+\text{FP}}=\frac{\text{Number of Correctly Identified Negatives}}{\text{Total Number of Actual Negatives}}$

• Interpretation: "Of all the people who are actually healthy, what percentage did our model correctly clear?"
• A model with high specificity is good at avoiding false alarms. It has a low false positive rate.

3. The Sensitivity vs. Specificity Trade-Off: A Functional Understanding

This is one of the most critical concepts in classification. Rarely can you maximize both sensitivity and specificity simultaneously. Improving one often comes at the cost of the other. The "best" model depends entirely on the real-world consequences of its errors.

Consider two scenarios:

1. Airport Security Screening:

   • Positive Case: A weapon is present.
   • Cost of a False Negative (FN): Catastrophic. A weapon gets onto a plane.
   • Cost of a False Positive (FP): Annoying but manageable. A benign object (like a water bottle) is flagged, leading to a manual bag check.
   • Priority: We need extremely high sensitivity. We are willing to tolerate low specificity (many false alarms) to ensure we don't miss a true threat.

2. Spam Email Filtering:

   • Positive Case: An email is spam.
   • Cost of a False Negative (FN): Annoying. A spam email lands in your inbox.
   • Cost of a False Positive (FP): Potentially very high. An important email (e.g., a job offer) is sent to the spam folder and missed.
   • Priority: We need very high specificity. We want to be absolutely sure that when we call an email "not spam," it's truly not spam. We are willing to tolerate lower sensitivity (letting some spam through) to avoid this critical error.

This trade-off is not just a theoretical concept; it's something you actively control. In logistic regression, you control it by changing the probability cut-off.

• A low cut-off (e.g., predict "disease" if probability > 0.10) makes it easy to classify as positive. This will increase sensitivity (you'll catch more true positives) but decrease specificity (you'll generate more false positives).
• A high cut-off (e.g., predict "disease" if probability > 0.90) makes it hard to classify as positive. This will increase specificity (fewer false alarms) but decrease sensitivity (you'll miss more true positives).

Conclusion

In this lesson, we moved from assessing model fit to evaluating predictive performance. You now have a solid functional understanding of the tools used to measure how well a classification model performs its primary task.

Key Takeaways:

• The confusion matrix is the starting point for evaluating classification performance, organizing predictions into True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN).
• Sensitivity measures the model's ability to identify true positives ($\frac{\text{TP}}{\text{Actual Positives}}$). It answers: "How many of the actual sick people did we find?"
• Specificity measures the model's ability to identify true negatives ($\frac{\text{TN}}{\text{Actual Negatives}}$). It answers: "How many of the actual healthy people did we clear?"
• There is a fundamental trade-off between sensitivity and specificity, controlled by the probability cut-off. Prioritizing one over the other depends entirely on the real-world costs of false positive versus false negative errors.

Preview of the Next Lesson:

Choosing a single cut-off to calculate one sensitivity/specificity pair gives a narrow view of the model's performance. What if we wanted to see how the model performs across all possible cut-offs? In the next lesson, we will introduce the Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC), which do exactly that, providing a comprehensive view of a model's discriminative power.