ROC Curves and AUC for Model Evaluation

1. Beyond a Single Threshold

Recall that classifiers like Logistic Regression don't just output a class label (like 1 or 0). They output a probability—for instance, a 75% probability that an email is spam. We then use a classification threshold to make a final decision. By default, this threshold is usually 0.5: if the probability is > 0.5, we classify it as 1; otherwise, as 0.

But what if 0.5 isn't the best threshold for our problem?

• For medical diagnosis, we might lower the threshold to 0.2 to catch as many potential cases as possible (maximizing recall), even if it means more false alarms for healthy patients.
• For email spam filtering, we might raise the threshold to 0.9 to be extremely sure an email is spam before hiding it (maximizing precision), even if some spam gets through.

The choice of threshold directly changes our confusion matrix, and thus our precision and recall scores. This raises two questions:

1. How can we visualize a model's performance across all of these potential thresholds?
2. How can we compare two different models without having to worry about picking the "best" threshold for each one?

The ROC curve and AUC are the answers.

2. Building the ROC Curve

An ROC curve is a graph that plots a model's performance by looking at two key rates as the classification threshold is varied from 0.0 to 1.0.

The axes are:

• Y-axis: True Positive Rate (TPR). This is simply another name for Recall or Sensitivity. It answers: What proportion of the actual positives did the model correctly identify?
  $\text{TPR}=\frac{\text{True Positives}}{\text{Actual Positives}}=\frac{\text{TP}}{\text{TP}+\text{FN}}$
• X-axis: False Positive Rate (FPR). This is a new metric for us. It answers: What proportion of the actual negatives did the model incorrectly classify as positive?
  $\text{FPR}=\frac{\text{False Positives}}{\text{Actual Negatives}}=\frac{\text{FP}}{\text{FP}+\text{TN}}$

The ROC curve shows the trade-off between these two metrics. Ideally, we want a high TPR (we find all the real positives) and a low FPR (we don't create many false alarms).

The following video from StatQuest provides an exceptionally clear, step-by-step guide to how an ROC curve is constructed.

3. Interpreting ROC Curves and the AUC Metric

Now that you've seen how an ROC curve is made, let's discuss how to interpret it.

• Perfect Classifier: A perfect model would have a curve that goes straight up the Y-axis to (0, 1) and then across to (1, 1). This corresponds to a 100% True Positive Rate with a 0% False Positive Rate. The curve "hugs" the top-left corner.
• Random Classifier: A model that is no better than random guessing will have a curve that lies on the diagonal line from (0, 0) to (1, 1).
• Good Classifier: A useful model will have a curve that bows up and to the left, as far away from the diagonal line as possible.

While the curve is a great visualization, we often need a single number to quantify and compare model performance. This is the Area Under the (ROC) Curve, or AUC.

AUC is exactly what it sounds like: the area under the ROC plot. It provides a single score that summarizes the model's performance across all thresholds.

• AUC = 1.0: Perfect classifier.
• AUC = 0.5: Random-guess classifier.
• AUC > 0.5: The model has discriminative ability. The closer to 1.0, the better.

A very useful interpretation of AUC is that it represents the probability that the model ranks a randomly chosen positive instance higher than a randomly chosen negative instance. This makes AUC an excellent measure of a model's ability to separate the classes and explains why it's so robust, especially with imbalanced datasets.

The next video explains this interpretation and how AUC helps in model comparison.

For those who enjoy the mathematical details, the following article provides a deeper look into the two common ways AUC is calculated: the trapezoidal rule (summing up the areas of trapezoids under the curve) and the ranking method. This is optional but recommended if you want to solidify your theoretical understanding.

4. Implementation in Python with Scikit-Learn

As always, we'll turn to scikit-learn for a practical implementation. The key tools are:

• predict_proba(): To get the predicted probabilities from our trained classifier. This is essential, as ROC/AUC operates on probabilities, not final class predictions.
• roc_curve(): To compute the False Positive Rates, True Positive Rates, and thresholds needed to plot the curve.
• roc_auc_score(): To directly calculate the AUC score.

Let's compare two models we've worked with before—Logistic Regression and a K-Nearest Neighbors classifier—to see which one has better discriminative power on our test data.

import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import roc_curve, roc_auc_score




# Generate a sample dataset
X, y = make_classification(
    n_samples=1000, 
    n_classes=2, 
    n_features=20, 
    weights=[0.9, 0.1], # Imbalanced dataset
    random_state=42
)




# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)




# --- Train Models ---
# 1. Logistic Regression
log_reg = LogisticRegression()
log_reg.fit(X_train, y_train)




# 2. K-Nearest Neighbors
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)




# --- Predict Probabilities ---
# Get probabilities for the positive class (class 1)
lr_probs = log_reg.predict_proba(X_test)[:, 1]
knn_probs = knn.predict_proba(X_test)[:, 1]




# --- Calculate AUC Scores ---
lr_auc = roc_auc_score(y_test, lr_probs)
knn_auc = roc_auc_score(y_test, knn_probs)

print(f"Logistic Regression AUC: {lr_auc:.4f}")
print(f"KNN AUC: {knn_auc:.4f}")




# --- Calculate ROC Curve points ---
lr_fpr, lr_tpr, _ = roc_curve(y_test, lr_probs)
knn_fpr, knn_tpr, _ = roc_curve(y_test, knn_probs)




# --- Plot the ROC Curves ---
plt.figure(figsize=(8, 6))
plt.plot(lr_fpr, lr_tpr, linestyle='--', label=f'Logistic Regression (AUC = {lr_auc:.2f})')
plt.plot(knn_fpr, knn_tpr, marker='.', label=f'KNN (AUC = {knn_auc:.2f})')



# Plot the random guess line
plt.plot([0, 1], [0, 1], linestyle='--', color='grey', label='Random Guess')




# Axis labels and title
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve Comparison')
plt.legend()
plt.show()

This code trains two models, calculates their AUC scores, and plots their ROC curves on the same graph. This is a standard and powerful way to compare model performance at a glance. The model whose curve is further to the top-left and has a higher AUC score is generally the better classifier.

For a final reference on the Python code, the following article provides a clear, runnable example.

Conclusion

In this lesson, you have added a sophisticated and widely-used evaluation technique to your skillset. You can now assess model performance in a way that is independent of a specific classification threshold.

Key Takeaways:

• Classification performance depends on the chosen threshold, which creates a trade-off between precision and recall (or TPR and FPR).
• The ROC curve visualizes this trade-off by plotting the True Positive Rate vs. the False Positive Rate across all possible thresholds.
• A good model has an ROC curve that bows towards the top-left corner of the plot.
• The Area Under the Curve (AUC) quantifies the entire ROC curve into a single value between 0 and 1, making it perfect for model comparison.
• An AUC of 1.0 represents a perfect model, while an AUC of 0.5 represents a model with no better than random performance.

In our next lesson, we will tackle a crucial step in building high-performing models: performing cross-validation and tuning hyperparameters using grid search. Now that you are an expert at evaluating models, you'll learn how to systematically find the best settings for your models to maximize these evaluation metrics.