ROC Curves and AUC Calculation

1. The Core Concepts: TPR and FPR

To build an ROC curve, we need to understand its axes. The curve plots the True Positive Rate (TPR) against the False Positive Rate (FPR).

Let's quickly refresh these terms, which are derived from the confusion matrix components (TP, FP, TN, FN):

• True Positive Rate (TPR): Also known as Sensitivity or Recall. It answers: "Of all the actual positive instances, what fraction did we correctly identify?"
  $\text{TPR}=\frac{\text{TP}}{\text{TP}+\text{FN}}$

• False Positive Rate (FPR): It answers: "Of all the actual negative instances, what fraction did we incorrectly label as positive?"
  $\text{FPR}=\frac{\text{FP}}{\text{FP}+\text{TN}}$

The fundamental challenge in classification is the trade-off between these two. If we want to catch every possible positive (high TPR), we might have to lower our standards, leading to more false alarms (high FPR). The ROC curve visualizes this trade-off.

For a clear explanation of these rates, the following article is a helpful reference.

2. From Probabilities to a Curve: The Threshold Mechanism

A classification model, like the logistic regression you'll build soon, typically outputs a probability between 0 and 1. To make a final decision (0 or 1), we use a threshold. For example, if probability > 0.5, we predict 1; otherwise, we predict 0.

The ROC curve is generated by doing this for every possible threshold:

1. Start with a threshold of 1.0. At this level, we likely classify everything as 0, giving TPR=0 and FPR=0. This is our first point: (0, 0).
2. Gradually lower the threshold. As we do, we start classifying more instances as 1. Both TPR and FPR will increase.
3. For each distinct threshold, we calculate a new (FPR, TPR) pair.
4. End with a threshold of 0.0. At this level, we classify everything as 1, giving TPR=1 and FPR=1. This is our last point: (1, 1).

Plotting all these (FPR, TPR) points gives us the ROC curve. A model that is better at distinguishing between classes will have a curve that "hugs" the top-left corner, achieving a high TPR with a low FPR.

3. Implementation from Scratch: The ROC Curve

Your preference for from-scratch implementation is key here. The following resource provides a fantastic, clean Python implementation of the logic we just discussed. It shows how to iterate through thresholds and calculate the TPR and FPR at each step.

Let's adapt this logic into our own script. We'll create a function that takes true labels and predicted probabilities, then returns the lists of FPRs and TPRs needed for plotting.

import numpy as np
import matplotlib.pyplot as plt

def calculate_roc_points(y_true, y_pred_probs):
    """
    Calculates the points for an ROC curve from scratch.
    
    Args:
        y_true (np.ndarray): Array of true binary labels (0 or 1).
        y_pred_probs (np.ndarray): Array of predicted probabilities for the positive class.
        
    Returns:
        tuple: A tuple containing two lists (fprs, tprs).
    """
    # Define a set of thresholds to test
    thresholds = np.linspace(0, 1, 101)
    
    fprs = []
    tprs = []
    
    for threshold in thresholds:
        # Classify predictions based on the current threshold
        y_pred = (y_pred_probs >= threshold).astype(int)
        
        # Calculate TP, FP, TN, FN
        tp = np.sum((y_true == 1) & (y_pred == 1))
        tn = np.sum((y_true == 0) & (y_pred == 0))
        fp = np.sum((y_true == 0) & (y_pred == 1))
        fn = np.sum((y_true == 1) & (y_pred == 0))
        
        # Calculate TPR and FPR
        # Add epsilon to denominators to avoid division by zero
        tpr = tp / (tp + fn + 1e-8)
        fpr = fp / (fp + tn + 1e-8)
        
        tprs.append(tpr)
        fprs.append(fpr)
        
    # The points are calculated by decreasing threshold, but plotting is clearer
    # if we sort by FPR. We reverse because thresholds go from 0 to 1,
    # which naturally sorts the rates in reverse.
    return np.array(fprs)[::-1], np.array(tprs)[::-1]

# --- Example Data ---
# Let's simulate a decent, but not perfect, model
np.random.seed(42)
y_true_sample = np.concatenate([np.zeros(50), np.ones(50)])
# Good predictions for class 0, slightly worse for class 1
y_pred_probs_sample = np.concatenate([
    np.random.rand(50) * 0.4, # Probs for true negatives
    np.random.rand(50) * 0.6 + 0.4 # Probs for true positives
])

# Calculate the points
fpr_points, tpr_points = calculate_roc_points(y_true_sample, y_pred_probs_sample)

# Plotting the curve
plt.figure(figsize=(8, 6))
plt.plot(fpr_points, tpr_points, marker='.', label='Our Model')
plt.plot([0, 1], [0, 1], linestyle='--', color='gray', label='Random Guess')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve from Scratch')
plt.legend()
plt.grid(True)
plt.show()

4. Quantifying Performance: Area Under the Curve (AUC)

The ROC curve is a great visual, but often we need a single number to summarize model performance. This is the Area Under the (ROC) Curve (AUC).

The AUC represents the probability that a randomly chosen positive instance will be ranked higher by the model than a randomly chosen negative instance.

• AUC = 1.0: Perfect classifier.
• AUC = 0.5: Useless classifier (same as random guessing).
• AUC < 0.5: Classifier is worse than random.

To calculate this from our list of (FPR, TPR) points, we can approximate the area using the trapezoidal rule. We treat each segment between two points on our curve as a trapezoid and sum their areas.

The area of a single trapezoid between points (x1, y1) and (x2, y2) is (x2 - x1) * (y1 + y2) / 2.

5. Implementation from Scratch: AUC

The same CodeSignal article also provides a from-scratch implementation for calculating AUC using this exact method.

Let's integrate this into our script.

def calculate_auc(fprs, tprs):
    """
    Calculates the Area Under the ROC Curve (AUC) from scratch using the trapezoidal rule.
    
    Args:
        fprs (np.ndarray): Array of false positive rates.
        tprs (np.ndarray): Array of true positive rates.
        
    Returns:
        float: The calculated AUC score.
    """
    # Sort by fprs to ensure correct order for area calculation
    sorted_indices = np.argsort(fprs)
    fprs_sorted = fprs[sorted_indices]
    tprs_sorted = tprs[sorted_indices]
    
    auc = 0.0
    for i in range(1, len(fprs_sorted)):
        # Area of trapezoid: width * average_height
        width = fprs_sorted[i] - fprs_sorted[i-1]
        avg_height = (tprs_sorted[i] + tprs_sorted[i-1]) / 2
        auc += width * avg_height
        
    return auc

# Calculate AUC for our example
auc_score = calculate_auc(fpr_points, tpr_points)
print(f"AUC from scratch: {auc_score:.4f}")

6. Full Script and Verification with Scikit-Learn

As always, it's good practice to verify our work against a standard library implementation. The following video shows how scikit-learn is used to achieve the same result.

Now, let's put everything into one script and compare our results with scikit-learn.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve, roc_auc_score

# --- Our From-Scratch Functions (from above) ---

def calculate_roc_points(y_true, y_pred_probs):
    thresholds = np.linspace(0, 1, 101)
    fprs, tprs = [], []
    for threshold in thresholds:
        y_pred = (y_pred_probs >= threshold).astype(int)
        tp = np.sum((y_true == 1) & (y_pred == 1))
        tn = np.sum((y_true == 0) & (y_pred == 0))
        fp = np.sum((y_true == 0) & (y_pred == 1))
        fn = np.sum((y_true == 1) & (y_pred == 0))
        tpr = tp / (tp + fn + 1e-8)
        fpr = fp / (fp + tn + 1e-8)
        tprs.append(tpr); fprs.append(fpr)
    return np.array(fprs)[::-1], np.array(tprs)[::-1]

def calculate_auc(fprs, tprs):
    sorted_indices = np.argsort(fprs)
    fprs_sorted, tprs_sorted = fprs[sorted_indices], tprs[sorted_indices]
    auc = 0.0
    for i in range(1, len(fprs_sorted)):
        auc += (fprs_sorted[i] - fprs_sorted[i-1]) * (tprs_sorted[i] + tprs_sorted[i-1]) / 2
    return auc

# --- Sample Data ---
np.random.seed(42)
y_true = np.concatenate([np.zeros(50), np.ones(50)])
y_pred_probs = np.concatenate([np.random.rand(50) * 0.4, np.random.rand(50) * 0.6 + 0.4])

# --- From-Scratch Calculation ---
fpr_scratch, tpr_scratch = calculate_roc_points(y_true, y_pred_probs)
auc_scratch = calculate_auc(fpr_scratch, tpr_scratch)

# --- Scikit-Learn Calculation for Verification ---
fpr_sk, tpr_sk, _ = roc_curve(y_true, y_pred_probs)
auc_sk = roc_auc_score(y_true, y_pred_probs)

print(f"AUC from scratch: {auc_scratch:.4f}")
print(f"AUC from scikit-learn: {auc_sk:.4f}")

# --- Plotting Both for Comparison ---
plt.figure(figsize=(10, 8))
plt.plot(fpr_scratch, tpr_scratch, 'b-o', label=f'Scratch Implementation (AUC = {auc_scratch:.2f})')
plt.plot(fpr_sk, tpr_sk, 'r--', label=f'Scikit-Learn (AUC = {auc_sk:.2f})')
plt.plot([0, 1], [0, 1], linestyle=':', color='gray', label='Random Guess')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve: Scratch vs. Scikit-Learn')
plt.legend()
plt.grid(True)
plt.show()

You'll notice the curves are nearly identical, and the AUC scores are very close. The minor differences come from scikit-learn's more efficient way of choosing thresholds (it only uses the unique probability values in your data as thresholds) versus our fixed grid of 101 thresholds.

Conclusion

Congratulations! You've successfully implemented one of the most powerful diagnostic tools in machine learning from the ground up.

Key Takeaways:

• An ROC curve visualizes the trade-off between the True Positive Rate and the False Positive Rate across all classification thresholds.
• AUC provides a single, aggregate measure of a classifier's performance, independent of any specific threshold.
• The core mechanic of building an ROC curve is to iterate through thresholds, classify your data, compute TP/FP/TN/FN, and calculate the (FPR, TPR) point.
• The area under this curve of discrete points can be accurately approximated using the trapezoidal rule.

Preview of the Next Lesson:
We're continuing our exploration of visual diagnostics. While the ROC curve helps evaluate a final model's performance, how do we diagnose what's happening during training? In the next lesson, you will implement learning curve visualization to diagnose underfitting and overfitting, another indispensable tool for any machine learning practitioner.