Handling Imbalanced Datasets

1. The Problem with Imbalanced Data

Imagine building a model to detect a rare disease that affects only 1% of the population. A naive model could achieve 99% accuracy by simply predicting that no one has the disease. While technically accurate, this model is completely useless because it fails to identify any of the positive cases we actually care about.

This highlights two core issues with imbalanced datasets:

1. Model Bias: Most machine learning algorithms are optimized to minimize overall error. With imbalanced data, they become biased toward the majority class because doing so quickly reduces the total error. They don't get enough exposure to the minority class to learn its patterns.
2. Misleading Metrics: Accuracy becomes a deceptive measure of performance. We need more nuanced metrics that tell us how well the model performs on the minority class.

To start, let's watch a short video that clearly explains what class imbalance is and why accuracy is not the right metric to use.

As the video mentioned, we need to look at metrics like the F1-score, precision, recall, and the confusion matrix to get a true picture of a model's performance on an imbalanced problem.

2. Technique 1: Resampling with SMOTE

If the problem is that we don't have enough examples of the minority class, a logical solution is to get more. This is the core idea behind oversampling. A naive approach would be to simply duplicate existing minority class samples, but this doesn't provide the model with any new information.

A more intelligent approach is the Synthetic Minority Over-sampling Technique (SMOTE). Instead of duplicating data, SMOTE creates new, synthetic data points that are plausible examples of the minority class.

How does it work? For each minority class sample, SMOTE:

1. Finds its k nearest neighbors (typically k=5) that are also in the minority class.
2. Randomly selects one of these neighbors.
3. Creates a new, synthetic data point at a random position on the line segment connecting the original sample and its selected neighbor.

Let's dive into a more detailed explanation of the SMOTE algorithm and its implementation.

A Critical Warning: Data Leakage

When using any resampling technique, there's a crucial pitfall to avoid: data leakage. You must never apply resampling to your entire dataset before splitting it into training and testing sets. If you do, you are "leaking" information from your test set into your training process, as the synthetic samples in your training set might be based on information from samples that will end up in your test set. This will give you an overly optimistic performance estimate, and your model will perform poorly in the real world.

The correct approach is to apply resampling only to the training data.

The best way to ensure this is done correctly, especially during cross-validation, is to use a scikit-learn Pipeline from the imbalanced-learn library. This pipeline will apply the SMOTE transformation to each training fold within the cross-validation loop, but will leave the validation fold untouched.

The following resources drive this critical point home.

3. Technique 2: Class Weighting

Resampling changes the data. An alternative approach is to change the learning algorithm's behavior itself. This is known as cost-sensitive learning, and the most common way to implement it is through class weighting.

The idea is to modify the model's loss function to penalize misclassifications of the minority class more heavily than misclassifications of the majority class. By assigning a higher weight to the minority class, we force the model to pay more attention to it during training.

Most scikit-learn classifiers, including LogisticRegression and RandomForestClassifier, have a class_weight parameter.

• You can set class_weight='balanced', which automatically adjusts weights inversely proportional to class frequencies.
• You can provide a dictionary of custom weights, e.g., class_weight={0: 1, 1: 10}, to give 10 times more importance to class 1.

The following article gives an excellent conceptual and mathematical breakdown of how class weighting works, particularly in the context of logistic regression.

4. Comparing the Techniques in Practice

Now let's see these techniques in action on the same problem. The video from Ryan & Matt Data Science that we started with provides a complete, end-to-end comparison.

5. Hands-on Practice

It's time to apply what you've learned. The following code sets up an imbalanced classification problem. Your task is to train and evaluate three different models: a baseline, one using SMOTE, and one using class weighting.

import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from imblearn.pipeline import make_pipeline
from imblearn.over_sampling import SMOTE
import seaborn as sns
import matplotlib.pyplot as plt




# 1. Generate imbalanced data
X, y = make_classification(n_samples=1000, n_features=20, n_informative=2,
                           n_redundant=10, n_clusters_per_class=1, weights=[0.98],
                           flip_y=0, random_state=42)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)




# Function to print results
def evaluate_model(y_test, y_pred, model_name):
    print(f"--- {model_name} ---")
    print(classification_report(y_test, y_pred, zero_division=0))
    cm = confusion_matrix(y_test, y_pred)
    sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
    plt.title(f'Confusion Matrix - {model_name}')
    plt.xlabel('Predicted')
    plt.ylabel('Actual')
    plt.show()




# --- YOUR CODE GOES HERE ---




# 2. Baseline Model
# Train a standard Logistic Regression model
print("Training Baseline Model...")
baseline_model = LogisticRegression(solver='liblinear', random_state=42)
baseline_model.fit(X_train, y_train)
y_pred_baseline = baseline_model.predict(X_test)
evaluate_model(y_test, y_pred_baseline, "Baseline Logistic Regression")





# 3. SMOTE Model
# Use an imblearn pipeline to combine SMOTE and Logistic Regression
print("\nTraining SMOTE Model...")
smote_pipeline = make_pipeline(SMOTE(random_state=42), 
                               LogisticRegression(solver='liblinear', random_state=42))
smote_pipeline.fit(X_train, y_train)
y_pred_smote = smote_pipeline.predict(X_test)
evaluate_model(y_test, y_pred_smote, "SMOTE + Logistic Regression")





# 4. Class Weighting Model
# Train a Logistic Regression model with class_weight='balanced'
print("\nTraining Class Weighting Model...")
weighted_model = LogisticRegression(solver='liblinear', class_weight='balanced', random_state=42)
weighted_model.fit(X_train, y_train)
y_pred_weighted = weighted_model.predict(X_test)
evaluate_model(y_test, y_pred_weighted, "Weighted Logistic Regression")

Run this code and carefully compare the classification_report and confusion matrix for each model. Notice how the recall for the minority class (class 1) improves dramatically with SMOTE and class weighting, even if precision sometimes decreases. This trade-off is central to handling imbalanced problems.

Conclusion

In this lesson, you've learned to tackle one of the most common and challenging problems in classification: imbalanced datasets. You now have two powerful, complementary strategies in your toolkit.

Key Takeaways:

• Class imbalance leads to biased models and makes accuracy a poor evaluation metric. Focus on the F1-score, precision, recall, and the confusion matrix.
• Resampling (SMOTE) balances the dataset by creating new synthetic examples of the minority class.
• Cost-Sensitive Learning (Class Weighting) modifies the model's loss function to penalize mistakes on the minority class more severely.
• To prevent data leakage, resampling techniques must only be applied to the training data, ideally using a pipeline that integrates them into your cross-validation workflow.
• There is often a trade-off between correctly identifying more minority cases (higher recall) and incorrectly labeling majority cases (lower precision).

In our next lesson, we will cover another critical preprocessing step: feature selection. You'll learn how to identify and remove irrelevant or redundant features to build simpler, faster, and often more robust models.