Addressing Class Imbalance in Machine Learning

1. Why Imbalanced Data Is a Problem for Training

Most classification algorithms are designed to maximize overall accuracy. When one class overwhelmingly outnumbers another (e.g., 99% non-fraudulent vs. 1% fraudulent transactions), a model can achieve high accuracy by simply always predicting the majority class. This leads to a model with no predictive power for the very class we're often most interested in.

Let's watch a short segment that demonstrates this exact problem. A standard decision tree is trained on an imbalanced dataset, and its performance on the minority class is very poor.

Now that we've confirmed the problem, let's explore the solutions. We'll look at two main families of techniques:

1. Model-level methods: Adjusting the algorithm to pay more attention to the minority class.
2. Data-level methods: Changing the data itself to be more balanced.

2. Model-Level Solution: Using Class Weights

The first and often simplest approach is to tell the model that misclassifying the minority class is more "costly" than misclassifying the majority class. We do this by assigning a higher weight to the minority class during training.

The idea is to adjust the model's loss function. By applying a larger penalty for errors made on minority class examples, we force the model to pay more attention to them. A common strategy is to make the weights inversely proportional to the class frequencies.

If class 'A' is 100 times more frequent than class 'B', we might give class 'B' a weight 100 times larger than class 'A'.

Practical Implementation with Scikit-learn

Many scikit-learn classifiers, including LogisticRegression, DecisionTreeClassifier, and RandomForestClassifier, have a class_weight parameter. Setting class_weight='balanced' automatically calculates and applies these inverse-frequency weights for you.

Let's see this in action with a code example.

This is a powerful and computationally efficient method because it doesn't require creating or removing any data points. It simply changes the model's learning objective.

3. Data-Level Solution: SMOTE

Another popular approach is to modify the dataset itself. Instead of having the model adapt to the imbalance, we rebalance the data before training. A naive way to do this is random oversampling—simply duplicating examples from the minority class. However, this can lead to overfitting because the model sees the exact same data points multiple times.

A much smarter technique is SMOTE (Synthetic Minority Over-sampling Technique). Instead of duplicating samples, SMOTE creates new, synthetic samples for the minority class.

Here's how it works:

1. It picks a minority class sample at random.
2. It finds its k nearest neighbors (also in the minority class).
3. It randomly picks one of those neighbors.
4. It creates a new synthetic sample at a random point along the line segment connecting the original sample and its chosen neighbor in the feature space.

This process creates new, plausible examples of the minority class, expanding its decision region and helping the model to generalize better.

The following resource provides a detailed look at SMOTE, including the Python library you'll need to use it.

4. The Golden Rule: Preventing Data Leakage

This is arguably the most important practical concept in this lesson. When you use any resampling technique like SMOTE, you must only apply it to your training data.

Why? Your test set is supposed to be a proxy for new, unseen, real-world data. Real-world data is imbalanced. If you apply SMOTE to your entire dataset before splitting, or to your test set, you are "leaking" information from the training process into your evaluation. The synthetic samples in your test set make the evaluation easier and will give you a misleadingly optimistic sense of your model's performance.

The correct way to integrate resampling into your workflow is to apply it after splitting the data, or even better, to embed it within a cross-validation pipeline. This ensures that the resampling only happens on the data used for training in each fold.

The imbalanced-learn library is designed to work seamlessly with scikit-learn's Pipeline object to make this process safe and robust.

Hands-On Practice

Let's put this all together in a single code block. We'll create an imbalanced dataset, then compare three models:

1. A baseline model with no balancing.
2. A model using class_weight='balanced'.
3. A model using SMOTE inside a pipeline.

import numpy as np
from collections import Counter
from sklearn.datasets import make_classification
from sklearn.model_selection import RepeatedStratifiedKFold, cross_val_score
from sklearn.linear_model import LogisticRegression
from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE




# 1. Create a highly imbalanced dataset
X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, 
                           n_redundant=0, n_classes=2, weights=[0.99], 
                           flip_y=0, random_state=42)

print(f"Original dataset shape: {Counter(y)}")




# Define our evaluation procedure
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)




# 2. Evaluate different strategies
# Strategy 1: No balancing
model_plain = LogisticRegression(solver='liblinear')
scores_plain = cross_val_score(model_plain, X, y, scoring='roc_auc', cv=cv, n_jobs=-1)
print(f"No balancing - Mean ROC AUC: {np.mean(scores_plain):.3f}")




# Strategy 2: Using class weights
model_weighted = LogisticRegression(solver='liblinear', class_weight='balanced')
scores_weighted = cross_val_score(model_weighted, X, y, scoring='roc_auc', cv=cv, n_jobs=-1)
print(f"Class Weights - Mean ROC AUC: {np.mean(scores_weighted):.3f}")




# Strategy 3: Using SMOTE in a pipeline
steps = [('smote', SMOTE()), ('model', LogisticRegression(solver='liblinear'))]
pipeline_smote = Pipeline(steps=steps)
scores_smote = cross_val_score(pipeline_smote, X, y, scoring='roc_auc', cv=cv, n_jobs=-1)
print(f"SMOTE Pipeline - Mean ROC AUC: {np.mean(scores_smote):.3f}")

When you run this code, you'll likely see that both class_weight and SMOTE provide a significant performance lift over the baseline model, as measured by ROC AUC (a metric that is sensitive to performance on both classes). The exact winner can depend on the dataset and model, but both are powerful tools to have.

Conclusion

Congratulations on completing this lesson and the classification module! You now have a solid understanding of not only how to train classifiers but also how to handle one of the most common and tricky real-world data problems: class imbalance.

Key Takeaways:

• Standard models perform poorly on imbalanced data because they are biased toward the majority class.
• Class weights adjust the model's loss function to penalize errors on the minority class more heavily. This is a simple, built-in solution for many classifiers (class_weight='balanced').
• SMOTE is a data-level technique that creates new, synthetic examples of the minority class to create a more balanced training set.
• Crucially, resampling techniques like SMOTE must only be applied to the training data. Using a pipeline from imbalanced-learn is the best practice to prevent data leakage during cross-validation.

Preview of the next lesson:
We have now explored how to build, evaluate, and troubleshoot classification models. But how do we get the best possible performance out of them? The next module, Model Tuning and Selection, will dive into that. We'll start by clarifying the difference between model parameters (which are learned from data, like the coefficients in logistic regression) and hyperparameters (which are set by you before training, like the k in KNN or the class_weight setting). Mastering this distinction is the first step toward systematic model optimization.