Balancing Imbalanced Software Quality Datasets with Over/Under-sampling

Introduction

In our previous lesson, we successfully assembled features from various sources into a single, model-ready feature matrix. This was a crucial step, preparing our data for the modeling phase. You now have a model_ready_df DataFrame, which is the direct input for the algorithms we'll explore.

Today, we'll address a very common and critical challenge in software quality datasets: class imbalance. Think about flaky tests or critical bugs—they are typically rare events. If we train a model on data where 99% of tests are stable and only 1% are flaky, the model can achieve 99% accuracy by simply guessing "stable" every time. This is accurate, but completely useless for our goal.

Our learning outcome for this lesson is to handle class imbalance in software quality datasets using over/under-sampling techniques (e.g., SMOTE). By the end of this lesson, you will understand why imbalance is a problem and how to use a powerful technique called the Synthetic Minority Oversampling TEchnique (SMOTE) to create a more balanced and effective training set.

The Problem with Imbalance

Machine learning models are often optimized for overall accuracy. In an imbalanced dataset, a model can achieve high accuracy by developing a bias towards the majority class and largely ignoring the minority class. This is a significant issue in software quality, where the minority class (e.g., flaky tests, security vulnerabilities, high-impact bugs) is almost always the one we're most interested in identifying.

To see this in action, we need to look beyond accuracy. Metrics like precision and recall are far more insightful:

• Precision: Of all the tests we predicted as "flaky," how many were actually flaky? (Measures the cost of false positives).
• Recall: Of all the actual flaky tests, how many did our model correctly identify? (Measures the cost of false negatives).

For detecting rare but critical events, recall is often the most important metric. We want to find as many of the flaky tests as possible, even if it means we occasionally misclassify a stable test.

Let's demonstrate this with code. We'll create a synthetic dataset that's highly imbalanced (99% class 0, 1% class 1) and see how a standard logistic regression model performs.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, accuracy_score
import pandas as pd
from collections import Counter




# 1. Create a highly imbalanced dataset
X, y = make_classification(n_samples=10000, n_features=10, n_informative=5,
                           n_redundant=0, n_classes=2, n_clusters_per_class=1,
                           weights=[0.99, 0.01], flip_y=0, random_state=42)

print(f"Original dataset shape: {Counter(y)}")




# 2. Split data and train a simple model
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)
model = LogisticRegression(solver='liblinear')
model.fit(X_train, y_train)




# 3. Evaluate the model
y_pred = model.predict(X_test)
print(f"\nAccuracy: {accuracy_score(y_test, y_pred):.2f}")
print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=['Stable Test (0)', 'Flaky Test (1)']))

Notice the results. The accuracy is around 99%, which looks great. But look at the classification report for the 'Flaky Test (1)' class. The recall is likely very low (often below 0.20), meaning the model failed to identify most of the actual flaky tests. This is the problem we need to solve.

SMOTE: Synthesizing New Data

How can we fix this? Simply duplicating existing minority samples (oversampling) can lead to overfitting. A more sophisticated approach is the Synthetic Minority Oversampling Technique (SMOTE).

Instead of copying data, SMOTE creates new, synthetic samples. It does this by:

1. Selecting a minority class instance at random.
2. Finding its k-nearest neighbors (also in the minority class).
3. Choosing one of those neighbors and creating a new synthetic data point somewhere on the line connecting the original point and its chosen neighbor.

This populates the feature space around the minority class, helping the model learn a more robust decision boundary.

To understand the mechanics in more detail, let's turn to a foundational guide on the topic.

As the article notes, SMOTE is often combined with random undersampling, which reduces the number of examples in the majority class. This combination can be very effective.

Practical Implementation with imbalanced-learn

The best way to implement SMOTE is with the imbalanced-learn library, a scikit-learn-contrib project designed specifically for these problems.

First, you'll need to install it:
pip install -U imbalanced-learn

Now, let's see how to apply it. A common mistake is to apply SMOTE to the entire dataset before splitting into training and test sets. This is wrong and leads to data leakage. Your model will "see" information from the test set during training (via the synthetic samples), resulting in overly optimistic performance metrics.

The correct approach is to apply resampling techniques ONLY on the training data. The imbalanced-learn library provides a Pipeline object that makes this easy and safe, especially when using cross-validation.

Let's apply this correct pipeline approach to our previous example and see the difference.

# Continuing from the previous code block...
from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE
from sklearn.model_selection import cross_val_score, RepeatedStratifiedKFold




# 1. Define the SMOTE-enhanced pipeline
steps = [('smote', SMOTE(random_state=42)), 
         ('model', LogisticRegression(solver='liblinear'))]
pipeline = Pipeline(steps=steps)




# 2. Evaluate the pipeline using cross-validation
# Note: We use the full dataset (X, y) here because the pipeline handles
# the split and resampling within each fold of the cross-validation.
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=42)
scores = cross_val_score(pipeline, X, y, scoring='recall_macro', cv=cv, n_jobs=-1)




# Report performance
import numpy as np
print(f"Mean Macro-Averaged Recall: {np.mean(scores):.2f}")




# For a more detailed report, let's fit on the train set and predict on the test set
# The pipeline ensures SMOTE is only applied to X_train, y_train
pipeline.fit(X_train, y_train)
y_pred_smote = pipeline.predict(X_test)

print("\nClassification Report AFTER SMOTE:")
print(classification_report(y_test, y_pred_smote, target_names=['Stable Test (0)', 'Flaky Test (1)']))

Compare this new classification report to the one from our first attempt. You should see a dramatic improvement in the recall for the 'Flaky Test (1)' class. While precision might decrease slightly (a common trade-off), the model is now much better at its primary job: finding the rare, positive cases. This is the power of handling class imbalance correctly.

SMOTE Variants for Advanced Cases

Vanilla SMOTE is a great starting point, but it's not perfect. It can sometimes create noisy samples if the minority class is very sparse or overlaps significantly with the majority class. To address this, several variants have been developed.

• Borderline-SMOTE: Focuses on generating samples near the decision boundary, where the model is most likely to be confused.
• ADASYN (Adaptive Synthetic Sampling): Generates more synthetic samples for minority examples that are harder to learn (i.e., those in sparser regions or surrounded by majority class examples).
• SMOTE-ENN / SMOTE-Tomek: Hybrid methods that combine SMOTE's oversampling with undersampling/cleaning techniques to remove noisy or overlapping points.

You don't need to master all of these now, but it's important to know they exist. The best choice often depends on your specific dataset.

Conclusion

You've now learned how to tackle one of the most common and deceptive problems in applied machine learning for software engineering. Properly handling class imbalance is often the difference between a model that looks good on paper and one that provides real-world value.

Key Takeaways:

• Class imbalance leads to models that are biased towards the majority class, resulting in poor performance (especially low recall) on the minority class we care about.
• Accuracy is a misleading metric for imbalanced datasets; use precision, recall, and the F1-score instead.
• SMOTE is a powerful technique that balances datasets by creating synthetic minority class samples.
• Crucially, resampling must only be applied to the training data. Using imblearn.pipeline.Pipeline is the best practice to prevent data leakage.
• Advanced SMOTE variants exist to handle more complex cases like noisy data or overlapping classes.

Preview of the Next Lesson:
With our feature matrix assembled and our strategy for handling class imbalance in place, we are fully prepared to train and evaluate our first classifier. In the next lesson, we will train a binary classifier (like a RandomForest or XGBoost) to detect flaky tests using the balanced dataset we've learned to create today.