Loan Data Preprocessing Strategy

1. A Strategic Approach to Missing Values

The first step in data preparation is to identify and handle missing data. Our goal is not simply to fill every blank cell, but to make reasoned decisions based on the business context and the nature of the data.

First, let's watch a brief segment of a project walkthrough to see how missing data is identified and handled in a practical credit risk scenario.

As the video shows, sometimes the best strategy is deletion. For saving_accounts and checking_accounts, the absence of data is highly significant. A bank would likely not approve a loan for an applicant whose account information is unavailable. Trying to impute—or guess—this information could introduce serious noise and risk. In this case, dropping the records is a defensible decision that mimics a real-world business rule.

However, we can't drop records for every missing value, especially if it means losing a large portion of our dataset. For other features, we need a more nuanced imputation strategy. A professional approach involves creating a tiered strategy based on the percentage of missing data for each feature.

Based on this reading, here is the professional strategy we will adopt:

1. Quantify the Problem: First, we'll calculate the percentage of missing values for every feature.
2. Separate Feature Types: We'll divide our features into numerical and categorical groups, as they require different imputation methods.
3. Apply a Tiered Strategy:
   • Low Missing Rate (<5%): For these features, simple and fast imputation is effective.
     • Numerical: Impute with the median. We prefer the median over the mean because financial data (like income or loan amounts) is often skewed, and the median is robust to extreme values.
     • Categorical: Impute with the mode (the most frequent value).
   • Moderate Missing Rate (5% - 30%): Here, a simple imputation might introduce bias. A more sophisticated approach is better.
     • Numerical: Use model-based imputation. For example, we can train a small regression model (like LightGBM, as shown in the article) to predict the missing values based on other features in the dataset. This is more accurate than just using the median.
     • Categorical: Can also use predictive imputation or fill with a special category like "Unknown", which the model can learn from.
   • High Missing Rate (>30%): A feature missing this much data is often a candidate for removal. Imputing such a large amount of data is unreliable. However, before dropping it, we must check its correlation with the target variable. If it's highly predictive, we might retain it and use an advanced imputation method or treat its absence as a separate category.

This structured approach ensures that our choices are efficient, justified, and tailored to the specific characteristics of each feature.

2. Tackling Outliers in Financial Data

Outliers are extreme values that lie far from the rest of the data points. In credit risk data, they can be genuine (e.g., a CEO's salary) or errors (e.g., a typo in age). If left untreated, they can skew statistical calculations and mislead our models.

The article we just reviewed also contains an excellent section on a domain-specific strategy for handling outliers.

The key insight here is that not all outliers are treated the same. The handling strategy should depend on the feature's meaning.

• Detection: A standard method for detecting outliers is the Interquartile Range (IQR) method. Any value that falls below Q1 - 1.5 * IQR or above Q3 + 1.5 * IQR is flagged as an outlier.

• Handling Strategies:

  • Capping (Winsorization): For key financial features like AMT_INCOME_TOTAL or AMT_CREDIT, we don't want to remove high values, as high income or a large loan is important information. Instead, we "cap" the outliers by setting any value above the 99th percentile to the 99th percentile value (and similarly for the 1st percentile). This reduces the outlier's skewing effect while retaining the information that the value is at the extreme high end.

*   **Log Transformation:** For highly skewed numerical data, like `DAYS_EMPLOYED`, applying a logarithmic transformation (`np.log1p`) can compress the range of the feature. This makes the distribution more normal, reduces the influence of extreme values, and often helps linear models find better patterns.

*   **Removal:** This is a last resort. We would only remove an outlier if we have strong evidence that it is a data entry error (e.g., an age of 200) and not a genuine, albeit extreme, value.

This context-aware approach is a hallmark of professional data science practice in the financial domain.

3. Addressing Class Imbalance

We've touched on this already: our dataset will have many more "good" (non-default) loans than "bad" (default) ones. This is a problem because most machine learning algorithms are designed to maximize overall accuracy, so they can become biased. A naive model might achieve 95% accuracy by simply predicting "good" for everyone, completely failing to identify the high-risk applicants we care about most.

To develop our strategy, let's first get a clear, conceptual understanding of the problem and the main data-level solutions.

As the video explains, we have several strategies available at the data-sampling level.

• Random Undersampling:

  • What it is: Deleting random samples from the majority class ("good" loans) until the classes are balanced.
  • Pros: Can make training faster on very large datasets.
  • Cons: High risk of losing important information. We might discard examples of "good" customers who are subtly different from others. This is generally not the preferred approach unless the dataset is massive.

• Random Oversampling:

  • What it is: Duplicating random samples from the minority class ("bad" loans) until the classes are balanced.
  • Pros: No information is lost.
  • Cons: Since we are just copying data, the model can easily overfit to these repeated examples, leading to poor performance on new, unseen data.

• SMOTE (Synthetic Minority Oversampling Technique):

  • What it is: A more intelligent form of oversampling. Instead of duplicating existing data, SMOTE creates new, synthetic minority class samples. It does this by selecting a minority data point, finding its nearest neighbors (also in the minority class), and creating a new point along the line segment connecting them.
  • Pros: Creates a richer, more diverse dataset for the model to learn from without simply duplicating information. This helps prevent overfitting and often leads to better generalization.
  • Cons: Can create noise if the minority class samples are very sparse or overlapping with the majority class. However, it is one of the most effective and widely used techniques.

For our project, SMOTE is the recommended strategy. It directly addresses the weakness of simple oversampling and is a standard, robust technique for credit risk modeling. We will plan to integrate SMOTE into our modeling pipeline in the next module.

Conclusion

In this lesson, we have moved from high-level planning to defining a concrete, professional-grade data preparation strategy. We recognized that data cleaning is not a rote exercise but a series of justified decisions that enhance the final model's quality.

Key Takeaways:

• Missing Values Strategy: Employ a tiered approach. Delete rows only with strong business justification. For imputation, use simple methods (median/mode) for low missing rates and advanced, model-based methods for moderate rates. Drop features with high missing rates unless they are highly predictive.
• Outlier Handling Strategy: Base your approach on the feature's business context. Use capping (Winsorization) for financial features to limit their influence without losing information, and use log transformations to handle skewed distributions.
• Class Imbalance Strategy: To combat model bias towards the majority class, we will use a data-level resampling technique. We have chosen SMOTE (Synthetic Minority Oversampling Technique) as it creates new, synthetic data points, which is a more robust approach than simple over- or undersampling.

Next Steps:

With a solid strategy for data preparation in place, we are ready to tackle the more creative side of preprocessing. In the next lesson, we will focus on feature engineering and selection. We'll learn how to create new, powerful predictors from the existing data (e.g., debt-to-income ratio) and select the most impactful features for our model.