Missing Data: Deletion & Simple Imputation

The Landscape of Missing Data Solutions

Before we dive into code, let's revisit the high-level map of our options. As we discussed, handling missing values primarily involves two main branches of techniques: Deletions and Imputations.

We will start with the most straightforward approach: deletion.

1. Deletion: The Simplest, Riskiest Strategy

Deletion involves removing the data that is missing. This can mean dropping entire rows (listwise deletion) or entire columns.

When to Consider Deletion

Deletion is a viable strategy only under specific circumstances. As a general rule, it's safest when:

• The data is Missing Completely at Random (MCAR). If not, you risk introducing bias by removing a non-random subset of your data.
• The proportion of missing data is small (e.g., less than 5% of the total dataset).
• You have a very large dataset, and the loss of a few records will not significantly impact the model's performance or statistical power.

Deletion in Practice with pandas

The pandas library makes deletion straightforward with the .dropna() method. The codebasics channel has an excellent video demonstrating this.

Here's a quick summary of the key dropna() parameters:

• axis=0: Drop rows with missing values (default).
• axis=1: Drop columns with missing values.
• how='any': Drop the row/column if any value is missing (default).
• how='all': Drop the row/column only if all values are missing.
• thresh=N: Keep rows/columns that have at least N non-missing values.

While simple, remember that deletion is often a last resort because of the information loss it entails. Imputation is usually a preferred alternative.

2. Simple Imputation: An Educated Guess

Imputation is the process of filling in missing values with substitutes. Simple, or univariate, imputation uses statistical summaries of the data from the column with the missing values itself.

However, this approach comes with significant trade-offs. By filling missing values with a single number, you can distort the variable's natural distribution and its relationship with other variables.

Let's explore the most common simple imputation methods and how to implement them.

Choosing Your Imputation Strategy

The right choice depends on the type of data and its characteristics:

• Mean Imputation: Best for numerical columns that are not skewed and don't have significant outliers. Since you have a background in statistics, you'll know that the mean is sensitive to extreme values.
• Median Imputation: The preferred choice for numerical columns that are skewed or contain outliers. The median is a more robust measure of central tendency.
• Mode (Most Frequent) Imputation: The standard choice for categorical (non-numeric) columns.

Imputation in Practice with pandas and scikit-learn

There are two primary ways to perform simple imputation in Python.

A) Using pandas.fillna()

This is the most direct method for quick data manipulation within a DataFrame. The codebasics video also provides a great walkthrough of this method.

B) Using scikit-learn.SimpleImputer

For machine learning projects, SimpleImputer is the recommended approach. It's designed to be part of a scikit-learn pipeline, which helps prevent data leakage by ensuring the imputation strategy is "learned" on the training data only and then applied consistently to any new data (like a test set). Given your goal of building ML models, this is the tool you'll use most often.

The following video from Ryan & Matt Data Science is a comprehensive guide to SimpleImputer.

A Practical Example with SimpleImputer

Let's tie this all together. Imagine we have a small dataset about employees.

import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer




# Sample data with missing values
data = {'age': [25, 32, 45, np.nan, 38, 65],
        'salary': [50000, 75000, np.nan, 62000, 88000, 95000],
        'department': ['Sales', 'IT', 'Sales', 'HR', np.nan, 'IT']}
df = pd.DataFrame(data)

print("Original DataFrame:")
print(df)
print("\nMissing values count:")
print(df.isnull().sum())

We have one missing value in each column. Based on our rules:

• age: Let's assume the distribution might be skewed by the value 65, so median is a safe choice.
• salary: Let's assume salary is roughly normally distributed, making mean a reasonable choice.
• department: This is categorical, so we must use the mode (most frequent).

Here is how you would apply these strategies using SimpleImputer. In a real project, you would often combine these steps using a ColumnTransformer, as briefly shown at the end of the SimpleImputer video, but for clarity, we'll do them one by one.

# 1. Impute 'age' with the median
imputer_age = SimpleImputer(strategy='median')
df['age'] = imputer_age.fit_transform(df[['age']])




# 2. Impute 'salary' with the mean
imputer_salary = SimpleImputer(strategy='mean')
df['salary'] = imputer_salary.fit_transform(df[['salary']])




# 3. Impute 'department' with the mode
imputer_dept = SimpleImputer(strategy='most_frequent')
df['department'] = imputer_dept.fit_transform(df[['department']])

print("\nImputed DataFrame:")
print(df)
print("\nMissing values after imputation:")
print(df.isnull().sum())

Notice how fit_transform is used. The .fit() step calculates the statistic (mean, median, or mode) from the available data, and the .transform() step applies it to fill the missing values. This two-step process is fundamental to how scikit-learn estimators work and is key to preventing data leakage when you split your data into training and testing sets.

Conclusion

In this lesson, you have learned the practical skills to implement foundational missing data techniques. You now know not only what these methods are but also when and how to apply them using the standard tools of the trade.

Here are the key takeaways:

• Deletion (.dropna()): A quick but risky method best reserved for small amounts of MCAR data in large datasets.
• Simple Imputation (.fillna(), SimpleImputer): Preserves your data but can distort distributions. The choice between mean, median, and mode is critical and depends on data type and characteristics.
• SimpleImputer for ML: For any machine learning task, scikit-learn's SimpleImputer is the preferred tool because it integrates seamlessly into preprocessing pipelines and helps prevent data leakage.

You are now well-equipped to handle one of the most common data quality issues. However, our data cleaning process isn't complete. Datasets often contain another type of problematic data: outliers. In our next lesson, we will explore how to detect outliers using statistical methods like the Interquartile Range (IQR) and Z-score.