Missing Data: Deletion & Imputation

Identifying Missing Data

Before you can handle missing values, you must first find them. In pandas, missing data is most commonly represented by np.nan (Not a Number). The first step in any data cleaning process is to get a clear picture of where and how often these NaN values appear in your DataFrame.

The most direct way to do this is by chaining two methods: .isnull() and .sum().

• df.isnull(): This returns a DataFrame of the same shape as the original, but with boolean values: True where a value is missing and False where it is not.
• .sum(): When applied to this boolean DataFrame, it treats True as 1 and False as 0, effectively counting the number of missing values in each column.

import pandas as pd
import numpy as np




# Sample DataFrame
data = {'product': ['A', 'B', 'C', 'D', 'E'],
        'sales': [100, 120, np.nan, 150, 90],
        'region': ['East', 'West', 'East', np.nan, 'West']}
df = pd.DataFrame(data)




# Count missing values in each column
print(df.isnull().sum())

Output:

product    0
sales      1
region     1
dtype: int64

This simple command immediately tells you which columns contain missing data and how much. It's a fundamental first step you'll perform on virtually every dataset you encounter.

Strategies for Handling Missing Values

Once you've identified the missing data, you have two main strategic paths you can take: Deletion or Imputation. The choice between them is a critical decision that depends on the nature of your data and your analysis goals.

Strategy 1: Deletion

Deletion is the most straightforward strategy: you simply remove the rows or columns that contain missing values.

Dropping Rows (Listwise Deletion)

The most common deletion technique is to drop any row containing at least one missing value. This is done with the dropna() method.

While simple, this method should be used with caution. If many rows have sporadic missing values, you could end up deleting a large portion of your dataset, losing valuable information.

The dropna() method is highly configurable. The video below provides a clear, practical guide to using its key arguments like axis, how, and subset to perform deletion more strategically.

Dropping Columns

You might also choose to drop an entire column if it has a very high percentage of missing values and is not essential for your analysis. You can do this with df.dropna(axis=1). For example, if a "customer_feedback" column is 95% empty, it might be more practical to remove it entirely than to try and impute the values.

Strategy 2: Imputation

Imputation is the process of filling in missing values with substitutes. This approach is often preferred over deletion because it retains all your data points. However, it also introduces artificial data, so the method of imputation must be chosen carefully.

The fillna() method is your primary tool for imputation in pandas. A comprehensive guide is provided in the reading below.

Context-Aware Imputation with groupby()

A more sophisticated approach is to impute values based on subgroups within your data. For example, instead of filling a missing salary with the overall average salary, you could fill it with the average salary for that person's job title. This makes the imputed value much more realistic.

This is achieved by combining groupby(), transform(), and fillna().

A Realistic Workflow: The Stack Overflow Survey

Now let's apply these concepts to a real-world scenario. In data science, you often find that columns you expect to be numeric are loaded as strings (object type in pandas) because they contain non-numeric text. A common reason for this is the presence of custom missing value markers or descriptive strings.

The following video demonstrates a complete mini-project: calculating the average years of coding experience from the Stack Overflow survey data. The process involves:

1. Discovering that a numeric calculation fails because the column is of type object.
2. Investigating the column's unique values to find non-numeric strings like 'Less than 1 year'.
3. Using replace() to convert these strings to numerical equivalents.
4. Casting the cleaned column to a numeric type (float).
5. Finally, calculating the mean.

This is a perfect example of the iterative, problem-solving nature of data cleaning.

Conclusion

In this lesson, we tackled the essential task of handling missing data, a critical step in any data science project. You've learned the two primary strategies and the pandas tools to implement them.

Here are your key takeaways:

• Identification is the first step: Always start by using df.isnull().sum() to quantify the extent of missing data in your dataset.
• Deletion is a trade-off: dropna() is a quick way to remove incomplete data, but it risks significant information loss. Use it strategically, for example, by using the subset argument to target critical columns.
• Imputation preserves data: fillna() allows you to replace missing values, preserving your sample size. Simple statistical measures (mean, median, mode) are a good starting point.
• Context matters: For more accurate results, consider context-aware imputation, such as filling missing values with the mean of a specific subgroup using groupby().transform().
• Real data is messy: Be prepared to use methods like .replace() to standardize custom missing value indicators before applying deletion or imputation techniques.

In our next lesson, we will move on to another fundamental data preparation task: merging multiple datasets. You'll learn how to combine data from different sources into a single, unified DataFrame, which is essential when the information you need is spread across multiple files or tables.