Welcome to the next step in our data journey! In the last lesson, we learned to identify the main types of data we'll be working with: numerical, categorical, and text. Now that we know what our data is made of, it's time to start cleaning it up. In the real world, data is almost never perfect. One of the most common problems you'll encounter is missing information.
Today's goal is to learn how to identify missing values in a dataset and explain why they cause problems for machine learning models. Think of your dataset as a puzzle. If some pieces are missing, you can't see the full picture. Our job today is to learn how to spot those missing pieces and understand why they can prevent our machine learning model from working correctly.
What Are Missing Values?
In the simplest terms, missing values are just empty spots or gaps in your data. Imagine a spreadsheet where some cells are blank. That's missing data. In data science, these gaps are often represented by special markers like NaN (which stands for "Not a Number"), NA, NULL, or sometimes even just a blank space.
Since you have a background in software engineering, you can think of a missing value as being very similar to a null or None type in a programming language. It's a placeholder for a value that doesn't exist.
To understand how these gaps appear and how they are represented, let's turn to a short reading.
Top 4 Techniques for Handling Missing Values in Machine Learning
This article from the Paperspace blog provides a clear and concise introduction to missing values.
Please read the section What are missing values? Why do they exist?. Pay close attention to the different ways missing values can be represented (like 'NA', 'NaN', 'NULL') and the list of common reasons why data goes missing.
As the article mentioned, data can go missing for many reasons, from a sensor failing to record a temperature, to a person simply skipping a question on a survey.
Why Are Missing Values a Problem?
So, we have some empty spots in our data. Why is this such a big deal for machine learning? There are two main reasons.
1. Most ML Models Don't Understand "Empty"
Machine learning models are fundamentally mathematical. They learn by performing calculations on numbers. Just as you can't perform the calculation 5 + [nothing] in your head, a computer can't perform math operations on a missing value. If you try to feed a dataset with NaN values directly into most ML algorithms, you will get an error. The model simply won't know what to do.
2. Missing Values Can Lead to Wrong Conclusions (Bias)
This is a more subtle but incredibly important problem. If the missing values aren't completely random, they can skew your model's understanding of the world. The model might learn incorrect patterns, leading to biased and unfair predictions.
The following video provides excellent, intuitive examples of why missing data is a problem for ML models.
Dealing with Missing Data in Machine Learning
This video from Intuitive Machine Learning breaks down why missing data can be so harmful.
Please watch the section from "Why does this matter?". The examples about predicting house prices and diagnosing heart attacks are great illustrations of how missing data can break a model or introduce dangerous bias.
The video's heart attack example is a perfect illustration of bias. If data is mostly missing for a certain group of people, the model might not learn how to make accurate predictions for them, which could have serious real-world consequences.
This idea of "bias" is so important that it's helpful to understand that not all missing data is the same. Data scientists often group missing values into three main categories based on why the data is missing. Understanding this helps you see how bias can creep in.
- Missing Completely at Random (MCAR): The missingness has no reason. A piece of data is missing due to a completely random event, like a computer glitch. This is the best-case scenario because the remaining data is still a fair representation of the whole.
- Missing at Random (MAR): The missingness is related to another piece of data we do have. For example, maybe men are less likely to fill out a survey about depression. The missingness in the "depression score" is related to the "gender" column, which we have.
- Missing Not at Random (MNAR): The missingness is related to the value that is missing itself. For example, people with very high incomes might be less likely to report their income. The fact that the "income" value is missing is directly related to how high that income would have been. This is the most dangerous type, as it can strongly bias your model.
This next video gives a very clear, 2-minute explanation of these three types.
3 Main Types of Missing Data | Do THIS Before Handling Missing Values!
This short video from AI For Beginners uses simple examples to explain the three main types of missing data.
Watch the explanations for MCAR, MAR, and MNAR. The examples of a malfunctioning device, doctor visit frequency, and unreported income are very helpful.
Conclusion
Fantastic work today! We've taken our first step into the essential process of data cleaning. You now have the fundamental knowledge to spot one of the most common issues in real-world data and, crucially, to understand the problems it can cause.
Here are the key takeaways:
- Missing values are gaps in a dataset, often represented by
NaN,NULL, or empty cells. - They can occur for many reasons, including equipment failure, data entry errors, or people intentionally not providing information.
- They are a major problem for machine learning because:
- Most algorithms require complete numerical data to perform calculations.
- They can introduce bias, causing the model to learn incorrect patterns and make unfair or inaccurate predictions.
- Understanding the reason data is missing (MCAR, MAR, or MNAR) helps us see how it might be biasing our results.
In our next lesson, we'll move from identifying the problem to solving it. You'll learn some of the most common and simple strategies for handling missing values, such as removing them or filling in the gaps.