Welcome back! In our previous lesson, we focused on getting our categorical data ready for a machine learning model by converting text labels into numbers. Now, we'll turn our attention to the other major data type: numerical data.
Today's lesson addresses what to do when your numerical features are on wildly different scales. We'll explore a technique called normalization, which rescales features to a common range. By the end of our session, you'll be able to normalize numerical data and explain exactly why this simple step is so important for helping many machine learning models learn effectively.
Why Do We Need to Normalize Data?
Imagine you're making a smoothie with strawberries and bananas. If you use one whole banana but only one tiny strawberry, your smoothie is just going to taste like banana. The banana's flavor completely overpowers the strawberry's. To get a balanced flavor, you need to use them in similar proportions.
Machine learning models can face a similar problem. Let's say we have a dataset with two features: a person's Age (e.g., 25 to 65) and their Salary (e.g., $40,000 to $150,000). The Salary numbers are thousands of times larger than the Age numbers.
For some ML models, this huge difference in scale is a problem. The model might incorrectly assume that Salary is thousands of times more important than Age, simply because its numbers are bigger. This is like the banana overpowering the strawberry.
To fix this, we need to bring all our numerical features to a common scale. This process is called feature scaling.
Data Science Blog - 365 DATA SCIENCE
This article from the 365 Data Science blog introduces the concept perfectly.
Read the first part of the article, starting from the smoothie analogy we just discussed. Continue reading through the explanation of why features with different scales can be a problem for certain algorithms.
As the article explains, two main types of models are sensitive to feature scales:
- Models that use distance: Algorithms like K-Nearest Neighbors (KNN) work by measuring the distance between data points. If one feature has a much larger scale, it will dominate these distance calculations.
- Models that use gradient descent: Algorithms like linear regression and neural networks learn by taking small steps to find the best solution. If features have different scales, the "steps" it takes will be uneven and wobbly, making it take much longer to find the solution.
Standardization vs Normalization Clearly Explained!
This video from Normalized Nerd gives a great visual explanation for both of these points.
First, watch the segment that explains why scaling is important for distance-based algorithms. Then, watch the explanation of how different scales can slow down gradient descent.
By putting all features on the same scale, we ensure they all get a fair chance to contribute to the model's prediction.
How to Normalize: Min-Max Scaling
The most common way to normalize data is a technique called Min-Max Scaling. The idea is simple: we squish or stretch the range of each feature so that all values fit neatly between 0 and 1.
Here’s how it works:
- The smallest value in the original feature becomes 0.
- The largest value in the original feature becomes 1.
- Every other value is rescaled to a number between 0 and 1.
This is done using a straightforward formula.
As shown in the image, the formula is:
Where:
- is the original value.
- is the minimum value in that feature's column.
- is the maximum value in that feature's column.
- is the new, normalized value between 0 and 1.
Let's look at the 'Age' column in the image. The minimum age is 27 and the maximum is 48.
- For the employee with age 27, the formula gives: .
- For the employee with age 48, the formula gives: .
- For an employee with age 35, it's: .
The same logic applies to the Salary column, using its own min and max values. After normalization, both Age and Salary are on the same 0-to-1 scale.
Demystifying Data Normalization in Machine Learning
This article provides another clear breakdown of Min-Max scaling.
Start by reading the brief introduction to what normalization is. Then, jump down to the section on Min-Max Scaling for a reinforcement of the formula and its purpose.
Important Considerations
While powerful, it's good to know two key things about normalization:
- Not all models need it. As we've discussed, normalization is crucial for distance-based and gradient-descent-based models. However, tree-based models (like Decision Trees and Random Forests) are not sensitive to feature scales and don't require normalization.
Why we perform feature normalization in ML
This video by DataMListic gives a quick rundown of which common models do and don't require normalization.
Watch this final segment of the video from the list of models.
- Sensitivity to outliers. Min-Max scaling is very sensitive to outliers (extremely high or low values). If you have a
Salarycolumn with values mostly under $100,000, but one person earns $1,000,000, that one outlier will become the 'max' value. As a result, all other salaries will be squished into a very tiny range (e.g., 0 to 0.1), which isn't very helpful. This is why we handled outliers in a previous module!
Conclusion
Well done! You now have another essential data preparation tool in your toolkit. We've taken our raw numerical data and made it fair and balanced for our ML model.
Here are the key takeaways from this lesson:
- Normalization is the process of scaling numerical features to a common range, usually 0 to 1.
- It's important because features with vastly different scales can cause some ML models to perform poorly or train slowly. This is especially true for models that calculate distances or use gradient descent.
- Min-Max Scaling is a common normalization technique that rescales data using the formula: .
We have now prepared both our categorical and numerical features. The final step before we can train a model is to divide our data. In the next lesson, we will learn how to split a dataset into training, validation, and test sets and understand the crucial role each part plays in building a reliable model.