Data Rescaling with Normalization

Why Do We Need to Scale Features?

Imagine you have a dataset with two features: a person's age (ranging from 18 to 70) and their income (ranging from $30,000 to $1,000,000). If you use this data directly in certain machine learning algorithms, the income feature, due to its much larger magnitude, could disproportionately influence the outcome. The algorithm might incorrectly interpret the larger numerical values of income as being more important than age, even if that's not the case.

Feature scaling solves this by putting all features on a level playing field. This is particularly important for two main classes of algorithms:

1. Algorithms that use distance metrics: Methods like K-Nearest Neighbors (KNN) or K-Means Clustering calculate distances between data points. If one feature's scale is vastly larger than another's, that feature will dominate the distance calculation.
2. Algorithms that use gradient descent: Models like linear regression and neural networks find optimal parameters by iteratively adjusting them (gradient descent). If features are on different scales, the optimization process can be slow and unstable, oscillating inefficiently before finding the best solution.

The following video provides an excellent visual explanation of these two points.

The "How": Min-Max Normalization

The most common type of normalization is Min-Max scaling. Its goal is to rescale a feature to a fixed range, usually [0, 1]. The logic is quite intuitive and can be broken down into two simple steps:

1. Shift the data: First, we shift the entire range of values so that the minimum value becomes 0. We do this by subtracting the minimum value of the feature from every data point: $x-\min (X)$.
2. Scale the data: Next, we scale the range down so that the maximum value becomes 1. The range of the shifted data is $\max (X)-\min (X)$. By dividing our shifted value by this range, we ensure all values fall between 0 and 1.

This leads us to the formal Min-Max normalization formula for a value $x$:

Where $X$ represents all the values of the feature. After this transformation, the original minimum value of the feature will map to 0, the original maximum value will map to 1, and all other values will be linearly scaled to fall somewhere in between.

The following resource provides a brilliant, first-principles derivation of this formula. Given your computer science background, you'll appreciate how it builds the formula from a basic problem statement.

A Practical Example

Let's see this in action. Consider a small dataset of employees with their age and salary. The features are on very different scales.

Let's walk through the calculation for an 'Age' of 35, as shown in the image.

• The feature is 'Age', and its values are [44, 27, 30, 38, 40, 35, 48].
• The minimum value is $\min (\text{Age})=27$.
• The maximum value is $\max (\text{Age})=48$.
• The value we want to normalize is $x=35$.

Plugging these into the formula:

As you can see, after normalization, features with vastly different original scales, like Age and Salary, are now directly comparable.

A Critical Implementation Rule: Avoiding Data Leakage

When using scaling in a machine learning context, there is one rule you must always follow to avoid a subtle but serious error known as data leakage. Data leakage occurs when information from your test set "leaks" into your training process, leading to an overly optimistic evaluation of your model's performance.

With scaling, this happens if you calculate the min and max values from the entire dataset before splitting it into training and testing sets. The correct procedure is:

1. Split your data into a training set and a testing set.
2. Calculate the min and max values from the training set only.
3. Use these min and max values to scale both the training set and the testing set.

This simulates a real-world scenario where your model is trained on past data and must then handle new, unseen data (the test set) using the knowledge it already has. The following guide explains this rule clearly and provides Python code demonstrating the correct workflow.

The Downside: Sensitivity to Outliers

Before we conclude, it's important to connect this lesson back to our previous one on outliers. Because Min-Max scaling uses the absolute minimum and maximum values in its formula, it is highly sensitive to outliers.

If you have a single extreme outlier, it can become the new minimum or maximum. This will cause the vast majority of your other data points to be "squished" into a very narrow part of the [0, 1] range, hiding their underlying structure. This is why it's generally good practice to handle outliers before applying Min-Max normalization.

This sensitivity is also a key difference between normalization and another scaling technique, standardization, which we will cover next. Standardization (or Z-score scaling) is less affected by outliers.

Conclusion

In this lesson, you've learned about the critical data preprocessing step of normalization. You now understand not just how to apply Min-Max scaling, but also why it's necessary for many machine learning algorithms.

Key Takeaways:

• Purpose of Scaling: Feature scaling is used to bring all features to a common scale, preventing features with larger magnitudes from unfairly dominating algorithms. It's crucial for distance-based and gradient descent-based models.
• Min-Max Normalization: This technique rescales features to a fixed range, typically [0, 1], using the formula $x_{\text{norm}}=(x-\min )/(\max -\min )$.
• No Data Leakage: In a machine learning workflow, you must fit your scaler (i.e., find the min and max) on the training data only and then use it to transform both the train and test sets.
• Outlier Sensitivity: Min-Max scaling is sensitive to outliers, as they can dictate the minimum and maximum values of the range.

In our next lesson, we will explore standardization (Z-score scaling). You'll learn how it works, how it differs from normalization, and in which scenarios it might be a more robust choice for scaling your data.