Feature Scaling: Standardization vs. Normalization

1. Why is Feature Scaling Necessary?

Imagine you're training a K-Nearest Neighbors (KNN) model, which relies on calculating the distance between data points. If one feature (like income) has a much larger range than another (like age), the distance calculation will be completely dominated by the income feature. The age feature's contribution will be almost negligible, and the model will fail to learn from it properly.

Feature scaling puts all features on a similar scale, ensuring that each one contributes proportionately to the model's learning process.

To start, let's watch a short video that clearly illustrates this problem with a practical example.

Which Algorithms Are Affected?

It's important to know that not all algorithms require feature scaling.

• Sensitive Algorithms: Algorithms that use distance measures or are based on gradient descent are highly sensitive to feature scales. This includes:

  • Linear Regression & Logistic Regression
  • K-Nearest Neighbors (KNN)
  • Support Vector Machines (SVM)
  • K-Means Clustering
  • Principal Component Analysis (PCA)
  • Neural Networks (Deep Learning)

• Insensitive Algorithms: Tree-based algorithms are generally not sensitive to feature scaling because they make splitting decisions based on single features at a time. This includes:

  • Decision Trees
  • Random Forest
  • Gradient Boosting models (like XGBoost, LightGBM)

Let's reinforce this with a quick read.

2. The Two Main Techniques: Normalization and Standardization

Now that we understand the "why," let's focus on the "how." The two most common scaling techniques are Normalization and Standardization. This image provides an excellent high-level comparison.

Let's break each one down.

Normalization (Min-Max Scaling)

Normalization scales your data into a fixed range, typically between 0 and 1.

• Formula: $y=\frac{x-x_{min}}{x_{max}-x_{min}}$
• Result: The minimum value of the feature becomes 0, the maximum value becomes 1, and all other values fall somewhere in between.
• Scikit-learn Class: MinMaxScaler

Standardization (Z-score Scaling)

Standardization rescales data to have a mean of 0 and a standard deviation of 1.

• Formula: $y=\frac{x-\mu }{\sigma }$
  (where $\mu$ is the mean and $\sigma$ is the standard deviation)
• Result: The feature will be "centered" around 0. Most values will lie between -3 and 3, but there's no fixed upper or lower bound.
• Scikit-learn Class: StandardScaler

To see how these are implemented in practice, let's watch a hands-on demonstration.

Hands-On Practice

Here is a Python example demonstrating how to apply these scalers. Notice the crucial pattern: we use .fit_transform() on the training data to learn the scaling parameters (min/max or mean/std) and then use .transform() on the test data to apply those same parameters. This prevents data leakage from the test set into our training process.

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler, StandardScaler




# Sample data with features on different scales
data = {'age': [25, 30, 45, 35, 50, 22],
        'salary': [50000, 60000, 120000, 80000, 150000, 45000]}
df = pd.DataFrame(data)




# Split data to simulate a real scenario
X_train, X_test = train_test_split(df, test_size=0.33, random_state=42)

print("--- Original Data ---")
print("Train:\n", X_train)
print("\nTest:\n", X_test)




# --- Normalization (MinMaxScaler) ---
min_max_scaler = MinMaxScaler()



# Fit on training data and transform it
X_train_normalized = min_max_scaler.fit_transform(X_train)



# Only transform the test data
X_test_normalized = min_max_scaler.transform(X_test)

print("\n--- After Normalization (MinMaxScaler) ---")
print("Train Scaled:\n", pd.DataFrame(X_train_normalized, columns=df.columns))
print("\nTest Scaled:\n", pd.DataFrame(X_test_normalized, columns=df.columns))





# --- Standardization (StandardScaler) ---
standard_scaler = StandardScaler()



# Fit on training data and transform it
X_train_standardized = standard_scaler.fit_transform(X_train)



# Only transform the test data
X_test_standardized = standard_scaler.transform(X_test)

print("\n\n--- After Standardization (StandardScaler) ---")
print("Train Scaled:\n", pd.DataFrame(X_train_standardized, columns=df.columns))
print("\nTest Scaled:\n", pd.DataFrame(X_test_standardized, columns=df.columns))




# Verify the properties of the standardized training data
print("\n\n--- Properties of Standardized Training Data ---")
print("Mean (close to 0):\n", pd.DataFrame(X_train_standardized, columns=df.columns).mean())
print("\nStandard Deviation (close to 1):\n", pd.DataFrame(X_train_standardized, columns=df.columns).std())

3. When to Use Which? The Decision-Making Process

This is the most important part of the lesson: choosing the right tool for the job. The decision depends on your algorithm and, critically, the distribution of your data.

Here's a summary of the key decision factors:

1. Data Distribution and Outliers:

   • Standardization is generally preferred if your data follows a Gaussian (bell-shaped) distribution. It is also less sensitive to outliers.
   • Normalization is useful when your data does not have a Gaussian distribution and you need your feature values to be in a bounded interval (e.g., [0, 1] for image pixel intensities or certain neural networks). However, it is very sensitive to outliers. A single extreme value can cause all other data points to be "squashed" into a very small range.

2. The Impact of Outliers: A Visual Explanation

The image below perfectly illustrates how different scalers handle outliers. Pay close attention to MinMaxScaler.

This visual shows why Standardization is often a safer default choice. The RobustScaler is even better at handling outliers, which is a technique we'll discuss in our next lesson.

To solidify your decision-making process, the following reading provides an excellent final summary.

The key takeaway is that there's no single "best" scaler. If you're unsure, standardization is a good starting point. For best results on a serious project, you would experiment to see which scaling method yields the best performance for your specific model.

Conclusion

Great job! You have now added another essential preprocessing technique to your data science toolkit. Understanding how and when to scale your features is crucial for building high-performing models.

Key Takeaways:

• Why we scale: To prevent features with large ranges from dominating algorithms that are sensitive to scale (e.g., distance-based or gradient-based).
• Normalization (MinMaxScaler): Scales features to a fixed range, usually [0, 1]. It's useful for algorithms that require bounded inputs but is highly sensitive to outliers.
• Standardization (StandardScaler): Scales features to have a mean of 0 and a standard deviation of 1. It's less affected by outliers and is a robust, common default choice.
• How to choose: Consider your algorithm and data distribution. Standardization is often a safe bet. When in doubt, you can always test multiple approaches.

Preview of the next lesson:

In this lesson, we saw how outliers can dramatically affect certain scaling techniques. This highlights the importance of dealing with them directly. In our next lesson, we will focus on applying common strategies for handling outliers, such as clipping and removal, building directly on the concepts we've touched on today.