Standard Scaling for Faster Convergence

1. Why Feature Scaling is Necessary

Imagine you're building a model to predict house prices. You have two features: size_in_sq_ft (ranging from 500 to 5000) and number_of_bedrooms (ranging from 1 to 5). The values for size_in_sq_ft are orders of magnitude larger than for number_of_bedrooms.

When we calculate the gradient of the loss function, the feature values directly influence the gradient's magnitude. A large feature value will lead to a large gradient for its corresponding weight. This means the weight for size_in_sq_ft will receive a much larger update than the weight for number_of_bedrooms, even if both are equally important.

This disparity creates a problem for gradient descent. The cost function surface becomes a deep, narrow ellipse instead of a more circular bowl. Gradient descent will struggle, oscillating back and forth across the narrow valley instead of heading straight for the minimum. This forces us to use a very small learning rate to avoid overshooting, leading to slow convergence.

By scaling our features, we make the cost function's contours more circular, which allows gradient descent to converge much faster and more reliably.

2. Types of Feature Scaling

There are two primary methods for scaling features. It's important to know both, although we will focus on implementing one.

Let's summarize the key formulas:

1. Standardization (or Z-score Normalization): This method rescales features so they have the properties of a standard normal distribution, with a mean ($\mu$) of 0 and a standard deviation ($\sigma$) of 1.
   $z=\frac{x-\mu }{\sigma }$
   This is the most common scaling technique and the one we will implement. It's particularly useful for algorithms that assume features are centered around zero, like SVMs and PCA.

2. Min-Max Scaling (often called Normalization): This method scales the data to a fixed range, usually 0 to 1.
   $X_{norm}=\frac{X-X_{min}}{X_{max}-X_{min}}$
   This is useful for algorithms like neural networks that require input values within a bounded interval, or for image processing where pixel values need to be scaled.

3. Implementing a StandardScaler from Scratch

In line with your goal of building everything from the ground up, we will create our own StandardScaler class in Python. This is a fantastic exercise, as it mirrors the design of popular libraries like scikit-learn, where data transformers (fit, transform) are separate from models.

Our scaler needs to do two things:

1. fit: Learn the scaling parameters (mean $\mu$ and standard deviation $\sigma$) from the training data.
2. transform: Apply the learned scaling to any new data (both the training set and the test set).

A critical rule: The mean and standard deviation must be calculated only from the training set. We then use these same parameters to scale the validation and test sets. This prevents "data leakage," where information from the test set inadvertently influences the training process.

Here is the implementation. Notice how we use NumPy's vectorized operations to make it clean and efficient.

import numpy as np

class StandardScaler:
    """
    A from-scratch implementation of Standardization (Z-score normalization).
    """
    def __init__(self):
        self.mean_ = None
        self.scale_ = None # This will be the standard deviation

    def fit(self, X):
        """
        Compute the mean and standard deviation for each feature in X.
        X: Training data, shape (n_samples, n_features)
        """
        # Calculate mean and std deviation for each feature (column-wise, axis=0)
        self.mean_ = np.mean(X, axis=0)
        self.scale_ = np.std(X, axis=0)
        
        # A safety check to avoid division by zero if a feature has zero variance
        self.scale_[self.scale_ == 0] = 1

    def transform(self, X):
        """
        Scale features of X using the stored mean and standard deviation.
        X: Data to transform, shape (n_samples, n_features)
        """
        if self.mean_ is None or self.scale_ is None:
            raise RuntimeError("You must call fit before using transform.")
        return (X - self.mean_) / self.scale_

    def fit_transform(self, X):
        """
        A convenience method to fit to the data and then transform it.
        """
        self.fit(X)
        return self.transform(X)

4. Demonstrating the Impact on Convergence

Now for the payoff. Let's demonstrate the difference scaling makes. We'll create a synthetic dataset with features on very different scales and train our LinearRegression model from the previous lesson on both the unscaled and scaled versions.

First, let's update our LinearRegression class slightly to store the cost at each iteration, so we can plot its convergence.

# --- Updated LinearRegression class from previous lesson ---
class LinearRegression:
    def __init__(self, learning_rate=0.01, n_iters=1000):
        self.lr = learning_rate
        self.n_iters = n_iters
        self.weights = None
        self.bias = None
        self.cost_history = [] # New: to store cost per iteration

    def fit(self, X, y):
        n_samples, n_features = X.shape
        self.weights = np.zeros(n_features)
        self.bias = 0
        if y.ndim > 1: y = y.ravel()
        
        self.cost_history = [] # Reset history

        for i in range(self.n_iters):
            y_predicted = np.dot(X, self.weights) + self.bias
            
            # Calculate cost (MSE) and store it
            cost = np.mean((y_predicted - y)**2)
            self.cost_history.append(cost)
            
            # Calculate gradients
            dw = (2 / n_samples) * np.dot(X.T, (y_predicted - y))
            db = (2 / n_samples) * np.sum(y_predicted - y)
            
            # Update parameters
            self.weights -= self.lr * dw
            self.bias -= self.lr * db
            
    def predict(self, X):
        return np.dot(X, self.weights) + self.bias

Now, let's run the experiment.

# --- Imports for demonstration ---
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# 1. Create synthetic data with features on different scales
np.random.seed(42)
# Feature 1: Large scale (e.g., house size)
X1 = np.random.uniform(1000, 3000, 100).reshape(-1, 1)
# Feature 2: Small scale (e.g., number of rooms)
X2 = np.random.uniform(2, 6, 100).reshape(-1, 1)

X_unscaled = np.hstack((X1, X2))

# True relationship: y = 2*X1 + 500*X2 + 100 + noise
y = 2 * X1 + 500 * X2 + 100 + np.random.normal(0, 500, (100, 1))

# Split data
X_train_unscaled, X_test_unscaled, y_train, y_test = train_test_split(
    X_unscaled, y, test_size=0.2, random_state=42
)


# --- Scenario 1: Training on UNSCALED data ---
# We need a small learning rate to avoid divergence
lr_unscaled = 0.0000001 
reg_unscaled = LinearRegression(learning_rate=lr_unscaled, n_iters=1000)
reg_unscaled.fit(X_train_unscaled, y_train)


# --- Scenario 2: Training on SCALED data ---
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train_unscaled)
X_test_scaled = scaler.transform(X_test_unscaled) # Use the same scaler!

# We can use a much larger learning rate now
lr_scaled = 0.1 
reg_scaled = LinearRegression(learning_rate=lr_scaled, n_iters=1000)
reg_scaled.fit(X_train_scaled, y_train)


# --- 5. Compare convergence ---
plt.figure(figsize=(10, 6))
plt.plot(reg_unscaled.cost_history, label=f'Unscaled Data (lr={lr_unscaled})')
plt.plot(reg_scaled.cost_history, label=f'Scaled Data (lr={lr_scaled})')
plt.title('Impact of Feature Scaling on Convergence Speed')
plt.xlabel('Iteration')
plt.ylabel('Mean Squared Error (Cost)')
plt.legend()
plt.yscale('log') # Use a log scale to see both curves clearly
plt.grid(True)
plt.show()

When you run this code, the resulting plot is striking. The blue line (unscaled data) decreases very slowly, even after 1000 iterations. The orange line (scaled data) plummets dramatically, reaching a much lower error in just a few dozen iterations. This is the power of feature scaling in action.

Conclusion

You have successfully implemented one of the most important preprocessing steps in machine learning. By ensuring all features contribute to the gradient update on a similar scale, you've made your gradient descent algorithm faster, more stable, and easier to tune.

Key Takeaways:

• Features on vastly different scales can cause the gradient descent algorithm to converge very slowly.
• Standard Scaling remaps features to have a mean of 0 and a standard deviation of 1, creating a more favorable cost surface for optimization.
• It's crucial to fit the scaler only on the training data and then use it to transform the train, validation, and test sets to prevent data leakage.
• With scaled features, you can often use a much larger learning rate, leading to significantly faster convergence.

Preview of the Next Lesson:
So far, we have focused on an iterative method (gradient descent) to find the best model parameters. But for linear regression, there's another way. In the next lesson, we will explore the Normal Equation, a mathematical formula that can solve for the optimal weights directly and analytically, without any iterations at all. We will implement it and compare its trade-offs with gradient descent.