Implementing Regularization in Linear and Logistic Regression

1. Generalizing Your Model Classes for Regularization

The first step is to refactor your existing LinearRegression and LogisticRegression classes so they can accept regularization parameters. The changes will be almost identical for both. We need to:

1. Update the __init__ method: Add new parameters to specify the type of regularization ('l1', 'l2', or None) and the regularization strength (lambda_).
2. Update the cost calculation method: Add logic to include the L1 or L2 penalty term in the final cost.
3. Update the gradient calculation method: Add logic to include the gradient of the L1 or L2 penalty term in the weight gradient.

The article "Create a Gradient Descent Algorithm with Regularization from Scratch in Python" provides a perfect blueprint for this. It shows how to modify a class to handle these exact cases.

2. Applying Regularization to Your LinearRegression Model

Let's start with linear regression. You already have a class that uses gradient descent to minimize the Mean Squared Error (MSE). Now, we'll augment it.

Modifying the Code

Below is a template for how you would modify your LinearRegression class. I've omitted the parts that don't change (like the predict method) to focus on the key additions.

import numpy as np

class LinearRegression:
    def __init__(self, learning_rate=0.01, n_iters=1000, regularization=None, lambda_=0.1):
        """
        Initializes the Linear Regression model.
        
        Args:
            learning_rate (float): The step size for gradient descent.
            n_iters (int): The number of iterations for gradient descent.
            regularization (str): Type of regularization. 'l1', 'l2', or None.
            lambda_ (float): Regularization strength parameter.
        """
        self.lr = learning_rate
        self.n_iters = n_iters
        self.regularization = regularization
        self.lambda_ = lambda_
        self.weights = None
        self.bias = None

    def fit(self, X, y):
        n_samples, n_features = X.shape
        self.weights = np.zeros(n_features)
        self.bias = 0

        for _ in range(self.n_iters):
            # Calculate predictions
            y_predicted = np.dot(X, self.weights) + self.bias
            
            # Compute gradients
            dj_dw, dj_db = self._compute_gradients(X, y, y_predicted)
            
            # Update parameters
            self.weights -= self.lr * dj_dw
            self.bias -= self.lr * dj_db

    def _compute_gradients(self, X, y, y_predicted):
        n_samples = X.shape[0]
        
        # Original gradients for MSE
        error = y_predicted - y
        dj_dw_original = (1 / n_samples) * np.dot(X.T, error)
        dj_db = (1 / n_samples) * np.sum(error)
        
        dj_dw = dj_dw_original
        
        # --- NEW: Add regularization gradient term ---
        if self.regularization == 'l2':
            l2_grad_term = (self.lambda_ / n_samples) * self.weights
            dj_dw += l2_grad_term
        elif self.regularization == 'l1':
            l1_grad_term = (self.lambda_ / n_samples) * np.sign(self.weights)
            dj_dw += l1_grad_term
            
        return dj_dw, dj_db

    # You could also add a _compute_cost method for monitoring convergence
    def _compute_cost(self, X, y, y_predicted):
        n_samples = X.shape[0]
        
        # Original MSE cost
        cost = (1 / (2 * n_samples)) * np.sum((y_predicted - y)**2)
        
        # --- NEW: Add regularization penalty to cost ---
        if self.regularization == 'l2':
            l2_penalty = (self.lambda_ / (2 * n_samples)) * np.sum(np.square(self.weights))
            cost += l2_penalty
        elif self.regularization == 'l1':
            l1_penalty = (self.lambda_ / n_samples) * np.sum(np.abs(self.weights))
            cost += l1_penalty
            
        return cost

    def predict(self, X):
        return np.dot(X, self.weights) + self.bias

Key changes:

• The __init__ method now accepts regularization and lambda_.
• The _compute_gradients method calculates the original MSE gradient and then adds the L1 or L2 gradient term conditionally. Remember, the bias term is not regularized.
• I've included an optional _compute_cost method to show how you'd add the penalty for monitoring the loss during training. The core logic resides in the gradient update.

The video below walks through building a Lasso regression class from scratch. It reinforces the structure we are implementing, particularly how the __init__ and gradient update functions work together.

3. Applying Regularization to Your LogisticRegression Model

The process for your LogisticRegression class is identical in principle. You will make the same structural changes, but apply them to the binary cross-entropy cost function and its gradient.

Modifying the Code

Here is the template for your LogisticRegression class:

import numpy as np

class LogisticRegression:
    def __init__(self, learning_rate=0.01, n_iters=1000, regularization=None, lambda_=0.1):
        self.lr = learning_rate
        self.n_iters = n_iters
        self.regularization = regularization
        self.lambda_ = lambda_
        self.weights = None
        self.bias = None

    def _sigmoid(self, z):
        return 1 / (1 + np.exp(-z))

    def fit(self, X, y):
        n_samples, n_features = X.shape
        self.weights = np.zeros(n_features)
        self.bias = 0

        for _ in range(self.n_iters):
            # Calculate predictions
            linear_model = np.dot(X, self.weights) + self.bias
            y_predicted = self._sigmoid(linear_model)
            
            # Compute gradients
            dj_dw, dj_db = self._compute_gradients(X, y, y_predicted)
            
            # Update parameters
            self.weights -= self.lr * dj_dw
            self.bias -= self.lr * dj_db

    def _compute_gradients(self, X, y, y_predicted):
        n_samples = X.shape[0]
        
        # Original gradients for Binary Cross-Entropy
        error = y_predicted - y
        dj_dw_original = (1 / n_samples) * np.dot(X.T, error)
        dj_db = (1 / n_samples) * np.sum(error)
        
        dj_dw = dj_dw_original
        
        # --- NEW: Add regularization gradient term ---
        if self.regularization == 'l2':
            # Note: We don't regularize the bias term, so we add the penalty
            # only to the weight gradients.
            l2_grad_term = (self.lambda_ / n_samples) * self.weights
            dj_dw += l2_grad_term
        elif self.regularization == 'l1':
            l1_grad_term = (self.lambda_ / n_samples) * np.sign(self.weights)
            dj_dw += l1_grad_term
            
        return dj_dw, dj_db

    # ... predict method would be here ...

As you can see, the code in _compute_gradients for adding the regularization term is exactly the same as it was for linear regression. This demonstrates the modular power of this approach: you can add regularization to any model that uses a cost function and gradient descent by simply augmenting the cost and gradient.

4. Seeing it in Action

Now, let's test your new, enhanced models. A great way to see the effect of regularization is to intentionally create an overfitting scenario and watch how L1 and L2 penalties fix it.

1. Create an Overfitting Problem: Generate a simple dataset (e.g., a sine wave with noise). Then, use polynomial features to create a high-degree feature set (e.g., degree 10). A linear model trained on these high-degree features will severely overfit the noise.
2. Train Three Models:
   • model_plain = LinearRegression(regularization=None)
   • model_l2 = LinearRegression(regularization='l2', lambda_=0.1)
   • model_l1 = LinearRegression(regularization='l1', lambda_=0.1)
3. Visualize the Results: Plot the original data and the prediction curves from all three models. You should see the unregularized model fit the noise, while the L1 and L2 models produce smoother, more generalized curves.

The following image perfectly illustrates the expected outcome of such an experiment.

This visual result is precisely what you are aiming to reproduce with your own from-scratch code.

Conclusion

Congratulations! You have successfully integrated regularization into your from-scratch machine learning models. This is a significant step, turning your simple implementations into more robust and practical tools.

Key Takeaways:

• You can apply L1 and L2 regularization to any gradient-based model by making small modifications to the __init__, cost, and gradient methods.
• The logic for adding the regularization penalty and its gradient is modular and can be applied to different cost functions (like MSE and Binary Cross-Entropy).
• By parameterizing your classes, you've created a flexible framework to easily switch between L1, L2, or no regularization, allowing you to experiment and find the best approach for a given problem.

Preview of the Next Lesson:
We have now completed our deep dive into linear models, optimization with gradient descent, and regularization. You have a solid foundation. In the next module, we will explore a completely different and powerful approach to classification: Support Vector Machines (SVMs). We'll start by understanding the core concepts of maximum margin classification and the role of support vectors.