Regularization to Prevent Overfitting

Hello! In our last lesson, we explored how models learn by using gradient descent to minimize a loss function, finding the parameters that best fit the training data. However, optimizing for the training data alone can lead to overfitting, where the model becomes too complex and fails to generalize to new, unseen data.

Today, we'll address this problem head-on. This lesson will explain regularization, a powerful set of techniques that prevent overfitting by directly penalizing model complexity. You will learn how L1 (Lasso) and L2 (Ridge) regularization work by modifying the loss function we studied previously. This is a crucial step in building robust machine learning models that perform well in the real world.

The Problem: Why Overfitting Leads to Large Coefficients

An overfitted model is often too "sensitive" to the training data. To perfectly fit every data point, including the noise, the model often develops extremely large coefficients (weights). A large coefficient means that a small change in an input feature can cause a large change in the output, leading to a "wiggly," unstable function.

Regularization works by adding a penalty term to the model's loss function. This new, modified loss function forces the model to strike a balance: it must still fit the data well, but it is now also incentivized to keep its coefficients small.

The image below provides a great visual intuition. The overfitted model (left) has a complex curve with large slopes to hit every point. The regularized models (right) are smoother and simpler because the algorithm was penalized for creating large coefficients.

An illustration of an overfitted model that perfectly fits noisy data (left) versus simpler, more generalized models produced by L1 and L2 regularization (right). Source: "Overfitting in Machine Learning" on Medium.

L2 Regularization (Ridge): Shrinking Coefficients

L2 Regularization, also known as Ridge Regression, combats overfitting by adding a penalty proportional to the square of the magnitude of the coefficients.

In our previous lesson, the objective was simply to minimize the Mean Squared Error (MSE), or Residual Sum of Squares (RSS). With L2 regularization, the objective becomes:

Objective = RSS + α j = 1 \sum p w_{j}^{2}

Where:

$w_{j}$ is the coefficient for the $j$ -th feature.
$α$ (alpha) is the regularization hyperparameter, which controls the strength of the penalty. A larger $α$ means a stronger penalty, resulting in smaller coefficients.
The term $\sum_{j = 1}^{p} w_{j}^{2}$ is called the L2 norm of the coefficient vector.

By adding this penalty, we are telling the model: "Find coefficients that reduce the error (RSS), but do it without making the coefficients too large."

To build a strong intuition for how this works, let's watch a segment from the excellent StatQuest series.

Regularization Part 1: Ridge (L2) Regression

This video, "Regularization Part 1: Ridge (L2) Regression," provides a wonderfully clear, step-by-step explanation of how Ridge regularization works.

Watch the following parts: The Problem of Overfitting: This section sets up the problem using a simple example, showing how fitting a small training set perfectly can lead to high variance on the test set. How Ridge Regression Works: This is the core of the explanation. Pay close attention to how the new objective function includes the penalty term (lambda is used here, which is the same as alpha) and how it leads to a different, less complex model. Effect on Slopes and Lambda: This part visualizes how the penalty leads to smaller slopes, making predictions less sensitive, and explains how the \alpha (lambda) parameter controls the strength of the penalty.

The key takeaway is that L2 regularization shrinks the coefficients towards zero, but it very rarely sets them exactly to zero. It tames all features, but keeps them all in the model.

L1 Regularization (Lasso): Feature Selection

L1 Regularization, also known as Lasso (Least Absolute Shrinkage and Selection Operator) regression, takes a slightly different approach. It penalizes the absolute value of the coefficients.

The L1 objective function is:

Objective = RSS + α j = 1 \sum p ∣ w_{j} ∣

The term $\sum_{j = 1}^{p} ∣ w_{j} ∣$ is the L1 norm. This seemingly small change—using the absolute value instead of the square—has a profound effect. While both L1 and L2 shrink coefficients, L1 is capable of shrinking some coefficients all the way to zero.

When a coefficient becomes zero, its corresponding feature is effectively removed from the model. This means L1 regularization performs automatic feature selection, resulting in a simpler, or "sparser," model. This is especially useful when you have a large number of features and you suspect that many of them are not very useful.

Ridge vs Lasso Regression, Visualized!!!

This StatQuest video, "Ridge vs Lasso Regression, Visualized!!!," does a fantastic job of comparing the two penalties.

Focus on these segments: Ridge Penalty (L2): Watch how increasing the penalty strength \alpha (lambda) pulls the optimal slope closer and closer to zero, but never quite reaches it. Lasso Penalty (L1): See how the L1 penalty creates a "kink" at zero. As \alpha increases, the optimal slope can get "stuck" at exactly zero. This is the mechanism behind feature selection. Summary: This provides a final, clear comparison of the two behaviors.

A Geometric View: Why L1 Creates Zeros

Why does the L1 penalty lead to zero coefficients while L2 does not? The best way to understand this is with a geometric analogy. Imagine the regularization penalty defines a "budget" or a "constraint region" within which the optimal coefficients must lie.

Geometric interpretation of Lasso (L1) and Ridge (L2) regularization. The optimal solution is the first point where the loss function contours (red ellipses) touch the constraint region (cyan). For Lasso, this often happens at a corner, setting a coefficient to zero. Source: AnalytixLabs.

L2 (Ridge): The penalty $\sum w_{j}^{2} \leq budget$ defines a circular constraint region (in 2D). The loss function's elliptical contours are likely to touch this circle at a point where neither coefficient is zero.
L1 (Lasso): The penalty $\sum ∣ w_{j} ∣ \leq budget$ defines a diamond-shaped region. The sharp corners of the diamond lie on the axes. It's highly probable that the loss function's contours will first touch the constraint region at one of these corners, forcing one of the coefficients to be exactly zero.

This visual provides a powerful, intuition-first reason for the different behaviors of L1 and L2 regularization.

L1 vs. L2: A Practical Guide

Now that you understand the mechanics, let's summarize the practical differences and when to choose one over the other.

Ridge and Lasso Regression in Python

This article from Analytics Vidhya provides excellent summaries and practical advice on using these techniques.

First, review the "Table of contents" and locate the "Comparison Between Ridge Regression and Lasso Regression" section to read. Then, review the introductory table. Start with the table under "What Are Ridge Regression and Lasso Regression?": This table gives a great at-a-glance comparison of the two methods. Then read the section "Comparison Between Ridge Regression and Lasso Regression": Focus on the subsections "Key Difference," "Typical Use Cases," and "Presence of Highly Correlated Features." This provides crucial context for real-world application.

Here's a summary of the key points from the article:

Aspect	L2 (Ridge) Regularization	L1 (Lasso) Regularization
Penalty	Square of coefficients ( $w^{2}$ )	Absolute value of coefficients ($
Effect	Shrinks coefficients towards zero	Shrinks some coefficients exactly to zero
Feature Selection	No, keeps all features	Yes, performs automatic feature selection
Sparsity	Creates non-sparse models	Creates sparse models (fewer non-zero coefficients)
Use Case	When you believe most features are useful and you want to prevent overfitting. Handles correlated features well.	When you have many features and suspect some are irrelevant. Useful for creating simpler, more interpretable models.

In practice, when implementing these in Python with scikit-learn, you will use models like Ridge and Lasso. For models like LogisticRegression, you can specify the penalty type (e.g., penalty='l1' or penalty='l2') and the regularization strength, which is often controlled by a parameter C. Note that C is the inverse of regularization strength: smaller values of C correspond to a stronger penalty.

Conclusion

In this lesson, you've learned how to go beyond simple loss minimization to build models that are more robust and less prone to overfitting. Regularization is a fundamental technique in the data scientist's toolkit.

Key Takeaways:

Regularization prevents overfitting by adding a penalty for model complexity to the loss function.
L2 (Ridge) regularization adds a penalty proportional to the square of the coefficients, shrinking them towards zero. It is effective at reducing model complexity while keeping all features.
L1 (Lasso) regularization adds a penalty proportional to the absolute value of the coefficients. This can shrink coefficients to exactly zero, thus performing feature selection and creating sparse models.
The choice between L1 and L2 depends on your assumptions about the data: use L2 when most features are likely useful, and L1 when you want to simplify your model by eliminating irrelevant features.
The strength of the penalty is a crucial hyperparameter ( $α$ or $C$ ) that must be tuned.

Now we understand a key technique for preventing overfitting. But how do we diagnose it in the first place? In our next lesson, we will explore learning curves, a powerful diagnostic tool that helps us visualize whether a model is suffering from high bias (underfitting) or high variance (overfitting).