Regularization: Lasso and Ridge

Hello! Welcome to your next lesson in the "Model Evaluation and Improvement" module.

In our last lesson, we diagnosed a critical problem in machine learning: overfitting. We saw that complex models can have high variance, performing exceptionally well on training data but failing to generalize to new, unseen test data. This happens because the model learns the "noise" in the training set, not just the underlying signal.

Today, we move from diagnosis to treatment. You'll learn about one of the most powerful techniques to combat overfitting: regularization. This technique directly addresses the learning outcome for this lesson: to apply L1 (Lasso) and L2 (Ridge) regularization to mitigate overfitting. By penalizing model complexity, regularization helps us find a better balance between bias and variance, leading to models that generalize better.

1. The Core Idea: Penalizing Complexity

Let's start by recalling the symptom of an overfit model: it often has very large coefficient values. The model contorts itself with large weights to fit every single data point in the training set, including the noise.

Regularization works by adding a penalty to the model's cost function (like Mean Squared Error). This penalty is proportional to the size of the model's coefficients. The new objective for the model during training becomes:

New Cost Function = Original Cost Function (e.g., MSE) + Penalty Term

By trying to minimize this new combined cost, the model is forced into a trade-off: it must reduce the training error (the original cost) while also keeping its coefficients small (to avoid a large penalty). This discourages overly complex, "squiggly" models and pushes the model towards a simpler, more general solution.

There are two primary ways to define this penalty term, leading to L2 and L1 regularization.

2. L2 Regularization: Ridge Regression

L2 regularization, also known as Ridge Regression, adds a penalty equal to the sum of the squared values of the coefficients. The cost function for a linear regression model with L2 regularization is:

J (θ) = MSE + α i = 1 \sum n θ_{i}^{2}

Here:

$θ_{i}$ represents the model's coefficients (weights).
$α$ (alpha, sometimes called lambda, $λ$ ) is the regularization hyperparameter. It controls the strength of the penalty.
- If $α = 0$ , there is no penalty, and it's just a standard linear regression.
- As $α$ increases, the penalty for large coefficients becomes stronger, forcing them to shrink closer to zero.

To build a strong intuition for how this works, let's watch a segment from the excellent StatQuest series.

Regularization Part 1: Ridge (L2) Regression

This video provides a fantastic, step-by-step visual explanation of how the L2 penalty term helps an overfit model become more robust.

Watch the video from 04:15 to 08:53. Focus on these key points: How Ridge Regression modifies the standard least-squares cost function. The numerical example showing how the penalty term makes the model prefer a line with a smaller slope (less complexity). The intuition that shrinking the slope makes the model's predictions less sensitive to small changes in the input data.

Applying Ridge Regression in Practice

Now let's see how this improves a model in a real coding scenario. We'll use a video that first demonstrates a model overfitting and then applies Ridge Regression to fix it.

Machine Learning Tutorial Python - 17: L1 and L2 Regularization | Lasso, Ridge Regression

This practical tutorial from 'codebasics' will show you the before-and-after effect of applying regularization.

Watch two segments: First, watch from 14:32 to 15:26. You'll see a linear regression model that is clearly overfitting: its score is high on the training data (68%) but very low on the test data (14%). Next, watch from 17:44 to 18:50. Here, the same model is trained with Ridge (L2) regularization. Notice how the test score dramatically improves (to 67%), and the gap between the train and test scores becomes much smaller. This is regularization successfully mitigating overfitting.

The key effect of Ridge regression is that it shrinks the coefficients towards zero, but it very rarely makes them exactly zero. All features are kept in the model, but their influence is reduced.

3. L1 Regularization: Lasso Regression

L1 regularization, also known as Lasso (Least Absolute Shrinkage and Selection Operator), works similarly, but with a crucial difference. It adds a penalty equal to the sum of the absolute values of the coefficients.

J (θ) = MSE + α i = 1 \sum n ∣ θ_{i} ∣

This seemingly small change—using the absolute value instead of the square—has a profound effect: Lasso can shrink some coefficients all the way to zero.

When a feature's coefficient becomes zero, that feature is effectively removed from the model. This makes Lasso a powerful tool not just for regularization but also for automatic feature selection. It helps create simpler, more interpretable models by discarding irrelevant features.

Why Does Lasso Perform Feature Selection?

The reason lies in the geometric shape of the penalty constraints.

Geometric Comparison of Lasso and Ridge Regression — This image provides the geometric intuition for why Lasso leads to sparse solutions. The cost function's elliptical contours expand until they touch the constraint region. For the circular L2 (Ridge) constraint, the contact point is unlikely to be on an axis (where a coefficient is zero). For the diamond-shaped L1 (Lasso) constraint, the corners lie on the axes, making it highly likely that the contact point will be at a corner, thus setting one of the coefficients to zero.

As shown in the image, the L2 penalty corresponds to a circular constraint, while the L1 penalty corresponds to a diamond-shaped constraint. It's much more likely that the optimal point (where the error contours touch the constraint) will be at a sharp corner of the diamond, where one of the coefficients is exactly zero.

Applying Lasso Regression in Practice

Let's return to our coding example and see what happens when we apply Lasso instead of Ridge.

Machine Learning Tutorial Python - 17: L1 and L2 Regularization | Lasso, Ridge Regression

Continuing with the 'codebasics' tutorial, let's see how Lasso (L1) tackles the same overfitting problem.

Watch the segment from 15:26 to 17:44. Notice that Lasso, like Ridge, dramatically improves the test score and closes the gap between training and test performance. The video also mentions that Lasso is L1 regularization.

Test your understanding!

You are building a model to predict a student's exam score based on 50 different features (time spent studying, sleep, previous grades, etc.). You suspect that many of these features are not actually useful.

Which regularization technique, Ridge or Lasso, would be a better choice if your goal is to build a simpler model that only uses the most important predictive features? Why?

Show answer

Lasso (L1) regularization would be a better choice. Because of its L1 penalty (sum of absolute values of coefficients), Lasso has the ability to shrink the coefficients of irrelevant features all the way to zero. This effectively performs feature selection, resulting in a "sparse" model that is simpler and more interpretable. Ridge (L2) would shrink the coefficients of unimportant features, but it would not eliminate them entirely.

4. Head-to-Head: Ridge (L2) vs. Lasso (L1)

Both L1 and L2 are powerful tools for fighting overfitting, but they have different strengths and are suited for different situations. Let's read a clear summary of their differences.

Understanding L1 and L2 regularization

This article provides a concise and clear comparison between L1 and L2 regularization, which is crucial for understanding when to use each.

Read the section titled 'Comparing L1 and L2 Regularization'. Pay close attention to the discussion on: How each handles multicollinearity (highly correlated features). The difference in model complexity and interpretability. The practical scenarios at the end that suggest when to use one over the other.

Here is a summary table of the key differences, which is essential knowledge for the GATE exam:

Feature	Ridge Regression (L2)	Lasso Regression (L1)
Penalty Term	Sum of squared coefficients ( $α \sum θ_{i}^{2}$ )	Sum of absolute coefficients ($\alpha \sum
Feature Selection	No. Shrinks coefficients towards zero but they rarely become exactly zero.	Yes. Can shrink coefficients exactly to zero, effectively removing features.
Sparsity	Produces non-sparse models (all features are kept).	Produces sparse models (only a subset of features remain).
Best Use Case	When you believe most features are useful and you want to manage multicollinearity. It's often a good general-purpose regularizer.	When you suspect many features are irrelevant and you want a simpler, more interpretable model.
Computational	Computationally efficient due to its smooth, differentiable cost function.	Can be more computationally complex because the absolute value function is not differentiable at zero.

Conclusion

Congratulations! You've just learned one of the most important and widely used techniques for improving machine learning models. By adding a penalty for complexity, regularization helps prevent overfitting and creates models that generalize better to new data.

Key Takeaways:

Regularization is a technique to combat overfitting by adding a penalty term to the cost function, discouraging large coefficient values.
L2 Regularization (Ridge) uses a penalty based on the sum of squared coefficients. It shrinks coefficients towards zero but keeps all features in the model.
L1 Regularization (Lasso) uses a penalty based on the sum of absolute coefficients. It can shrink coefficients all the way to zero, thus performing automatic feature selection and creating sparse models.
The hyperparameter $α$ controls the strength of the regularization. A larger $α$ means a stronger penalty and a simpler model.

Preview of the Next Lesson:
We've seen that the choice of the hyperparameter $α$ is crucial. But how do we find the best value for $α$ ? Manually trying different values is inefficient. In the next lesson, we will learn about k-fold cross-validation, a robust and systematic method for tuning hyperparameters and getting a more reliable estimate of our model's performance on unseen data.

Recommended resources