Regularization: Ridge and Lasso

The Problem with Large Coefficients

Let's start by revisiting the core issue with overfitting. As we saw with polynomial regression in the last lesson, when a model becomes too complex, its coefficients can become very large. The model starts fitting the noise in the training data, leading to a "wiggly" and unstable function. A large coefficient means that a small change in an input feature can cause a huge change in the output, making the model highly sensitive and prone to high variance.

Regularization tackles this problem head-on by adding a penalty to the model's loss function. Instead of just trying to minimize the error (like the Sum of Squared Errors, or SSE), the model now has to minimize:

This penalty term is designed to punish large coefficients, forcing the model to find a balance between fitting the data well and keeping its coefficients small.

Now, let's explore the two most common ways this penalty is applied: Ridge and Lasso regression.

Ridge Regression (L2 Regularization): The Shrinker

Ridge regression is our first tool for controlling model complexity. It adds a penalty proportional to the sum of the squares of the coefficients. This is also known as the L2 norm.

The objective function for Ridge regression becomes:

Where:

• $\sum (y_i-{\hat{y}}_i{)}^2$ is the standard Sum of Squared Errors (SSE).
• ${\beta }_j$ is the coefficient for the j-th feature.
• $\alpha$ (often denoted as $\lambda$) is the regularization strength. It's a hyperparameter we control.
  • If $\alpha =0$, the penalty is zero, and Ridge regression is identical to ordinary linear regression.
  • As $\alpha$ increases, the penalty for large coefficients becomes more severe, forcing the coefficients to shrink closer to zero.
• Note: The intercept term (${\beta }_0$) is typically not included in the penalty.

This penalty forces the model to use smaller coefficients. While it shrinks them towards zero, it will very rarely set any of them exactly to zero.

To build a strong intuition for how Ridge regression works, the following video is an excellent guide.

Lasso Regression (L1 Regularization): The Feature Selector

Lasso regression works in a similar way to Ridge, but with one crucial difference in its penalty. It adds a penalty proportional to the sum of the absolute values of the coefficients. This is known as the L1 norm.

The objective function for Lasso regression is:

This seemingly small change from squaring (${\beta }_j^2$) to taking the absolute value ($|{\beta }_j|$) has a profound effect. While Ridge shrinks coefficients towards zero, Lasso can shrink them all the way to zero.

This means Lasso can perform automatic feature selection. If a feature's coefficient becomes zero, that feature is effectively removed from the model. This is incredibly useful when you have a dataset with many features, some of which may be irrelevant.

Let's watch the follow-up StatQuest video to see this in action.

Applying Ridge and Lasso in Python

Now that we understand the concepts, let's see how to apply them. We'll use the polynomial regression example, which directly ties into our last lesson on the bias-variance tradeoff. By fitting a high-degree polynomial, we'll intentionally create an overfit model and then use Ridge and Lasso to tame it.

The following tutorial provides a great practical walkthrough of this exact process.

To consolidate this, let's look at a focused code example. Here, we create a dataset with 10 features, but only 2 of them are actually useful for prediction. The rest are just noise. This is a perfect scenario to see the power of Lasso.

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# 1. Generate synthetic data
np.random.seed(42)
n_samples = 100
n_features = 10
n_informative = 2 # Only 2 features will actually be useful

# Create a design matrix X
X = np.random.randn(n_samples, n_features)

# Create true coefficients: 2 are large, the rest are zero
true_coef = np.zeros(n_features)
true_coef[:n_informative] = [15, -8]

# Create the target variable y with some noise
y = X @ true_coef + np.random.randn(n_samples) * 5

# Scale the data - this is important for regularization!
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# 2. Fit the models
# Ordinary Linear Regression
lr = LinearRegression()
lr.fit(X_scaled, y)

# Ridge Regression (alpha is the regularization strength)
ridge = Ridge(alpha=1.0)
ridge.fit(X_scaled, y)

# Lasso Regression
lasso = Lasso(alpha=1.0)
lasso.fit(X_scaled, y)

# 3. Compare the coefficients
print("True Coefficients:\n", true_coef)
print("\nLinear Regression Coefficients:\n", np.round(lr.coef_, 2))
print("\nRidge Regression Coefficients:\n", np.round(ridge.coef_, 2))
print("\nLasso Regression Coefficients:\n", np.round(lasso.coef_, 2))

Running this code produces the following output:

True Coefficients:
 [15. -8.  0.  0.  0.  0.  0.  0.  0.  0.]

Linear Regression Coefficients:
 [14.61 -7.71 -0.47  0.37  0.42  0.03 -0.63 -0.59  0.72 -0.82]

Ridge Regression Coefficients:
 [13.79 -7.42 -0.42  0.31  0.38  0.06 -0.56 -0.52  0.64 -0.73]

Lasso Regression Coefficients:
 [13.48 -6.59 -0.    0.   -0.   -0.   -0.   -0.    0.   -0.  ]

Analysis of the output:

• Linear Regression: It correctly identifies the two important features (with coefficients 14.61 and -7.71), but it also assigns non-zero, noisy coefficients to all the useless features.
• Ridge Regression: It shrinks all the coefficients. The important ones get a bit smaller (13.79, -7.42), and the noisy ones also get a bit smaller. However, none of them become exactly zero.
• Lasso Regression: This is the star of the show. It correctly identifies the two important features (13.48, -6.59) and, crucially, it sets the coefficients for all the useless, noisy features to exactly zero. It has successfully performed feature selection and recovered a model that is much closer to the "true" underlying data-generating process.

Conclusion

In this lesson, we moved from the theory of overfitting to the practical application of regularization. You now have two powerful techniques in your toolkit to build more robust and generalizable models.

Key Takeaways:

• Regularization prevents overfitting by adding a penalty for large coefficients to the model's loss function.
• Ridge Regression (L2) uses a sum of squares penalty. It shrinks coefficients towards zero, which is effective for reducing model complexity, especially when many features are somewhat predictive.
• Lasso Regression (L1) uses a sum of absolute values penalty. It can shrink coefficients all the way to zero, performing automatic feature selection. This is extremely powerful when you suspect many of your features are irrelevant.
• The strength of regularization is controlled by a hyperparameter, $\alpha$, which must be tuned (often via cross-validation) to find the right balance between bias and variance.

Preview of the Next Lesson:

We've now seen what Ridge and Lasso do and how to apply them. But why does the L1 penalty lead to sparsity while the L2 penalty doesn't? In our next lesson, we will compare the effects of L1 and L2 penalties on model coefficients in greater depth. We will explore the geometric intuition behind these methods and analyze their behavior with correlated features, giving you a deeper understanding of when and why to choose one over the other.