Regularization: Ridge and Lasso

1. The Core Idea: Penalizing Complexity

Let's start with the model from our last lesson. A simple linear regression model has a cost function based on the Residual Sum of Squares (RSS):
$J(\beta )={\sum }_{i=1}^n(y_i-{\hat{y}}_i{)}^2$
When we used a high-degree polynomial, the model could minimize this RSS to almost zero by creating a very "wiggly" curve with large positive and negative coefficients.

Regularization introduces a simple but brilliant modification: it adds a penalty term to the cost function. This new cost function forces the model to perform a balancing act:

1. Fit the data well (keep the RSS low).
2. Keep the model simple (keep the coefficients small, thus minimizing the penalty).

The general form of a regularized cost function is:
$J_{regularized}(\beta )=\text{RSS}+\text{Penalty}$

This approach is a direct application of the bias-variance tradeoff. We intentionally introduce a small amount of bias (by not letting the model fit the training data perfectly) to achieve a significant reduction in variance (leading to better generalization on unseen data).

The following video provides an excellent intuitive introduction to this concept, showing how we can get a better long-term model by accepting a slightly worse fit on the training data.

Now, let's see how this penalty is specifically implemented in Ridge and Lasso regression.

2. Ridge Regression (L2 Regularization)

Ridge Regression adds a penalty proportional to the sum of the squares of the coefficients. This is also known as an L2 penalty.

The Ridge cost function is:
$J_{Ridge}(\beta )={\sum }_{i=1}^n(y_i-{\hat{y}}_i{)}^2+\lambda {\sum }_{j=1}^p{\beta }_j^2$

Let's break this down:

• ${\sum }_{j=1}^p{\beta }_j^2$ is the L2 penalty: the sum of the squared values of all coefficients (${\beta }_1$ to ${\beta }_p$). Note that the intercept term ${\beta }_0$ is usually not penalized.
• $\lambda$ (lambda) is a hyperparameter that controls the strength of the regularization.
  • If $\lambda =0$, the penalty term disappears, and Ridge regression becomes identical to ordinary least squares (OLS).
  • As $\lambda$ increases, the penalty becomes more influential, and the model's coefficients are "shrunk" more aggressively towards zero.
  • If $\lambda \to \infty$, all coefficients will approach zero, resulting in a flat line (underfitting).

The optimal value of $\lambda$ is usually found through cross-validation. In scikit-learn, this hyperparameter is named alpha.

To see how this works in practice, let's continue with the StatQuest video. It gives a fantastic numerical example of how the penalty term influences the final model.

A key property of Ridge regression is that it shrinks coefficients towards zero, but it will never set them exactly to zero (unless $\lambda$ is infinite). This means Ridge regression always keeps all features in the model; it only reduces their influence.

3. Lasso Regression (L1 Regularization)

Lasso Regression (Least Absolute Shrinkage and Selection Operator) works similarly, but it uses a different penalty: the sum of the absolute values of the coefficients. This is known as an L1 penalty.

The Lasso cost function is:
$J_{Lasso}(\beta )={\sum }_{i=1}^n(y_i-{\hat{y}}_i{)}^2+\lambda {\sum }_{j=1}^p|{\beta }_j|$

The only difference is the penalty term: $\lambda \sum |{\beta }_j|$ instead of $\lambda \sum {\beta }_j^2$. This might seem like a small change, but it has a profound effect.

The crucial difference is that Lasso can shrink some coefficients all the way to zero. This means Lasso can perform automatic feature selection, effectively removing useless features from the model by making their coefficients zero. This results in "sparse" models (models with few non-zero coefficients), which are often simpler and more interpretable.

The next video clearly explains this fundamental difference.

4. Geometric Intuition: Why Lasso Performs Feature Selection

You might be wondering why the L1 penalty leads to zero coefficients while the L2 penalty doesn't. A geometric interpretation makes this crystal clear.

Imagine a model with two coefficients, ${\beta }_1$ and ${\beta }_2$. Regularization constrains the possible values of these coefficients to a certain "budget."

• Ridge (L2): The constraint is ${\beta }_1^2+{\beta }_2^2\le s$, which defines a circle.
• Lasso (L1): The constraint is $|{\beta }_1|+|{\beta }_2|\le s$, which defines a diamond.

The model tries to find the coefficients within this budget that get closest to the OLS solution (the point that minimizes the RSS). This is like expanding an ellipse (representing the contours of the RSS function) until it just touches the constraint region.

As you can see in the image:

• The circular Ridge constraint is smooth. The ellipse is likely to touch it at a point where neither ${\beta }_1$ nor ${\beta }_2$ is zero.
• The diamond-shaped Lasso constraint has sharp corners that lie on the axes. The ellipse is very likely to touch one of these corners, forcing one of the coefficients to be exactly zero.

This visual explanation is a powerful way to understand the fundamental difference in behavior between the two methods.

5. Practical Implementation and Comparison

Now that you understand the theory, let's focus on the practical side. How do you choose between Ridge and Lasso, and how do you implement them in Python?

A crucial prerequisite for any regularized linear model is feature scaling. Because the penalty is based on the magnitude of the coefficients, features with larger scales (e.g., house size in square feet) would be unfairly penalized compared to features with smaller scales (e.g., number of rooms). Using StandardScaler from scikit-learn is standard practice.

The following guide provides a concise comparison and a complete Python implementation.

To summarize the key points from the article:

Here's how you would typically implement them using scikit-learn:

from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge, Lasso
from sklearn.model_selection import train_test_split

# Assume X and y are your feature matrix and target vector

# 1. Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 2. Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# 3. Fit Ridge model
# alpha is the equivalent of lambda (λ)
ridge_model = Ridge(alpha=1.0) 
ridge_model.fit(X_train_scaled, y_train)
print("Ridge Coefficients:", ridge_model.coef_)

# 4. Fit Lasso model
lasso_model = Lasso(alpha=0.1)
lasso_model.fit(X_train_scaled, y_train)
print("Lasso Coefficients:", lasso_model.coef_)

Conclusion

Congratulations! You've just learned one of the most important concepts for building robust machine learning models. Regularization is a standard tool for preventing overfitting and improving the generalizability of your models.

Key Takeaways:

• Regularization prevents overfitting by adding a penalty term to the cost function, discouraging large coefficients.
• Ridge Regression (L2) uses a squared penalty ($\lambda \sum {\beta }_j^2$). It shrinks all coefficients towards zero but does not eliminate any features. It's a great general-purpose regularizer.
• Lasso Regression (L1) uses an absolute value penalty ($\lambda \sum |{\beta }_j|$). It can shrink coefficients all the way to zero, performing automatic feature selection and creating sparse, interpretable models.
• The regularization strength is controlled by the hyperparameter $\lambda$ (or alpha in scikit-learn), which must be tuned.
• Feature scaling is a critical preprocessing step before applying regularized models.

Preview of the Next Lesson:
So far in this module, we have covered Simple Linear Regression, Multiple Linear Regression, Polynomial Regression, and now Ridge and Lasso Regularization. You have the building blocks for creating a variety of powerful regression models. In the next lesson, we will focus on the practical workflow: how to systematically train and evaluate these regression models using scikit-learn. We will consolidate what you've learned and establish a robust framework for building, tuning, and assessing your models' performance.