L1 vs. L2 Regularization: Impact on Model Parameters

In our last lesson, we explored the bias-variance tradeoff and established that overly complex models tend to overfit by learning noise from the training data, leading to high variance. This sets the stage perfectly for today's topic, where we'll examine a powerful set of techniques designed specifically to combat this problem: regularization.

This lesson will focus on two of the most common types of regularization, L1 (Lasso) and L2 (Ridge). Understanding their mechanisms and a clear articulation of their differences is a staple of machine learning interviews. By the end of this lesson, you will be able to compare L1 and L2 regularization and confidently explain their distinct effects on a model's parameters, equipping you with the precise language needed for your upcoming interview.

The Core Idea: Penalizing Complexity

In the context of linear models, complexity often comes from the model's coefficients (or weights). If a model has very large coefficients, it means that small changes in the input features can cause large changes in the output, which is a hallmark of a high-variance, overfit model.

Regularization works by adding a penalty term to the model's loss function. This penalty is based on the size of the coefficients. The model is then optimized to minimize not just the original loss (like the Residual Sum of Squares, or RSS), but the combined loss and penalty.

The standard loss function for linear regression aims to minimize the RSS:

Loss_{RSS} = i = 1 \sum n (y_{i} - y^_{i})^{2} = i = 1 \sum n (y_{i} - (β_{0} + j = 1 \sum p β_{j} x_{ij}))^{2}

With regularization, the objective becomes:

New Loss = Loss_{RSS} + Penalty Term

This forces the model into a trade-off: it can only fit the training data well (minimizing RSS) if it keeps its coefficients small (minimizing the penalty).

L2 Regularization (Ridge Regression)

L2 regularization, also known as Ridge regression, adds a penalty proportional to the sum of the squared values of the coefficients. The objective function becomes:

Loss_{Ridge} = i = 1 \sum n (y_{i} - y^_{i})^{2} + λ j = 1 \sum p β_{j}^{2}

Here:

$β_{j}$ are the model's coefficients.
$λ$ (lambda, sometimes written as $α$ ) is a hyperparameter that controls the strength of the penalty. A higher $λ$ imposes a greater penalty, forcing the coefficients to become smaller.

The key effect of the squared term is that L2 regularization prefers to shrink all coefficients towards zero, but it rarely sets them exactly to zero. It keeps all features in the model but reduces their influence. This is particularly useful for dealing with multicollinearity (highly correlated features), as it tends to distribute the weight more evenly among them instead of putting a large weight on just one.

The following video from StatQuest provides a fantastic visual intuition for how this works.

Ridge vs Lasso Regression, Visualized!!!

This video brilliantly visualizes how the cost function of a linear model is altered by the regularization penalty.

First, watch the initial setup, which shows how the sum of squared residuals forms a parabola, with the minimum point representing the best-fit model. Then, pay close attention to the section on the Ridge penalty. Notice how as the lambda value increases, the minimum of the new, penalized cost function gets "shrunk" closer and closer to a slope of zero, but never quite reaches it.

L1 Regularization (Lasso Regression)

L1 regularization, or Lasso (Least Absolute Shrinkage and Selection Operator), adds a penalty proportional to the sum of the absolute values of the coefficients. The objective function is:

Loss_{Lasso} = i = 1 \sum n (y_{i} - y^_{i})^{2} + λ j = 1 \sum p ∣ β_{j} ∣

This seemingly small change from squaring to taking the absolute value has a profound effect. Because of the nature of the absolute value function, L1 regularization can—and often does—shrink the coefficients of less important features all the way to exactly zero.

This means L1 regularization performs automatic feature selection, effectively removing irrelevant features from the model. This results in a "sparse" model, which can be simpler and more interpretable.

Let's return to the StatQuest video to see this in action.

Ridge vs Lasso Regression, Visualized!!!

Now let's see how the L1 penalty differs from L2.

Watch the segment on the Lasso penalty. The most important thing to observe is the "kink" that the absolute value penalty introduces into the cost function at zero. Notice how, with a strong enough lambda, this kink can become the lowest point, making the optimal coefficient exactly zero.

The Geometric Interpretation: Why L1 Creates Sparsity

To truly understand why L1 produces sparse models and L2 doesn't, we can look at the problem geometrically. The process of minimizing the regularized loss function is equivalent to a constrained optimization problem: finding the model coefficients ( $β$ ) that minimize the RSS, subject to the constraint that the penalty term is less than some constant $c$ .

For L2 (Ridge), the constraint is $\sum β_{j}^{2} \leq c$ , which defines a circular region (in 2D) or a hypersphere (in higher dimensions).
For L1 (Lasso), the constraint is $\sum ∣ β_{j} ∣ \leq c$ , which defines a diamond shape (in 2D) or a hyper-diamond (polytope) in higher dimensions.

The solution is the point where the elliptical contours of the RSS first "touch" this constraint region.

This image contrasts the geometric view of L2 (left) and L1 (right) regularization for a model with two weights, w1 and w2. The elliptical lines are the contours of the original loss function, with the unregularized minimum at the center. The circular (L2) and diamond (L1) shapes represent the regularization penalty constraint. The regularized solution is the point where the contours first touch the constraint shape. For L1, this is highly likely to occur at a corner, forcing one of the weights to be zero.

As you can see, the circular L2 constraint region has no sharp corners. The loss function is likely to touch it at a point where both coefficients are non-zero. In contrast, the diamond-shaped L1 constraint region has sharp corners that lie on the axes. It is much more probable that the expanding ellipses of the loss function will hit one of these corners, forcing the corresponding coefficient to be exactly zero.

The following video provides an excellent walkthrough of this geometric view, framed specifically for interview prep.

L1 and L2 Regularization in Machine Learning: Easy Explanation for Data Science Interviews

This video from Emma Ding provides a clear, interview-focused explanation of the geometry behind L1 and L2.

Watch the explanation of L1's geometric interpretation. Focus on how the diamond-shaped constraint region leads to solutions where one of the coefficients becomes zero. Next, watch the section on L2's geometric interpretation to see why the circular constraint region leads to non-zero coefficients.

L1 vs. L2: A Head-to-Head Comparison

For an interview, you should be ready to summarize the key differences concisely. The article "Understanding L1 and L2 regularization" provides a great overview.

Understanding L1 and L2 regularization: techniques for ...

This section from a Weights & Biases report provides a detailed comparison of the two techniques, covering their effects on coefficients, multicollinearity, interpretability, and optimization.

Please read the section titled Comparing L1 and L2 Regularization. As you read, focus on building a mental table that contrasts the two methods on these key aspects: Coefficient shrinkage and sparsity Feature selection capability How they handle correlated features (multicollinearity) Resulting model complexity and interpretability

Here is a summary table of the most critical distinctions:

Feature	L1 Regularization (Lasso)	L2 Regularization (Ridge)
Penalty Term	Sum of absolute coefficient values $(\lambda \sum	\beta_j
Effect on Weights	Can shrink coefficients to exactly zero.	Shrinks coefficients towards zero, but they remain non-zero.
Feature Selection	Performs automatic feature selection, creating sparse models.	Does not perform feature selection; keeps all features.
Sparsity	Produces sparse models (many zero coefficients).	Produces dense models (all non-zero coefficients).
Multicollinearity	Tends to arbitrarily select one feature among correlated ones.	More stable; distributes weights among correlated features.
Best Use Case	When you want feature selection or have many irrelevant features.	When you believe all features are relevant and/or have multicollinearity.
Computation	The penalty is not differentiable at zero, requiring specialized solvers.	The penalty is differentiable everywhere, making it computationally simpler.

Conclusion

In this lesson, we've broken down L1 and L2 regularization, two fundamental techniques for controlling model complexity and preventing overfitting. You are now equipped to explain not just what they do, but how and why they do it.

Key Takeaways for Your Interview:

Regularization's Goal: To prevent overfitting by adding a penalty for large coefficients to the loss function, thereby managing the bias-variance tradeoff.
L2 (Ridge): Uses a squared penalty $(β^{2})$ . It shrinks coefficients towards zero, providing stability, especially in the presence of correlated features. It does not perform feature selection.
L1 (Lasso): Uses an absolute value penalty $(∣ β ∣)$ . It can shrink coefficients to exactly zero, performing automatic feature selection and producing sparse, more interpretable models.
The Geometric View: The difference in their behavior comes down to the shape of their constraint regions—a circle for L2 and a diamond for L1. The sharp corners of the L1 diamond are what enable feature selection.

In our next lesson, we will shift our focus from regression to classification and explore logistic regression. You'll see that regularization is a general concept that applies there as well, helping to create robust classification models.