L1 vs. L2 Regularization: Impact on Coefficient Sparsity

1. The Shape of the Penalty: A Visual Intuition

The fundamental difference between L1 and L2 regularization lies in the shape of their penalty functions. As a reminder:

• L2 (Ridge) Penalty: $\lambda {\sum }_{j=1}^p{w}_j^2$ (a squared term)
• L1 (Lasso) Penalty: $\lambda {\sum }_{j=1}^p|w_j|$ (an absolute value term)

When we add these penalties to our original loss function (like MSE, which is parabolic), the location of the new minimum changes in distinct ways. L2's squared penalty is also a parabola, so adding it to the MSE parabola results in another smooth parabola. Its minimum is shifted closer to zero, but it rarely lands exactly on it.

L1's absolute value penalty, however, is a 'V' shape. When you add a V-shape to a parabola, you can get a sharp "kink" at the bottom. For a strong enough regularization parameter lambda_, the lowest point of this new combined function can fall exactly at this kink, where the weight is zero.

The "StatQuest" video provides an excellent animated visualization of this exact process.

2. The Geometric View: A Tale of a Diamond and a Circle

We can extend this intuition to a model with two weights, $w_1$ and $w_2$. In this case, minimizing the regularized loss is equivalent to minimizing the original loss (e.g., MSE) subject to a constraint:

• L1 Constraint: $|w_1|+|w_2|\le C$. This defines a diamond shape in the weight space.
• L2 Constraint: $w_1^2+w_2^2\le C$. This defines a circle.

The unregularized loss function forms a set of concentric ellipses (contours) in the weight space. The optimal solution is the point where the loss contour first touches the constraint region.

As you can see, the ellipses are more likely to touch the L1 diamond at one of its corners, which lie on the axes. A point on an axis means one of the weights is exactly zero. The L2 circle, being smooth, has no corners, so the contact point can be anywhere on its boundary, usually where both weights are non-zero.

This geometric view is a cornerstone of understanding regularization. The following resource provides a clear explanation of this visualization.

3. The Gradient's Perspective: A Constant Push vs. a Gentle Nudge

Since you have implemented the gradients yourself, let's connect this behavior directly to your code. The difference arises from the derivative of the penalty term.

• Gradient of L2 penalty: $\frac{\partial }{\partial w_j}(\frac{\lambda }{2n}{w}_j^2)=\frac{\lambda }{n}{w}_j$. This term is proportional to the weight w_j itself. As the weight gets smaller and approaches zero, the regularization gradient also shrinks to zero. It gives a gentle nudge, but the push weakens as you get closer to the target.

• Gradient of L1 penalty: $\frac{\partial }{\partial w_j}(\frac{\lambda }{n}|w_j|)=\frac{\lambda }{n}\text{sign}(w_j)$. This term is constant! It's either $+\frac{\lambda }{n}$ or $-\frac{\lambda }{n}$ (or undefined at zero). This means that even when a weight is very small (e.g., 0.0001), gradient descent still subtracts a fixed amount (learning_rate * \lambda / n) from it on every step. This constant, relentless push is what eventually forces the weight to cross the zero line and stay there.

The following article offers a great summary of this gradient-based reasoning.

4. A Practical Experiment in Code

Let's prove this with the LinearRegression class you modified in the last lesson. We'll create a synthetic dataset where some features are useful and others are pure noise. Then we'll see which coefficients L1 and L2 regularization decide to keep.

Assuming your modified LinearRegression class from the previous lesson is saved in a file named linear_regression.py, you can run the following script:

import numpy as np
# Make sure your updated LinearRegression class is accessible
from linear_regression import LinearRegression 

# 1. Generate synthetic data
# Let's create a dataset with 5 features.
# The target y will only depend on the first two features.
# The last three features are irrelevant noise.
np.random.seed(42)
X = np.random.rand(100, 5) * 10
# y = 3*X0 + 5*X1 + noise. True weights are [3, 5, 0, 0, 0]
y = 3 * X[:, 0] + 5 * X[:, 1] + np.random.randn(100) * 2

# 2. Train an L1 (Lasso) regularized model
# We use a relatively high lambda to encourage sparsity
lasso_model = LinearRegression(learning_rate=0.01, n_iters=1000, regularization='l1', lambda_=1.0)
lasso_model.fit(X, y)

# 3. Train an L2 (Ridge) regularized model
ridge_model = LinearRegression(learning_rate=0.01, n_iters=1000, regularization='l2', lambda_=1.0)
ridge_model.fit(X, y)

# 4. Compare the final weights
print("--- Comparing L1 and L2 Effects on Sparsity ---")
print(f"True weights should be close to: [3, 5, 0, 0, 0]\n")

print("L2 (Ridge) Coefficients:")
print(np.round(ridge_model.weights, 4))
print("--> Notice all coefficients are small but non-zero.\n")

print("L1 (Lasso) Coefficients:")
print(np.round(lasso_model.weights, 4))
print("--> Notice the last three coefficients are exactly zero!")

When you run this code, you should see that the L2 model assigns small, non-zero weights to all features. The L1 model, however, should drive the weights for the noisy features (the last three) all the way to zero, effectively performing automatic feature selection.

Conclusion

You have now explored the crucial differences between L1 and L2 regularization from multiple angles. This understanding elevates you from someone who can simply implement regularization to someone who can reason about when and why to use each type.

Key Takeaways:

Preview of the Next Module:
We have now completed our deep dive into linear models, optimization, and regularization. In the next module, we will explore a powerful and mathematically elegant approach to classification: Support Vector Machines (SVMs). We'll begin by building the intuition behind maximum margin classification and understanding the critical role of "support vectors" in defining a decision boundary.