Regularization for Linear Models

The Problem of Overfitting

When we train a model, we want it to learn the underlying patterns in the data, not the noise. If a model becomes too complex, it can essentially "memorize" the training examples, including their random quirks. This leads to excellent performance on the training set but poor performance on any new data. This phenomenon is called overfitting. In contrast, a model that is too simple might not even capture the underlying pattern, which is called underfitting.

A model's complexity is often related to the magnitude of its weights. Large weights can cause the model's output to change dramatically in response to small changes in the input features, leading to a jagged, unstable decision boundary that fits the training noise. The core idea of regularization is to discourage the learning algorithm from finding solutions with excessively large weights.

The following video provides an excellent high-level introduction to these concepts and sets the stage for how regularization helps find a better balance.

L2 Regularization (Ridge Regression)

The most common type of regularization is L2 regularization, also known as Ridge regression in the context of linear regression. It modifies the cost function by adding a penalty term proportional to the sum of the squares of the weight values.

Our original cost function for logistic regression (Binary Cross-Entropy) over $m$ samples was:

With L2 regularization, the new cost function becomes:

Let's break down the new term:

• $\mathbf{w}$ is the vector of our model's weights. Notice that the bias term $b$ is typically not regularized.
• ${\sum }_{j=1}^n{w}_j^2$ is the squared Euclidean norm (or L2 norm) of the weight vector, $||\mathbf{w}|{|}_2^2$.
• $\lambda$ (lambda) is the regularization hyperparameter. It controls the strength of the penalty. A larger $\lambda$ means stronger regularization, forcing the weights to be smaller. If $\lambda =0$, we recover the original cost function.

Effect on Gradient Descent

Given your calculus background, we can see exactly how this changes the learning algorithm. To update the weights, we need the new partial derivatives. The derivative of the cost function with respect to a weight $w_j$ now includes a term from the penalty:

The gradient descent update rule for $w_j$ thus becomes:

We can rearrange this to see something interesting:

This shows that in every step, before we apply the regular gradient update, we first shrink the weight $w_j$ by a factor of $(1-\alpha \frac{\lambda }{m})$. This is why L2 regularization is often called weight decay. It consistently pulls the weights towards zero, preventing any single weight from growing too large.

For a deeper analytical and probabilistic understanding, the following document explains L2 regularization as arising from placing a Gaussian prior on the model's weights in a Bayesian framework.

L1 Regularization (Lasso Regression)

A second, powerful type of regularization is L1 regularization, also known as Lasso (Least Absolute Shrinkage and Selection Operator). It differs from L2 by using the sum of the absolute values of the weights.

The L1-regularized cost function is:

Here, ${\sum }_{j=1}^n|w_j|$ is the L1 norm of the weight vector, $||\mathbf{w}|{|}_1$. While this looks like a small change, it has a profound effect: L1 regularization can shrink some weights to be exactly zero.

This means L1 can be used for feature selection, effectively removing irrelevant features from the model. This property of inducing sparsity is one of the most important differences between L1 and L2 regularization.

Geometric Intuition for Sparsity

Why does L1 produce sparse solutions while L2 does not? The reason lies in the geometry of the penalty terms. Imagine we are trying to minimize the loss function (the ellipses in the diagram below) subject to a constraint that the regularization term is less than or equal to some constant $C$.

• For L1, the constraint is $||\mathbf{w}|{|}_1\le C$, which forms a diamond shape in two dimensions.
• For L2, the constraint is $||\mathbf{w}|{|}_2^2\le C$, which forms a circular shape.

The optimal solution, $w^{\ast }$, is the point where the contours of the loss function first touch the constraint region.

As you can see, with the sharp corners of the L1 diamond, it is highly probable that the optimal point will lie on an axis (e.g., where $w_1=0$). With the smooth L2 circle, the contact point can be anywhere on the boundary, making it very unlikely that any weight will be exactly zero.

The following video provides a fantastic visualization of this key difference and shows how the penalty term's shape influences the final solution.

Similar to L2, L1 regularization also has a probabilistic interpretation, which you can explore in the same document as before. It corresponds to placing a Laplace prior on the weights.

Applying Regularization in Practice

Now let's see how to implement this. We can modify our MyLogisticRegression class from the previous lesson, but in practice, we'll almost always use a library like scikit-learn.

Regularization in scikit-learn

The sklearn.linear_model.LogisticRegression class has built-in support for regularization. The key parameters are:

• penalty: A string that can be 'l1', 'l2', 'elasticnet', or 'none'.
• C: A positive float that represents the inverse of regularization strength. This is a crucial convention: smaller C means stronger regularization (equivalent to a larger $\lambda$).
• solver: Different optimization algorithms support different penalties. For example, 'liblinear' supports both L1 and L2, while 'lbfgs' only supports L2.

Here's how you would instantiate regularized models:

from sklearn.linear_model import LogisticRegression




# L2 Regularized Logistic Regression (default)
# C=1.0 is a moderate amount of regularization
l2_model = LogisticRegression(penalty='l2', C=1.0, solver='lbfgs')




# L1 Regularized Logistic Regression
# Note the change in solver
l1_model = LogisticRegression(penalty='l1', C=1.0, solver='liblinear')

The scikit-learn documentation provides a concise reference for the exact formulas used.

The following image perfectly illustrates the effect of L1 regularization as the hyperparameter C is varied. On the left, you can see how the accuracy on the training and test sets changes. On the right, you see the values of the model's coefficients.

This clearly demonstrates the power of L1 for feature selection and the importance of choosing the right value for the regularization hyperparameter.

Conclusion

In this lesson, we've tackled the critical problem of overfitting and introduced regularization as a primary solution. You now have a solid theoretical and practical understanding of the two most common forms of regularization.

Here are the key takeaways:

• Overfitting occurs when a model learns the training data too well, capturing noise instead of the underlying pattern, which leads to poor generalization.
• Regularization prevents overfitting by adding a penalty term to the cost function that discourages large model weights.
• L2 Regularization (Ridge) uses a penalty based on the squared magnitude of the weights ($||\mathbf{w}|{|}_2^2$). It shrinks weights towards zero, a process also known as weight decay.
• L1 Regularization (Lasso) uses a penalty based on the absolute value of the weights ($||\mathbf{w}|{|}_1$). It can shrink weights to be exactly zero, thus performing implicit feature selection.
• The strength of regularization is controlled by a hyperparameter ($\lambda$ or its inverse, $C$), which must be tuned to find the best balance between underfitting and overfitting.

The need to tune the regularization hyperparameter sets the stage perfectly for our next topic. To find the best value for C, we first need a robust way to measure our model's performance. In the next lesson, we will explore a suite of evaluation metrics for classification tasks, such as precision, recall, and the F1-score, moving beyond simple accuracy to get a more nuanced view of model performance.