Mitigating Overfitting and Underfitting

1. Mitigating Underfitting: When Your Model Is Too Simple

Underfitting is often the more straightforward problem to solve. If your model performs poorly even on the data it was trained on, it's a clear sign that it's too simple to capture the underlying patterns. The solution, therefore, is to give the model more power or more information.

The IBM article "What Is Overfitting vs. Underfitting?" provides a great overview of these strategies.

Based on the reading, here are the key strategies to combat underfitting:

• Increase Model Complexity: If a simple linear model isn't working, perhaps the relationship isn't linear. You could try a higher-degree polynomial regression, or if you're using neural networks, you could add more layers or more neurons per layer.
• Add More Features (Feature Engineering): Your model might be underfitting because it doesn't have the right information. By creating new features from your existing data (e.g., interaction terms like price_per_square_foot from price and square_footage), you can provide the model with more predictive signals.
• Reduce Regularization: We will cover regularization in detail in the next section, but in short, it's a technique that constrains model complexity to prevent overfitting. If your model is underfitting, you might be constraining it too much. Reducing the strength of regularization gives the model more flexibility.
• Train for Longer: Sometimes a model simply needs more time to learn. By increasing the number of training epochs (passes through the dataset), you give the algorithm more opportunities to adjust its weights and converge to a better solution.

2. Mitigating Overfitting: When Your Model Knows Too Much

Overfitting is a more common and subtle challenge in machine learning. Your model has learned the training data perfectly—including the noise—and has lost its ability to generalize. Our goal is to prevent it from memorizing and encourage it to learn the true underlying patterns.

Let's explore the main strategies.

Strategy 1: Increase Training Data

The most effective way to fight overfitting is to train the model on more data. With more examples, the model has a harder time memorizing every single data point and is forced to learn the general patterns that hold true across the entire dataset.

However, collecting new data is often expensive or impossible. A practical alternative, especially in fields like computer vision, is data augmentation.

Strategy 2: Regularization

This is perhaps the most important and widely used set of techniques for combating overfitting. Regularization works by adding a penalty to the model's loss function for being too complex. In practice, this means penalizing large coefficient (weight) values.

Why penalize large weights? A model with large weights can make sharp, drastic changes in its predictions based on small changes in the input features. This is a sign that it's relying too heavily on specific features, essentially memorizing their effect. By penalizing large weights, we encourage the model to use all features more modestly, leading to a smoother, more generalizable function.

The following article provides a fantastic, comprehensive guide to the main regularization techniques.

Let's summarize the three main types of regularization from the article:

• L2 Regularization (Ridge Regression):
  • Penalty: Adds the sum of the squared coefficients to the loss function: $\lambda \sum w_j^2$.
  • Effect: It shrinks all coefficients towards zero but never forces them to be exactly zero. It's great at handling situations where many features are correlated and you believe all are somewhat useful.
• L1 Regularization (Lasso Regression):
  • Penalty: Adds the sum of the absolute value of the coefficients: $\lambda \sum |w_j|$.
  • Effect: It can shrink some coefficients all the way to exactly zero. This makes it a powerful tool for automatic feature selection, as it effectively removes irrelevant features from the model.
• Elastic Net:
  • Penalty: A combination of both L1 and L2 penalties.
  • Effect: It offers a middle ground, providing the benefits of both. It can perform feature selection like Lasso while also handling correlated features gracefully like Ridge.

The parameter $\lambda$ (often called alpha in Python libraries like scikit-learn) controls the strength of the regularization. A larger $\lambda$ means a stronger penalty and a simpler model.

Strategy 3: Early Stopping

Early stopping is a simple and effective regularization technique. As we saw in the previous lesson's learning curves, an overfit model's performance on the validation set will eventually stop improving and start to get worse. Early stopping simply says: "Let's stop training at that point!"

You monitor the validation error during training and stop the process when the error hasn't improved for a certain number of epochs. This prevents the model from continuing into the overfitting phase.

Strategy 4: Dropout (for Neural Networks)

Dropout is a regularization technique specifically designed for neural networks. During training, it randomly "drops out" (sets to zero) a fraction of the neurons in a layer for each training step.

This forces the network to learn redundant representations and prevents neurons from co-adapting too much, meaning no single neuron can rely on the presence of another. It's like forcing a team to work together where any member might be absent at any given time, making the whole team more resilient and less dependent on individuals.

3. Practical Best Practices

When applying these strategies, especially regularization, there are a couple of crucial rules to follow.

The two most important takeaways from this section are:

1. Always Scale Your Features: Regularization penalizes the magnitude of coefficients. If your features are on different scales (e.g., age from 20-80 and income from 30,000-200,000), the penalty will unfairly affect the feature with the larger scale. You must standardize your data (e.g., using StandardScaler from scikit-learn, which you'll remember from Module 5) before applying regularization.
2. Tune Your Hyperparameters: The strength of regularization ($\lambda$ or alpha) is a hyperparameter you must choose. You can't just guess! The best value is found using techniques like cross-validation, which we covered earlier.

Conclusion

You now have a solid understanding of the main strategies for managing the bias-variance tradeoff. Knowing how to both diagnose and treat underfitting and overfitting is a fundamental skill that separates beginners from proficient machine learning practitioners.

Key Takeaways:

• To fix underfitting (high bias):
  • Increase model complexity (e.g., more features, more layers).
  • Add more informative features through feature engineering.
  • Reduce the strength of regularization.
• To fix overfitting (high variance):
  • Get more data or use data augmentation.
  • Use regularization (L1, L2, Elastic Net) to penalize model complexity.
  • Use early stopping to halt training before the model starts to overfit.
  • For neural networks, use dropout.
  • Always scale your data before applying regularization and tune hyperparameters using cross-validation.

Preview of the Next Lesson:
This lesson was the perfect theoretical setup for our next module, "Supervised Learning: Regression". In the very first lessons of that module, we will implement linear regression and then immediately apply Ridge (L2) and Lasso (L1) regularization to see these concepts in action with Python and scikit-learn. You'll get to see firsthand how adding regularization improves a model's performance on unseen data.