Learning Curves for Diagnosing Overfitting and Underfitting

What is a Learning Curve?

At its core, a learning curve is a plot that shows a model's performance as a function of the number of training examples used to train it. Typically, we plot two curves on the same axes:

1. Training Error: The error of the model measured on the same data it was trained on.
2. Validation Error: The error of the same model measured on a separate, unseen validation set.

The x-axis represents the size of the training set, and the y-axis represents the error (or another performance metric).

To generate these curves, we follow a specific procedure:

• We take progressively larger subsets of our full training dataset (e.g., 10%, 20%, ..., 100%).
• For each subset size, we train a new model from scratch using only that data.
• We then evaluate this trained model on both the subset it was trained on (to get a point for the training error curve) and on the entire, fixed validation set (to get a point for the validation error curve).
• Plotting these error values against the subset sizes gives us our learning curves.

Let's look at the typical shape of these curves.

Why do they have this shape? The following video provides an excellent and detailed explanation. It will walk you through the logic behind constructing these curves and the intuition for why they behave the way they do.

As the video explained, it's easy for a model to perfectly memorize a tiny dataset, resulting in near-zero training error. As the dataset grows, memorization becomes harder, so the training error increases. Conversely, a model trained on very little data generalizes poorly, leading to high validation error. As the model sees more data, it learns a more robust representation of the underlying pattern, and its validation error decreases.

A Quick Clarification

You may also encounter plots that show a model's loss decreasing over training epochs or iterations. While also called "learning curves" in some contexts (especially deep learning), they serve a different purpose: monitoring the optimization process of a single model. The technique we are discussing today—plotting error against training set size—is the standard method for diagnosing bias and variance.

Diagnosing Model Performance with Learning Curves

The real power of learning curves comes from interpreting their shape and the relationship between the two curves. They provide a clear visual signature for both high bias and high variance.

The following presentation slides offer a concise summary of how to read these signatures.

Let's break down these two key scenarios.

Scenario 1: High Bias (Underfitting)

A model with high bias is too simple for the data. Its learning curve has a distinct signature:

• The training error is high. It plateaus at an unacceptably high error level because the model is not even powerful enough to fit the training data well.
• The validation error is also high and converges to a value very close to the training error.
• The gap between the two curves is small.

This learning curve, adapted from the Cornell lecture, is characteristic of a high bias problem. Both training (blue) and validation (red) errors plateau at a high level, far from the desired performance (green).

Diagnosis: When you see this pattern, your model is underfitting. It has a fundamental inability to capture the complexity of the data. The most important takeaway is that getting more training data will not help. The curves have already flattened out; more data along the x-axis won't bring the error down.

Scenario 2: High Variance (Overfitting)

A model with high variance is too complex; it memorizes the training data instead of learning the general pattern. Its learning curve looks very different:

• The training error is very low. The model has no trouble fitting the training data perfectly.
• The validation error is much higher than the training error.
• There is a large gap between the training and validation curves.

This is the signature of high variance. There is a significant gap between the low training error and the high validation error. The validation error is still decreasing, suggesting the model could improve.

Diagnosis: This is a classic case of overfitting. The model is not generalizing well to unseen data. However, unlike the high-bias case, there is hope. The validation curve is still trending downwards, and the gap is narrowing as the training set grows. This indicates that getting more training data is likely to help improve the model's performance.

The same Cornell lecture video provides an excellent summary of interpreting these two patterns.

From Diagnosis to Action

Diagnosing the problem is the first step. The next is to fix it. Your diagnosis directly informs your strategy. The scikit-learn documentation provides a fantastic overview and practical code examples.

As you saw in the documentation and the resources above, the appropriate action depends entirely on the diagnosis:

If you diagnose High Bias (Underfitting):

• Increase model complexity: Use a more powerful model (e.g., switch from linear regression to a polynomial one, or use a deeper decision tree).
• Add more features: Engineer new features that might help the model capture the underlying signal.
• Decrease regularization: Regularization is designed to prevent overfitting, so reducing its strength can help a high-bias model become more flexible.

If you diagnose High Variance (Overfitting):

• Get more training data: This is often the most effective remedy.
• Decrease model complexity: Use a simpler model or reduce the complexity of your current one (e.g., use a shallower decision tree).
• Increase regularization: Add or strengthen regularization (like L1 or L2, which we'll cover later) to penalize model complexity.
• Use feature selection: Reduce the number of features to prevent the model from focusing on noise.

To see these concepts applied in a full coding example, the last part of the Cornell lecture is highly recommended.

Conclusion

Today, we've moved from the theory of the bias-variance tradeoff to a concrete, practical tool for diagnosing model performance. Learning curves are an indispensable part of the machine learning practitioner's toolkit.

Key Takeaways:

• Learning curves plot a model's training and validation error as a function of training set size.
• High Bias (Underfitting) is diagnosed when both training and validation error are high and converge. The primary solution is to increase model complexity, not to add more data.
• High Variance (Overfitting) is diagnosed by a large gap between a low training error and a high validation error. The primary solution is often to add more training data or decrease model complexity.
• By interpreting learning curves, you can make informed decisions about how to invest your time and resources to improve your model.

In this lesson and the last, we've focused on the nature of error (bias vs. variance) and how to diagnose it. In our next lesson, we will focus on the quantification of error. We will explore standard evaluation metrics for regression tasks, such as Mean Squared Error (MSE), Mean Absolute Error (MAE), and R-squared (R²), which allow us to precisely measure and compare the performance of different models.