Diagnosing Underfitting and Overfitting from Training and Validation Results

Hello. Last lesson, you designed splits that make an unseen-data check credible. Now you can use those splits to answer a different question: has a model learned too little, learned useful patterns, or fitted its training examples too closely?

This lesson focuses on diagnosing underfitting and overfitting from training and validation results. You’ll read loss values, interpret curves, and decide what to investigate next. Allow about 35–40 minutes.

What the two results tell you

Training results show how well a model handles examples it was allowed to learn from. Validation results show how well it handles held-out examples during development. The distinction matters because a model can perform impressively on familiar data without making useful predictions on new requests.

Underfitting means the model has not captured enough of the useful pattern: its training performance is poor, and validation performance is usually poor too. Overfitting means it performs well on training examples but substantially worse on validation examples. A model that fits well has useful validation performance, not necessarily a perfect training score.

Read Google for Developers’ Overfitting page for its concise account of these terms and its explanation of paired loss curves.

To compare two results, first make sure they measure the same thing. For example, compare training loss with validation loss computed using the same loss definition; lower loss means better performance. If instead you compare accuracy, higher is better. The absolute meaning of “good” also depends on the task: a loss of 0.5 is not automatically acceptable or unacceptable without a reference point.

Read a pair of loss curves

The graph below plots training and validation loss against training iterations. At first, both curves fall: the model is improving on familiar and held-out examples. Later, training loss continues to fall while validation loss turns upward. More training is improving the model’s fit to the training set while worsening its observed performance on the validation set.

The useful checkpoint is near the lowest validation loss, not necessarily the final iteration or the point with the lowest training loss. Real curves can fluctuate, so one small upward step is less persuasive than a sustained deterioration. Keeping a checkpoint selected using validation results is called early stopping; it does not make the final test set part of training.

Watch the indicated portions of Underfitted’s A Critical Skill People Learn Too LATE: Learning Curves In Machine Learning for a visual explanation. The video calls its held-out curve a testing curve. In the development process from the previous lesson, use a validation curve for this diagnosis and leave the test set untouched.

A gap between curves matters, but a small gap does not by itself mean a good model. If both losses remain high, the model may simply be doing poorly everywhere. Conversely, some training–validation gap is normal: the model was fitted to the training examples, not the validation examples.

Diagnose results rather than labels

Suppose you predict whether an arriving request will need specialist review. The following are hypothetical measurements of the same loss, on the same valid split, for three training checkpoints. Lower is better; a simple fixed prediction rule has validation loss of about 0.60.

The early checkpoint’s curves are close together, but its validation loss is no better than the simple reference. At the late checkpoint, the training result is excellent, yet the middle checkpoint makes better held-out predictions. Choose based on validation performance, then use the reserved test set for the final check of the chosen approach.

These labels describe evidence, not guaranteed causes. Persistently high training loss could mean a model is too restricted to represent the pattern, that informative features are missing, or simply that optimization has not progressed far enough. A large training–validation gap could reflect fitting noise in a small training set, but it could also warrant checking whether validation data differs substantially from training data. And unexpectedly outstanding validation results are a reason to revisit the leakage checks from the previous lesson—not proof that the model has generalized.

There is another useful kind of curve. So far, the horizontal axis has represented training iterations. A curve against number of training examples asks a different question: would more representative data help? Read the selected parts of scikit-learn’s Validation curves: plotting scores to evaluate models to see how these diagnoses relate to model flexibility and dataset size. Its examples use scores, where higher is better, rather than loss.

In broad terms, a model that is too constrained may have high bias: it cannot fit even the training pattern well. One that is highly sensitive to the particular training examples may have high variance: it can fit them closely without performing as well on new examples. These are helpful explanations for observed results, not diagnoses you can establish from one number alone. More data may help a model with a substantial training–validation gap, but adding data will not necessarily repair a model that cannot learn the training pattern in the first place.

A practical diagnostic sequence

When a result looks disappointing, keep the investigation anchored to the actual prediction setting:

1. Check comparability. Use the same performance measure and target definition for training and validation. Confirm that validation rows and features obey the availability and split rules from the previous lesson.
2. Look at training performance first. If it remains poor, investigate insufficient training, missing useful inputs, or a model that is too restrictive. Do not treat more training data as the automatic remedy.
3. Then examine validation performance and the gap. If training is strong but validation is weak, investigate overfitting and whether the two sets represent the same intended prediction situation. If validation worsens as training continues, consider an earlier checkpoint.
4. Change one plausible cause and recheck validation. For example, a less restrictive model might address underfitting; a simpler model, stronger constraints, more representative data, or an earlier checkpoint might address overfitting. The diagnosis guides an experiment rather than guaranteeing its outcome.

Throughout, reserve the test set for evaluating the selected approach. Repeatedly choosing changes because they improve test results would turn the test set into another validation set.

Key takeaways

Poor training and validation performance points toward underfitting; strong training performance paired with substantially weaker validation performance points toward overfitting. When both losses initially fall but validation loss later rises, the best checkpoint is usually near the validation minimum—not the end of training. Interpret gaps alongside the absolute results, the task’s reference performance, and the quality of the split.

Next, you’ll examine which evaluation metric reflects the real cost of false positives and false negatives. That will give “good validation performance” a more useful meaning for a decision.