Understanding Data Splits: Training, Validation, and Test Sets

The Goal of Generalization

The ultimate objective of a supervised learning model is not to perform well on the data it was trained on, but to generalize—to make accurate predictions on new, unseen data. A model that has simply memorized the training data might look perfect on paper but will be useless in practice. This gap between performance on seen versus unseen data is a central challenge in machine learning.

From a statistical standpoint, we are trying to build a model that minimizes the true risk or generalization error, which is the expected error on all possible data points from the underlying data distribution. However, we only have a finite sample of data. The error we calculate on this sample, the training error, is often an overly optimistic estimate of the true generalization error.

Your background in statistics and probability will be helpful here. The training error is a biased estimator of the generalization error. To understand why, let's explore a more formal treatment.

The key takeaway is that by fitting the model to the training data, we have used up some of the information in that data. The model becomes specifically adapted to the noise and quirks of that particular sample, a phenomenon known as overfitting. Therefore, evaluating the model on the same data it was trained on gives us a misleadingly low error.

The Basic Solution: A Train/Test Split

The most straightforward way to get a more honest assessment of our model's performance is to hold back a portion of our data. We split our dataset into two parts:

1. Training Set: The majority of the data, used to train the model. The model's parameters (e.g., the coefficients in a linear regression, or the weights of a neural network) are learned by minimizing a loss function on this set.
2. Test Set: A smaller, held-out portion of the data. This set is not used during training. It serves as a proxy for unseen data, and we use it to evaluate the final model's generalization performance.

This simple "hold-out" method is the first step towards robust model evaluation.

A high-level overview of data splitting. The training set is for building the model, while the test set is for evaluating it.

The Problem with a Two-Way Split: Hyperparameter Tuning

This two-set approach works if you have a single, fixed model. But what happens when you need to make design choices about the model itself?

• In a k-Nearest Neighbors model, what is the best value for k?
• In a polynomial regression model, what is the optimal degree of the polynomial?
• When using regularization (a topic we'll cover soon), what is the right amount of regularization strength (λ)?

These choices are not learned directly from the training data; they are hyperparameters that we, the developers, must set. A common approach is to try several different values for a hyperparameter, train a model for each, and see which performs best.

But where do we measure that performance? If we use the test set to compare these models and pick the best one, we introduce a subtle but critical flaw. We are now using the test set to make decisions that influence our final model. In doing so, information from the test set "leaks" into our model selection process. The test set is no longer truly "unseen," and our final evaluation on it will be biased and overly optimistic. We have inadvertently overfitted our hyperparameters to the test set.

To see a practical demonstration of this problem, let's watch a segment from a video by Greg Hogg.

The Gold Standard: Train, Validation, and Test Sets

To solve this problem, we introduce a third data split: the validation set (also sometimes called the development or dev set). This creates a robust, three-part workflow:

1. Training Set: Used to fit the parameters of a model with a fixed set of hyperparameters.
2. Validation Set: Used to evaluate and compare models with different hyperparameters. You train multiple models (e.g., with different k values or λ strengths) on the training set and select the one that performs best on the validation set.
3. Test Set: This is the final, sacrosanct dataset. You take the single model architecture and hyperparameter combination that you chose using the validation set, train it one last time on the full training set (or sometimes the combined training and validation sets), and evaluate its performance once on the test set. This result is your unbiased estimate of the model's generalization performance.

Andrew Ng explains this standard workflow very clearly.

This three-part split ensures a clean separation of concerns: fitting, tuning, and final evaluation. This workflow is fundamental to almost all applied machine learning.

This flowchart from the scikit-learn documentation illustrates the complete process. The initial dataset is split into "Training data" and "Test data." The training data is then used for model selection (here shown as "Cross-validation," a more advanced form of using a validation set) to find the "Best parameters." A final model is then retrained and gets its "Final evaluation" on the untouched test data.

To see this correct workflow in action, let's return to the coding example.

Data Splitting Strategies and Ratios

A common question is how to decide the size of each split.

• Traditional Rule of Thumb: For small to medium datasets (hundreds to tens of thousands of examples), common splits are 70% train / 30% test, or 60% train / 20% validation / 20% test.
• Modern Practice for Large Datasets: In the era of big data (millions of examples), the percentages for validation and test sets can be much smaller. The goal is not to hit a certain percentage, but to have validation and test sets that are large enough to yield statistically meaningful results. For a dataset with 10 million examples, a 98% train / 1% validation / 1% test split might be perfectly reasonable. This still provides 100,000 examples for validation and 100,000 for testing, which is more than enough to reliably compare models.

Andrew Ng also discusses this modern perspective.

Avoiding Data Leakage

The integrity of this entire process hinges on preventing data leakage, where information from a set used for evaluation (validation or test) contaminates the training process.

What if the Dataset is Small? Cross-Validation

When your dataset is small, holding out a significant portion for validation and testing feels wasteful. You want to use as much data as possible for training. In such cases, a technique called k-fold cross-validation is preferred.

Instead of a single validation set, you divide your training data into 'k' folds (e.g., 5 or 10). Then, you iteratively train your model k times. In each iteration, you use one fold as a temporary validation set and the remaining k-1 folds for training. You then average the performance scores across all k iterations to get a more robust estimate of the model's performance for a given set of hyperparameters. We will implement and discuss k-fold cross-validation in detail in Module 4, but it's important to know that it serves the same purpose as a validation set—hyperparameter tuning and model selection—but in a more data-efficient manner.

Conclusion

In this lesson, we established the fundamental methodology for developing and evaluating supervised learning models. The process of splitting data is not just a technical step but a core principle for ensuring scientific rigor and building models that are trustworthy.

Here are the key takeaways:

• The goal of machine learning is generalization to unseen data, and the training error is an optimistic, biased estimate of this performance.
• A training set is used to learn a model's parameters.
• A validation (or dev) set is used to tune hyperparameters and select the best model, preventing overfitting to the final evaluation set.
• A test set is held out until the very end and used only once to provide an unbiased estimate of the final, chosen model's performance on unseen data.
• This separation prevents data leakage and ensures that our performance metrics are credible.

This framework of splitting data to assess generalization error leads us directly to our next topic. The errors on the training and validation sets can tell us a lot about what is going wrong with our model. By analyzing them, we can diagnose whether our model is suffering from high bias (underfitting) or high variance (overfitting). In the next lesson, we will formally define the bias-variance tradeoff, a central concept in machine learning that helps us understand and address these issues.