Data Splitting Explained

1. The Core Problem: How Do We Grade Our Model?

Imagine you're a student preparing for a final exam. If your teacher gives you the exact questions and answers that will be on the exam, you could memorize them and get a perfect score. But would that score reflect your actual understanding of the subject? Of course not. You'd likely fail if you were given a different set of questions on the same topics.

Machine learning models face the same issue. If we train and evaluate a model on the same data, it might just "memorize" the answers. This phenomenon is called overfitting. The model learns the training data so perfectly—including its noise and quirks—that it fails to perform well on new, unseen data.

Our goal in machine learning is generalization: we want to build models that learn the underlying patterns in the data so they can make accurate predictions on data they've never encountered before. So, how do we test for generalization? We need a fair "exam" for our model.

2. Why a Simple Train-Test Split Isn't Always Enough

The most basic way to create a fair "exam" is to split our data into two parts:

• Training Set: The majority of the data, used to teach the model. This is like the textbook and practice problems for our student.
• Test Set: A smaller, held-out portion of the data that the model never sees during training. This is the final exam. We use it at the very end to get an unbiased assessment of the model's performance.

This is a good start, but it leaves a critical question unanswered: what if we want to improve our model?

For instance, you know about the K-Nearest Neighbors algorithm. One of its settings, or hyperparameters, is the number of neighbors (k). Should you use k=3, k=5, or k=10? To find out, you'd typically:

1. Train a model with k=3 on the training set.
2. Train another model with k=5 on the training set.
3. Compare their performance to see which k is better.

But on which data should you compare them?

• If you use the training set, you might pick a k that leads to overfitting.
• If you use the test set, you are breaking the cardinal rule: the test set is the final exam, not a study guide. By using it to choose the best k, you are "leaking" information from the test set into your model selection process. Your final performance score will be overly optimistic because you've tuned your model to do well on that specific set of "exam questions".

This is where the third dataset comes in.

3. The Solution: The Train-Validation-Test Split

To solve this problem, we introduce a validation set. This acts as a "mock exam" that we can use to make decisions during model development without touching our final test set.

Here are the distinct roles of each of the three sets:

1. Training Set: The model learns from this data. It adjusts its internal parameters to find patterns in these examples.
2. Validation Set: This dataset is used to guide model development. You use it to:
   • Tune hyperparameters: Find the best value for k in KNN or the optimal max_depth for a Decision Tree.
   • Select a model: Compare the performance of completely different models (e.g., is a Decision Tree better than a Logistic Regression model for this problem?).
3. Test Set: This dataset is kept in a "lockbox" until the very end. After you have used the validation set to select your single best model, you use the test set only once to get a final, unbiased estimate of its performance on truly unseen data.

This three-part split ensures that your model's final evaluation is fair and representative of how it will perform in a production environment.

Let's watch a video that uses a great analogy to explain this process and solidify the purpose of each split.

4. The Rules of the Road

To ensure the integrity of your model evaluation process, there are some firm rules to follow. The most important one concerns the test set. Let's get a clear explanation from a concise article.

As the article emphasizes, looking at the test set performance to influence your choice of model is a form of data leakage. It contaminates your final results and gives you a false sense of confidence in your model's abilities.

5. What are the standard split ratios?

A common question is how to divide the data. While there's no universal rule, there are common conventions.

Common splits include:

• 70% Training, 15% Validation, 15% Test
• 80% Training, 10% Validation, 10% Test

For very large datasets (millions of records), the validation and test sets can be a smaller percentage (e.g., 98/1/1), because even 1% of the data provides more than enough examples for robust evaluation.

Conclusion

You have now learned one of the most important foundational concepts in practical machine learning: how to properly partition your data for robust model development and evaluation.

Key Takeaways:

• The goal of splitting data is to get an honest estimate of how a model will generalize to new, unseen data and to avoid overfitting.
• The Training Set is for teaching the model its internal parameters.
• The Validation Set is a "mock exam" used to tune hyperparameters and select the best model type.
• The Test Set is the "final exam," used only once at the very end to provide an unbiased report of the final model's performance.
• Never use the test set to make decisions about your model. Peeking at it leads to overly optimistic and misleading results.

Preview of the next lesson:

Today, we discussed using a single validation set. But what if, just by chance, we got a "lucky" or "unlucky" split, and our validation set isn't truly representative? To get an even more reliable estimate of model performance, we can use a technique called k-fold cross-validation. In the next lesson, we will see how this powerful method repeatedly splits the training data into smaller train/validation folds, giving us a much more robust way to tune and select our models.