Hello! In our last lesson, we saw how learning curves reveal a model's character by plotting its training and validation errors. This powerful diagnostic tool depends on one critical assumption: that we have separate, distinct datasets for training and evaluation.
Today, we'll dive into the practical and principled way to create these datasets. Your goal is to master the implementation of train-test-validation splits, a fundamental practice for building reliable machine learning models. We will explore why we need three distinct sets and how to use them correctly to avoid one of the most common pitfalls in data science: overly optimistic performance estimates.
1. The Three Sets: A Framework for Reliable Evaluation
To build an intuition for data splitting, let's use a classic analogy: a student preparing for a final exam.
- Training Set: This is the student's collection of homework problems and textbook examples. They study this material to learn the concepts and fit the patterns (i.e., train the model).
- Validation Set: This is a set of practice exams. The student uses these to gauge their understanding, identify weak spots, and decide what to study next. They might take multiple practice exams and adjust their study strategy based on the results. This is analogous to hyperparameter tuning and model selection.
- Test Set: This is the final, proctored exam. It's a one-time assessment on material the student has never seen before. The score on this exam provides the most realistic, unbiased measure of their true knowledge. This is your model's final performance evaluation.
The key idea is that decisions made during the "practice" phase (validation) can subtly influence the final outcome. If you were to use the final exam as a practice test, your final score wouldn't be a true reflection of your knowledge. The same principle applies to machine learning.
For a sharp, conceptual overview of this distinction, watch this short video from Cassie Kozyrkov, Head of Decision Intelligence at Google.
Machine Learning: Validation vs Testing
Watch "Machine Learning: Validation vs Testing" to solidify your understanding of the roles each dataset plays.
Focus on the student exam analogy (the analogy) and the core idea she presents: validation is for "redemption" (improving the model), while testing is for the final check.
This workflow—train, validate, tweak, and finally test—is central to applied machine learning. The diagram below shows this iterative loop.
2. Implementing Splits with Scikit-Learn
In practice, we use the train_test_split function from Scikit-Learn's model_selection module. It's a versatile tool for creating the splits we need.
from sklearn.model_selection import train_test_split
Let's begin by exploring the core concepts and the syntax for creating these splits.
Split Your Dataset With scikit-learn's train_test_split() – Real Python
This article from Real Python provides a comprehensive guide to why and how we split data.
Read the sections "The Importance of Data Splitting" and "Application of train_test_split()". Pay close attention to the function's main parameters: test_size (or train_size): Defines the proportion of the split. random_state: Ensures your splits are reproducible. This is crucial for debugging and sharing your work. shuffle: Determines whether to shuffle the data before splitting (usually a good idea). stratify: An important parameter for classification tasks.
The "Two-Way" Split: Train and Test
The simplest split is into two sets: training and testing. This is appropriate when your model requires no hyperparameter tuning (a rare case) or when you use cross-validation (which we'll cover next).
# Assuming X contains your features and y contains your target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
This code holds out 20% of the data for the test set and uses the remaining 80% for training.
The "Three-Way" Split: Train, Validation, and Test
This is the standard, most robust approach. You create it by applying train_test_split twice:
- First, split the original data into a larger training set and a smaller test set.
- Second, split the larger training set into a new, smaller training set and a validation set.
# Step 1: Create the initial train/test split (e.g., 80% train, 20% test)
X_train_full, X_test, y_train_full, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Step 2: Split the training data again to create a validation set (e.g., 25% of the 80% becomes validation)
# This results in: 60% train, 20% validation, 20% test
X_train, X_val, y_train, y_val = train_test_split(X_train_full, y_train_full, test_size=0.25, random_state=42)
# Now you have:
# (X_train, y_train) for model fitting
# (X_val, y_val) for hyperparameter tuning
# (X_test, y_test) for final evaluation
The article below provides further examples and clarifies how to achieve exact proportions if needed.
Scikit-Learn's train_test_split() - Training, Testing and Validation Sets
This StackAbuse article reinforces the concepts and provides clear code snippets.
Focus on the sections "What About Validation Sets?" and "Creating a Validation Set with train_test_split()" to see the two-step splitting process in action.
3. A Practical Walkthrough: From Flawed to Flawless Evaluation
Now, let's see this entire process unfold in a real coding scenario. The following video by Greg Hogg is an excellent practical demonstration. He intentionally makes the common mistake of tuning on the test set first, then demonstrates why that's problematic and how to fix it with a proper validation set.
This workflow directly uses Pandas and Scikit-Learn, which is great practice for the skills you are building.
Train, Test, & Validation Sets | How to Train Machine Learning Models (Properly!!!)
Watch "Train, Test, & Validation Sets" to see a complete, code-first demonstration of the model evaluation workflow.
Follow the entire video. The key moments are: Data Setup: See how the data is loaded and prepared. The Mistake: The instructor trains different models and tunes hyperparameters (n_estimators, max_depth) while monitoring the test error. This is "data leakage"—information from the test set is leaking into the model selection process. The Correction: He introduces a third "holdout" set (our validation set) by splitting the test data. Now, he has three sets: train, validation, and a final test (which he calls holdout). The Right Way: He uses the validation set to pick the best model parameters. Final Evaluation: After selecting the best model using the validation set, he performs a single, final evaluation on the untouched holdout/test set to get an unbiased performance metric.
Conclusion
You have now learned the proper procedure for splitting data to reliably evaluate machine learning models. This three-set methodology—Train, Validate, Test—is a cornerstone of sound data science practice.
Key Takeaways:
- Training Set: Used exclusively to fit the model's parameters.
- Validation Set: Used to guide model selection, tune hyperparameters, and make other modeling decisions. This is your "practice exam."
- Test Set: A fire-and-forget dataset used only once at the very end to obtain a final, unbiased estimate of your chosen model's performance on unseen data.
- You can implement a robust train-validation-test split by applying Scikit-Learn's
train_test_splitfunction twice. - Always use
random_stateto ensure your experiments are reproducible. For classification, usestratifyto maintain class proportions across splits.
A single train-validation split, while correct, can sometimes be sensitive to which particular data points landed in the validation set. What if you got an "easy" or "hard" split by chance? In our next lesson, we will address this by exploring k-fold cross-validation, a more powerful and robust technique for model evaluation that essentially creates multiple validation sets.