Hello! In our last two lessons, we got our data into shape by converting categorical text into numbers and then normalizing our numerical data to a common scale. Our dataset is now clean and properly formatted. We're on the verge of being able to train a machine learning model.
But before we do that, we need to address one final, crucial preparation step. We can't use all our data to teach the model. Why? Because if we do, we'll have no way of knowing if the model actually learned or if it just memorized the answers.
Today's lesson covers how to properly split a dataset into training, validation, and test sets. You'll learn the distinct purpose of each set and why this division is fundamental to building a reliable model that works in the real world.
The School Analogy: Why We Split Data
Imagine you're a teacher preparing a student for a big national math competition. You have a big book of practice problems.
- Homework (Training): You give your student a large set of problems to work through. The student studies these problems, learns the patterns, and figures out the methods to solve them. This is how they learn.
- Practice Exams (Validation): Every week, you give the student a short quiz with problems they haven't seen before. Based on their score, you can tell if your teaching method is working. Maybe you need to focus more on algebra, or maybe they need a different technique for geometry. You use these quizzes to tune your teaching strategy and help the student improve.
- The Final Competition (Testing): Finally, the student goes to the competition. This is the ultimate test, with brand new problems they have never encountered. Their score here tells you how well they will actually perform in the real world. You can't help them during the competition; their performance is based entirely on what they learned.
This is exactly how we handle data in machine learning. We split our single dataset into three smaller ones, each with a specific job.
Validation data: How it works and why you need it - Machine Learning Basics Explained
This video from Galaxy Inferno Codes provides an excellent explanation using this very analogy of a teacher preparing students for a competition.
Watch the first three minutes to understand the distinct roles of each dataset. Pay close attention to the difference between the overview of the three sets and the detailed student analogy that follows.
The Three Datasets: A Closer Look
As the video explained, the three sets are the training, validation, and test sets. Let's formalize their roles.
1. The Training Set: Learning the Material
The training set is the "homework." It's the largest part of the data and is used to teach the model. The model looks at the features (inputs) and the corresponding labels (correct outputs) in this set and adjusts its internal logic to find the patterns connecting them.
Training Set vs Validation Set vs Test Set
This Codecademy article gives a clear and concise definition for each of the sets.
Read the first section, "What is a training set?", to solidify your understanding of its purpose.
2. The Validation Set: Tuning the Model
The validation set is the "practice exam." After the model has trained on the training data, we use the validation set to check its performance on data it hasn't seen before. The results from the validation set help us make decisions:
- Is the model learning well?
- Should we change some of its settings (called hyperparameters) and train it again?
- If we have multiple different models, which one is performing best?
This process of training, validating, and tweaking is a loop. We repeat it until we're happy with the model's performance on the validation set.
Training Set vs Validation Set vs Test Set
Now, let's look at the validation set's role in the same Codecademy article.
Read the section titled "What is a validation set?". Notice how it highlights the problem of overfitting—when a model memorizes the training data instead of learning to generalize.
3. The Test Set: The Final Grade
The test set is the "final competition." This slice of data is kept completely separate and is never touched during the training and tuning process. Once we have chosen our final model (the one that performed best on the validation set), we use the test set exactly once to get a final, unbiased score. This score tells us how the model is likely to perform on brand new, real-world data.
Datasets: Dividing the original dataset | Machine Learning | Google for Developers
This article from Google's Machine Learning Crash Course explains why having just a train and test set isn't enough, and why the third validation set is so important.
Read the section on training, validation, and test sets. The interactive exercise within this section does a great job of explaining why you would inadvertently "teach to the test" if you only used a test set for tuning.
If the model does well on the test set, we can be confident about deploying it. If it does poorly, we know it's not ready, and we might need to go back, collect more or different data, and start the process over.
How Do We Split the Data?
There isn't a single magic rule for how to divide the data, but a common practice is to use a ratio like:
- 70% for the Training Set
- 15% for the Validation Set
- 15% for the Test Set
Other common splits are 80/10/10 or 60/20/20. The most important thing is that the training set is the largest, and you have enough data in your validation and test sets to get a reliable performance score. When you use ML tools, you will often see an interface that lets you define these splits.
The most critical rule is to keep the test set sacred. You cannot use it to tune the model. Doing so would be like giving your student the final exam questions to study beforehand. They would get a perfect score, but it wouldn't be a true measure of their knowledge. In machine learning, this is called data leakage, and it leads to overly optimistic performance estimates and models that fail in the real world.
Conclusion
Great work! You've now learned the final and one of the most important steps in data preparation. Building a trustworthy ML model is as much about how you test it as it is about how you train it.
Let's recap the key takeaways:
- We split our data into three sets: training, validation, and test.
- The training set is for teaching the model the basic patterns.
- The validation set is for tuning the model and choosing the best one (like a practice exam).
- The test set provides the final, unbiased performance score (like the final exam).
- This process prevents overfitting and gives us confidence that our model will generalize to new, unseen data.
We have now covered the entire data preparation pipeline: cleaning, transforming, and splitting. In our next module, we'll start exploring the specific AWS services, like Amazon S3, AWS Glue, and Amazon SageMaker Data Wrangler, that allow you to perform all these steps at scale in a real-world cloud environment.