Train-Test-Validation Splits for Model Evaluation

1. The Need for an Unbiased Evaluation

Imagine you're developing your UPSC curriculum generator. You give it a set of 1,000 history topics to structure. It produces a perfect curriculum. Success?

Not necessarily. If you then ask it to structure the same 1,000 topics again, it will likely perform perfectly. But what happens when you give it 200 new topics from Political Science? This is the real test.

Testing a model on the same data it was trained on is like giving a student a practice exam and then making the final exam identical. Their score on the final wouldn't prove they learned the subject, only that they could memorize the answers. This is why we must test our models on data they have never seen before. This brings us to the most basic data split.

2. The First Split: Training and Test Sets

The simplest way to create a "seen" and "unseen" dataset is to split your original data into two parts:

1. Training Set: The majority of the data (e.g., ~80%), used to teach the model. The model sees the inputs and the correct outputs and adjusts its internal parameters to find patterns. This is the model's "textbook and study material."
2. Test Set: A smaller, held-out portion of the data (e.g., ~20%). This data is kept completely separate during training. It serves as the "final exam" to give a single, unbiased measure of the model's performance on unseen data.

This is a good start, but it introduces a subtle but critical problem.

3. The Flaw in the Two-Set System: Contaminating the Final Exam

During the development of a model, you and your team need to make many decisions:

• "Which model architecture should we use? Model A or Model B?"
• "How complex should the model be?"
• "How long should we train the model for?"

These are known as hyperparameters—settings that you choose before the training process begins. To make these decisions, you need to evaluate which choices lead to the best performance.

If you only have a training and a test set, you might be tempted to use the test set to make these decisions. For example:

1. Train Model A on the training set. Evaluate its score on the test set.
2. Train Model B on the training set. Evaluate its score on the test set.
3. You see that Model B got a higher score, so you choose it as your final model.

The problem? You've used the test set to "tune" your model choice. Information from the test set has now leaked into your model design process. The test set is no longer a truly "unseen" final exam; it has become part of the study process. Your final evaluation will be overly optimistic, and you've inadvertently overfitted your design choices to the test set.

To solve this, we introduce a third dataset.

4. The Gold Standard: Training, Validation, and Test Sets

The industry-standard solution is to divide your data into three distinct subsets. This methodology provides a robust framework for both model development and final evaluation.

To understand the distinct role of each set, please read the following article.

Let's consolidate these definitions using our own analogy: preparing for the UPSC exam.

• Training Set (The Textbooks): This is the largest part of your data (e.g., 60-80%). The model uses this to learn the core patterns and relationships. This is equivalent to a student studying standard textbooks like NCERTs and reference books to learn the fundamental concepts of history, polity, etc.

• Validation Set (The Mock Exams): This is a separate chunk of data (e.g., 10-20%). It's used periodically during the development process to tune the model's hyperparameters and make design choices. For a student, this is like taking a series of mock exams. Based on their performance, they might change their study strategy (e.g., "I'm weak in Modern History, I need to focus there," or "My current note-taking method isn't working").

• Test Set (The Final UPSC Exam): This is the final, untouched piece of data (e.g., 10-20%). It is used only once, after all development and tuning is complete, to get a final, unbiased measure of the model's performance. For the student, this is the actual UPSC Prelims or Mains exam. Their performance here is the ultimate measure of their knowledge.

The entire workflow is about using the validation set to pick the best model/strategy before subjecting that single best choice to a final, fair evaluation.

The following resource from Google's Machine Learning Crash Course provides a fantastic visual workflow of this process.

This workflow—train, validate, tweak, repeat, and then finally test—is central to applied machine learning. The image below shows this in action, where a team evaluates multiple models and uses the validation set to select the winner for final testing.

5. The Mechanics: How to Split Data Correctly

As an architect, you need to ensure your team follows best practices when creating these splits. A flawed split can invalidate your entire evaluation process.

Typical Ratios:
Common splits are 60% training, 20% validation, 20% testing, or 70/15/15. There's no single magic number. For very large datasets (millions of examples), the validation and test sets can be a much smaller percentage (e.g., 98/1/1) because even 1% constitutes a very large number of examples. The key is that the validation and test sets must be large enough to be statistically representative.

Key Principles for a High-Quality Split:

1. Shuffle the Data: Before splitting, always randomly shuffle the entire dataset. If your data is ordered in some way (e.g., by date or topic), failing to shuffle could lead to your training set containing only old data and your test set containing only new data, which would not be a fair evaluation.

2. Stratified Sampling: When dealing with classification problems, it's crucial that the class proportions are maintained across all three sets. For example, if your UPSC question dataset is 15% "Polity" and 85% other subjects, each of your train, validation, and test sets should also have roughly 15% "Polity" questions. This is called stratified sampling, and it prevents a situation where, by bad luck, one of your sets has no examples of a particular class.

3. Prevent Data Leakage: This is the cardinal rule. No data from the validation or test sets should ever be used to train the model. A common way this rule is broken is by having duplicate examples across the sets. For instance, if the same question appears in both the training set and the test set, your model's test score will be artificially inflated. Your data preparation pipeline must rigorously de-duplicate data before splitting.

Conclusion

This lesson concludes our foundational module. You now understand the rigorous process for evaluating machine learning models, which is essential for moving from a promising prototype to a reliable production system.

Key Takeaways:

• Purpose of Splitting: To get an unbiased estimate of how a model will perform on new, unseen data (i.e., its ability to generalize).
• Training Set: The data the model learns from.
• Validation Set: The data used to tune model hyperparameters and make design choices (e.g., selecting between different architectures). It's the "mock exam."
• Test Set: Held-out data used only once at the very end to provide a final, unbiased performance report. It's the "final exam."
• Mechanics: Proper evaluation requires shuffling data, using stratified sampling for classification, and preventing any data leakage between the sets.

We've now covered the high-level paradigms of ML, the goals and challenges (loss, bias-variance, overfitting), and the framework for evaluation. With this solid foundation, you are ready to start looking inside the models themselves.

In our next module, "Neural Networks: From Perceptrons to RNNs," we will begin our journey into the architectures that power modern AI. We'll start with the most fundamental building block of them all: the artificial neuron, or perceptron.