Dataset Splitting for Model Evaluation

Why We Can't Use All Our Data for Training

The primary goal of a machine learning model is to generalize—to perform well on new, unseen data. If we train our model on our entire dataset and then evaluate it on that same data, we get a misleadingly optimistic score. The model might have simply memorized the "answers" without learning the general principles. This phenomenon is called overfitting.

To get a true measure of performance, we need to test the model on data it has never seen before. This leads to the most basic data split: a training set and a test set.

However, the process of building a good model isn't just a one-shot train-and-test cycle. We often need to:

• Compare different types of models (e.g., Linear Regression vs. a Decision Tree).
• Tune the model's settings, called hyperparameters (e.g., the complexity of a model).

If we use the test set to make these decisions, we are indirectly using it to guide our model-building process. Our choices will start to become tailored to that specific test set, and it will no longer be a truly "unseen" dataset. The performance estimate we get will again be over-optimistic.

This is where a third dataset becomes essential.

The Three Essential Datasets: Train, Validate, Test

The standard practice in machine learning is to split your data into three distinct subsets.

To understand the specific role of each, let's watch a short, insightful video that uses a great analogy.

As the video explained, each dataset has a unique and critical purpose:

1. Training Set (The Textbook):

   • This is the largest portion of the data.
   • Its sole purpose is to train the model. The model learns the relationships between the input features and the output labels from this data.

2. Validation Set (The Practice Exam):

   • This is a medium-sized portion of the data.
   • Its purpose is to tune the model and select the best one. You use this set to:
     • Adjust hyperparameters.
     • Compare the performance of different algorithms.
   • The model never learns from this data, it is only evaluated on it during the development phase.

3. Test Set (The Final Exam):

   • This is a smaller portion of the data that is held out until the very end.
   • Its purpose is to provide an unbiased evaluation of the final, chosen model.
   • You should only use this set once. After you have trained and tuned your model using the training and validation sets, you use the test set to estimate how well it will perform in the real world.

A Practical Walkthrough: Seeing the Splits in Action

Now that you understand the "why," let's move to the "how." The best way to see the importance of this process is to walk through a coding example. We'll use Python, the pandas library you're familiar with, and scikit-learn, the most important machine learning library in Python.

The following video provides a fantastic, hands-on demonstration. It shows what happens when you only use a train/test split and why that's problematic, before introducing the correct train/validation/test workflow.

Implementing the Split in Python

As you saw in the video, scikit-learn provides a very convenient function for this task: train_test_split.

Let's formalize the process. Since train_test_split only splits data into two sets at a time, we perform the split in two steps.

Here is the common pattern, assuming X contains your features and y contains your labels:

from sklearn.model_selection import train_test_split

# Assume X are your features and y is your target
# First split: Create the training set (e.g., 80%) and a temporary set (20%)
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.2, random_state=42)

# Second split: Split the temporary set into validation and test sets (e.g., 50/50 split of the 20%)
# This results in 10% validation and 10% test of the original data
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)

print(f"Training set size: {len(X_train)}")
print(f"Validation set size: {len(X_val)}")
print(f"Test set size: {len(X_test)}")

Key Parameters:

• test_size: Specifies the proportion of the data to be allocated to the test split. You can also use train_size.
• random_state: This is a crucial parameter for reproducibility. By setting random_state to any integer (e.g., 42), you ensure that the same random split is generated every time you run the code. This is essential for debugging and getting consistent results. If you leave it out, you'll get a different split each time.

Splitting Ratios and Other Methods

• Common Ratios: There's no single perfect ratio, but common splits are 80/10/10 (Train/Val/Test) or 70/15/15. For very large datasets (millions of records), you might even use something like 98/1/1, because 1% is still a huge number of data points for validation and testing.

• Advanced Splitting: While random splitting is the default, some situations require different strategies.

  • Stratified Splitting: When you have a classification problem with imbalanced classes (e.g., 95% "not fraud" and 5% "fraud"), a random split might result in a validation or test set with very few or no examples of the minority class. Stratified splitting ensures that the proportion of each class is the same across the train, validation, and test sets. You can do this by setting the stratify parameter in train_test_split.
  • Time-Based Splitting: For time-series data (like stock prices), you cannot randomly shuffle the data, as that would destroy the temporal order. In these cases, you must split the data chronologically, for instance, using past data for training and more recent data for testing.

Conclusion

You have now mastered a fundamental practice in machine learning model development. Properly separating your data is the first line of defense against building models that look great on paper but fail in the real world.

Key Takeaways:

• The goal of splitting data is to get an honest estimate of how your model will perform on unseen data.
• Training Set: Used to teach the model its parameters.
• Validation Set: Used to tune hyperparameters and compare different models.
• Test Set: Used only once at the end for a final, unbiased performance evaluation.
• Using the test set for tuning leads to data leakage and an over-optimistic performance estimate.
• In Python, you can use sklearn.model_selection.train_test_split to perform the splits, typically in a two-step process.

Preview of the Next Lesson:
Splitting the data is a crucial organizational step. The next step is to prepare the data within those splits for the model to use effectively. In the next lesson, you will learn how to handle missing values and apply feature scaling (normalization, standardization). These are essential data preprocessing techniques that can significantly improve your model's performance.