Data Splitting: Train, Validate, Test

Lesson Introduction

This lesson focuses on a cornerstone of reliable model evaluation: splitting data into training, validation, and test sets. You will learn the distinct purpose of each set and how to implement this three-way split in Python using scikit-learn. Mastering this technique is fundamental to building models that you can trust to perform well in the real world.

1. The "Why": Understanding the Role of Each Dataset

Before we write any code, let's build a strong conceptual foundation for why we need three datasets, not just two. A common analogy is that of a student preparing for a final exam.

The analogy makes it clear: you can't properly evaluate yourself on questions you've already seen the answers to. To make this more concrete in a machine learning context, let's watch a short video.

To summarize and formalize these crucial concepts:

• Training Set: This is the bulk of your data. The model uses this set to learn the underlying patterns by adjusting its internal parameters. For a neural network, this means learning the weights and biases. For a linear regression model, it means learning the coefficients.

• Validation Set (or Development Set): This dataset is used to tune the model's hyperparameters. Hyperparameters are the high-level configuration choices you make as a data scientist before training begins, such as the number of trees in a random forest or the learning rate in gradient descent. You train various models with different hyperparameters on the training set and then evaluate them on the validation set to see which configuration performs best on data it hasn't been trained on.

• Test Set: This dataset is the final checkpoint. It is kept completely separate and is used only once after you have selected your final model (based on its validation set performance). The test set provides an unbiased estimate of how your model will generalize to new, unseen data. Its performance score is the one you would report to stakeholders.

This workflow is visualized perfectly in the following flowchart:

2. The "How": Splitting Data with Scikit-Learn

Now that we understand the "why," let's move to the "how." In Python, the scikit-learn library provides a convenient function called train_test_split that is the standard tool for this job.

However, train_test_split only splits data into two sets. To get our desired three sets (train, validation, test), we simply apply the function twice.

Implementing the Two-Step Split

Let's walk through the code. Imagine we have our data loaded into a pandas DataFrame data and we want to split it into a 60% training, 20% validation, and 20% test set.

import pandas as pd
from sklearn.model_selection import train_test_split

# Assume 'data' is a pandas DataFrame with your features and target
# For demonstration, let's create a dummy DataFrame
data = pd.DataFrame({
    'feature1': range(100),
    'feature2': range(100, 200),
    'target': range(200, 300)
})

# Separate features (X) and target (y)
X = data[['feature1', 'feature2']]
y = data['target']

# Step 1: Split the data into training+validation (80%) and test (20%)
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

# Step 2: Split the train_val set into training (75% of 80%) and validation (25% of 80%)
# 0.25 * 0.80 = 0.20, so this gives us a 20% validation set from the original data.
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val, y_train_val, test_size=0.25, random_state=42)

# Print the sizes of the resulting sets
print(f"Full dataset shape: {X.shape}")
print(f"Training set shape: {X_train.shape}")
print(f"Validation set shape: {X_val.shape}")
print(f"Test set shape: {X_test.shape}")

# Expected output:
# Full dataset shape: (100, 2)
# Training set shape: (60, 2)
# Validation set shape: (20, 2)
# Test set shape: (20, 2)

Notice the calculation in Step 2. To get a 20% validation set from the original total, we take 25% of the train_val set, which constitutes 80% of the data (0.25 * 80% = 20%).

Important train_test_split Parameters

The train_test_split function has a few key parameters you should know.

Let's recap the key parameters:

• test_size (or train_size): A float between 0.0 and 1.0 representing the proportion of the dataset to include in the split. Common splits are 70/15/15 or 80/10/10. The ideal ratio depends on the size of your dataset; with very large datasets, you might use smaller percentages for validation and testing (e.g., 98/1/1).
• random_state: This is an integer that acts as a seed for the random number generator. By setting random_state, you ensure that the split is reproducible. Every time you run the code with the same random_state, you will get the exact same split. This is crucial for debugging and sharing your work. If you leave it as None, you'll get a different random split each time.
• stratify: This parameter is very important for classification tasks, especially with imbalanced datasets. If you have a dataset with 90% class A and 10% class B, a random split might result in a validation set with very few or even zero examples of class B. By setting stratify=y (where y is your target labels), you ensure that the training, validation, and test sets all have the same proportion of each class as the original dataset.

X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size=0.15, random_state=42)

X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val, test_size=0.1765, random_state=42)

3. A Complete Workflow in Action

To see how these splits are used in a complete, (though simplified) modeling process, let's watch one more video. This will tie together what we've learned today with the concepts of model training and evaluation.

This practical demonstration reinforces the importance of the three-set split for honest model building and evaluation.

Conclusion

You have now learned one of the most important procedural skills in machine learning. Properly separating your data ensures that you can train your model, tune it effectively, and, most importantly, produce a reliable estimate of its real-world performance.

Key Takeaways:

• A three-way data split is essential for robust model development.
• Training Set: Used by the model to learn its parameters.
• Validation Set: Used by the data scientist to tune hyperparameters and select the best model.
• Test Set: A "sacred," held-out set used only once for the final, unbiased performance evaluation.
• In Python, you can achieve this by applying scikit-learn's train_test_split function twice.
• Parameters like random_state ensure reproducibility, and stratify is crucial for maintaining class proportions in classification tasks.

Preview of the Next Lesson:
A single train-validation-test split works well, but it has a weakness: the model's performance can depend heavily on which specific data points happened to fall into the validation set. If you have a small dataset, this can be a significant problem. In our next lesson, we will explore a more robust and data-efficient technique for model evaluation called cross-validation.