Splitting Data: Train-Test Explained

1. The Rationale: Why We Must Split Our Data

Imagine you are studying for an important exam. The professor gives you a set of 100 practice questions with the answers. If you simply memorize the answers to those 100 questions, you might score 100% on a test that uses only those exact questions. But what happens when the real exam contains new questions you've never seen before? Your performance would likely be poor, because you memorized the answers instead of learning the underlying concepts.

Machine learning models can fall into the same trap. This phenomenon is called overfitting.

Overfitting occurs when a model learns the training data too well, including its noise and random fluctuations, to the point that it negatively impacts the model's performance on new, unseen data. The model has "memorized" the training set instead of "generalizing" the patterns.

To prevent this, we need a way to test our model on data it has never seen during training. The solution is to split our original dataset.

The Two-Set Split: Training and Testing

The simplest way to split the data is into two parts:

1. Training Set: A subset of the data (e.g., 80%) used to train the model. The model's parameters are fit using this data.
2. Test Set: A separate subset of the data (e.g., 20%) that is "held back." This data is used only at the very end to evaluate the final model's performance on unseen data. It serves as our "final exam."

This ensures we get an unbiased estimate of how our model will perform in the real world. To see a practical demonstration of this split and its purpose, watch the following video.

2. A More Robust Approach: The Three-Way Split

In a real-world project, you don't just train one model. You experiment. You might try different algorithms (e.g., Linear Regression vs. a Neural Network) or tune your model's settings, which are called hyperparameters.

If you use the test set repeatedly to see which model or which settings perform best, you will start to inadvertently "leak" information from the test set into your model design choices. Your model will start to overfit to the test set, and the test set will no longer be an unbiased measure of real-world performance. You'd be "teaching to the test."

To solve this, we introduce a third set: the validation set.

1. Training Set: Used to fit the model's parameters.
2. Validation Set: Used to evaluate the model during development. You use this set to tune hyperparameters and select the best-performing model architecture. Think of it as a series of quizzes you take while studying.
3. Test Set: Used only once at the very end to report the final, unbiased performance of your chosen model. This is the final exam.

The following resource from Google's Machine Learning Crash Course explains this three-way split in more detail.

This next video uses a helpful analogy to solidify the distinct purpose of each of the three data splits.

3. Practical Implementation with scikit-learn

Now that you understand the theory, let's see how to perform this split in Python. The industry-standard library for this is scikit-learn, and the function is train_test_split.

# First, we need to import the function
from sklearn.model_selection import train_test_split
import numpy as np




# Let's create some dummy data
# X represents our features: 100 samples, 10 features each
X = np.random.rand(100, 10) 



# y represents our labels: 100 labels, one for each sample
y = np.random.randint(0, 2, 100) 




# Now we perform the split. 
# We'll create a training set and a temporary set.
# The temporary set will be further split into validation and test sets.
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.3, random_state=42
)




# Now we split the temporary set into validation and test sets
# We want validation to be 15% and test to be 15% of the original data.
# Since X_temp is 30% of the original, we split it in half (0.5).
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.5, random_state=42
)

print(f"Original data shape: {X.shape}")
print(f"X_train shape: {X_train.shape}") # 70% of original
print(f"X_val shape: {X_val.shape}")   # 15% of original
print(f"X_test shape: {X_test.shape}")    # 15% of original

Let's break down the key parameters in train_test_split:

• X, y: The arrays containing your features and labels.
• test_size: A float between 0.0 and 1.0 representing the proportion of the dataset to include in the test split (e.g., 0.2 for 20%). You can also use train_size to specify the proportion for the training set.
• random_state: An integer. By setting this, you guarantee that the split you generate is always the same. If you don't set it, you'll get a different random split every time you run the code. Using a random_state is crucial for reproducible results, a cornerstone of good software and science.

For a comprehensive guide on this function and its capabilities, the following article is an excellent reference.

Conclusion

In this lesson, you've learned one of the most fundamental and critical practices in machine learning: splitting your data to ensure an honest evaluation of your model.

Key Takeaways:

• Rationale: We split data to get an unbiased estimate of our model's performance on new, unseen data and to detect overfitting.
• Train-Test Split: The most basic split involves a training set to build the model and a test set to evaluate its final performance.
• Train-Validation-Test Split: A more robust method adds a validation set, which is used during development to tune hyperparameters and compare different models without "contaminating" the final test set.
• Practical Tool: The sklearn.model_selection.train_test_split function is the standard tool in Python for performing these splits, with random_state being a key parameter for ensuring reproducibility.

Preview of the Next Lesson:
So far, we've discussed datasets in abstract terms (X and y). In the real world, data comes in files (like CSVs) and needs to be loaded and prepared before it can be split and used for training. In the next lesson, we will address the learning outcome: "Use scikit-learn to load a dataset and prepare it for training." You will learn how to take a raw data file and transform it into the clean feature matrices and label vectors that our models require.