Data Splitting Strategies

The "Why": Simulating the Real World

Imagine you're studying for a final exam. If your professor gives you the exact exam questions and answers to practice with, you could memorize them and get a perfect score. But would that mean you've truly learned the subject? Of course not. You'd likely fail if given a different set of questions.

Machine learning models face the same risk. If we train and evaluate a model on the same data, it might just "memorize" the answers. This is called overfitting. The model performs perfectly on the data it has seen but fails miserably when it encounters new, real-world data.

To prevent this, we split our dataset into separate parts, each with a distinct purpose.

Let's watch a short video that provides an excellent intuition for the three main data sets.

To summarize the roles:

1. Training Set (The Textbook): This is the largest portion of your data. The model uses this set to learn the underlying patterns. The entire learning process, like adjusting coefficients in a regression or weights in a neural network, happens here.

2. Validation Set (The Practice Quizzes): After training a few different models (or one model with different settings), how do you choose the best one? You can't use the test set, as that would be like peeking at the final exam. Instead, you use the validation set. You evaluate each model's performance on this set and select the one that does best. This process is known as model selection and hyperparameter tuning.

3. Test Set (The Final Exam): This set is kept in a "lockbox" until the very end. Once you have selected and tuned your final model using the training and validation sets, you use the test set only once to get a final, unbiased measure of its performance on completely unseen data. This result is what you would report to stakeholders as an estimate of how the model will perform in the real world.

Using the test set for anything other than the final evaluation leads to information leakage, where knowledge about the test set inadvertently influences the model. This makes the final evaluation optimistic and untrustworthy.

The "How": Implementing Splits with Scikit-Learn

Now that you understand the "why," let's move on to the "how." The scikit-learn library, a cornerstone of machine learning in Python, provides a straightforward function for this task: train_test_split.

The following video from Real Python demonstrates how to use this function and explains its most important parameters.

A Practical Code Example

Let's consolidate what you just saw. Here is the standard workflow for a simple train-test split.

First, you separate your features (the independent variables, X) from your target (what you want to predict, y).

# Assuming you have a pandas DataFrame 'df'
features = ['bedrooms', 'bathrooms', 'sqft_living']
target = ['price']

X = df[features]
y = df[target]

Next, you use train_test_split:

from sklearn.model_selection import train_test_split




# Split the data into training (80%) and testing (20%) sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)




# Check the shapes of the resulting sets
print("X_train shape:", X_train.shape)
print("X_test shape:", X_test.shape)
print("y_train shape:", y_train.shape)
print("y_test shape:", y_test.shape)

Key Parameters for "Appropriate" Splitting

The "appropriate" way to split data depends on a few key parameters:

• test_size (or train_size): This determines the proportion of the data to allocate to the test set. A test_size of 0.2 means 20% of the data will be for testing and 80% for training. Common splits are 80/20, 75/25, or 70/30.
• random_state: This is a pseudo-random number generator seed. By setting it to an integer (e.g., 42), you ensure that you get the exact same split every time you run the code. This is crucial for reproducibility, debugging, and collaboration. Without it, you'd get a different random split each time, making it hard to compare results.
• shuffle: This is True by default, meaning the data is shuffled randomly before being split. This is important to prevent any bias that might arise from the order of the data (e.g., if the data is sorted by date).
• stratify: This parameter is essential for classification problems. Imagine you have a dataset where 90% of your data is Class A and 10% is Class B. A random split might accidentally put very few samples of Class B in your test set. By setting stratify=y, you ensure that the proportion of classes in your training and testing sets is the same as in the original dataset. This guarantees your evaluation is representative.

Creating the Train-Validation-Test Split

You might have noticed that train_test_split only creates two sets (train and test). So how do we get our third set, the validation set?

The standard procedure is a two-step split:

1. First Split: Carve out the final test set from the full dataset. This set will not be touched again until the very end.
2. Second Split: Take the remaining larger portion (the training data from step 1) and split it again to create your new, smaller training set and your validation set.

Here's how you would implement this in code, aiming for a 60% train, 20% validation, 20% test split:

# Assume X and y are your full feature and target sets




# Step 1: Split into a large training set (80%) and a test set (20%)
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=(y if 'classification_task' else None)
)




# Step 2: Split the large training set into a final training set (75% of 80% = 60%)
# and a validation set (25% of 80% = 20%)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val, y_train_val, test_size=0.25, random_state=42, stratify=(y_train_val if 'classification_task' else None)
)

print(f"Total samples: {len(X)}")
print(f"Training samples: {len(X_train)} ({len(X_train)/len(X):.0%})")
print(f"Validation samples: {len(X_val)} ({len(X_val)/len(X):.0%})")
print(f"Test samples: {len(X_test)} ({len(X_test)/len(X):.0%})")

This two-step process correctly isolates the test set from all model development and tuning activities, which is the cornerstone of reliable machine learning practices.

Conclusion

In this lesson, you've taken the first critical step in the machine learning workflow. You now understand not only why data splitting is essential but also how to implement it correctly in Python.

Key Takeaways:

• Splitting data prevents overfitting and allows for an unbiased evaluation of model performance.
• The training set is for teaching the model its core patterns.
• The validation set is for tuning model hyperparameters and selecting the best-performing model.
• The test set is reserved for a single, final evaluation of the chosen model on unseen data.
• Use sklearn.model_selection.train_test_split to perform the splits.
• Always use random_state for reproducible results and stratify for classification tasks to maintain class proportions.

With your data correctly partitioned, you are now ready to build a model. In our next lesson, we will use the X_train and y_train sets you've just created to implement linear regression and interpret its coefficients.