Building a Dataset Split From Scratch

Why Split Your Data? The Problem of Overfitting

Imagine you're studying for an exam. You have a set of practice questions with answers. If you study these questions over and over until you get 100% on them, have you mastered the subject? Not necessarily. You might have just memorized the specific answers to those specific questions. The real test is whether you can answer new questions you've never seen before.

Machine learning models face the same issue. If we train and evaluate a model on the same data, it might achieve a perfect score by simply "memorizing" the training examples. This phenomenon is called overfitting. The model performs beautifully on the data it has seen but fails miserably on new data.

To combat this, we split our data into at least two sets:

• Training Set: The data the model learns from (the practice questions).
• Test Set: The data held aside to evaluate the final model's performance (the real exam).

This is a good start, but it introduces a new problem. What if we want to try different models or tune our model's settings (known as hyperparameters)? If we use the test set to guide these decisions, we are indirectly leaking information from the test set into our model design. Our model will start to overfit to the test set, and our final performance score will be overly optimistic.

The solution is to introduce a third set.

The Three Essential Datasets: Train, Validation, and Test

The professional standard is to split data into three distinct subsets:

1. Training Set: Used to fit the model's parameters. This is the bulk of your data, typically 60-80%.
2. Validation Set (or Development Set): Used to tune hyperparameters, select features, and make choices about the model architecture (e.g., comparing a linear model vs. a random forest). It acts as a proxy for the test set during the development phase. Typically 10-20% of the data.
3. Test Set (or Holdout Set): Used only once at the very end to provide an unbiased evaluation of the final, tuned model. You must not make any design choices based on the test set results. Typically 10-20% of the data.

This workflow is crucial for developing robust models.

Seeing the Problem in Action

To make this concept concrete, let's watch a video that demonstrates the pitfalls of not using a proper validation set.

This video makes it clear: the validation set is for tuning, and the test set is for the final, honest report card.

Implementing the Split From Scratch

As per your preference, we will now implement this entire process from scratch in Python. Our goal is to create a function that takes a dataset (features X and labels y) and splits it into training, validation, and test sets.

A naive approach would be to simply slice the dataset sequentially. For example, take the first 60% for training, the next 20% for validation, and the final 20% for testing.

The key takeaway is that we must shuffle the data before splitting. This ensures that all three sets are random samples and are likely to have a similar distribution of data.

A robust way to implement a shuffled split is:

1. Generate a sequence of indices from 0 to N-1, where N is the total number of samples.
2. Shuffle these indices randomly.
3. Use the shuffled indices to select rows from your data arrays.

The Stack Overflow page below contains several excellent from-scratch implementations. We'll focus on the one that is both clean and robust.

Our From-Scratch Function

Based on that logic, here is a complete Python function that performs a train-validation-test split. It uses NumPy for efficiency, which aligns with the libraries you're already familiar with. This function will be a reusable utility for us throughout the course.

import numpy as np

def train_val_test_split(X, y, train_ratio=0.7, val_ratio=0.15, test_ratio=0.15, random_state=42):
    """
    Splits feature and label arrays into training, validation, and test sets.

    Args:
        X (np.ndarray): Feature data.
        y (np.ndarray): Label data.
        train_ratio (float): Proportion of the dataset for training.
        val_ratio (float): Proportion of the dataset for validation.
        test_ratio (float): Proportion of the dataset for testing.
        random_state (int): Seed for the random number generator for reproducibility.

    Returns:
        tuple: A tuple containing (X_train, y_train, X_val, y_val, X_test, y_test).
    """
    # Ensure the ratios sum to 1
    assert train_ratio + val_ratio + test_ratio == 1.0, "Ratios must sum to 1"

    # Get the total number of samples
    n_samples = X.shape[0]

    # Create and shuffle indices
    # Using np.random.permutation is a standard and efficient way to do this.
    np.random.seed(random_state) # for reproducibility
    shuffled_indices = np.random.permutation(n_samples)

    # Calculate the split points
    train_end = int(train_ratio * n_samples)
    val_end = int((train_ratio + val_ratio) * n_samples)

    # Get the indices for each set
    train_indices = shuffled_indices[:train_end]
    val_indices = shuffled_indices[train_end:val_end]
    test_indices = shuffled_indices[val_end:]

    # Split the data using the indices
    X_train, y_train = X[train_indices], y[train_indices]
    X_val, y_val = X[val_indices], y[val_indices]
    X_test, y_test = X[test_indices], y[test_indices]
    
    return X_train, y_train, X_val, y_val, X_test, y_test

# --- Example Usage ---
# Create some dummy data
X_dummy = np.arange(100).reshape(100, 1) * 10
y_dummy = np.arange(100)

# Perform the split
X_train, y_train, X_val, y_val, X_test, y_test = train_val_test_split(
    X_dummy, y_dummy, train_ratio=0.7, val_ratio=0.15, test_ratio=0.15
)

# Print the shapes to verify
print("--- Shapes of the splits ---")
print(f"X_train: {X_train.shape}, y_train: {y_train.shape}")
print(f"X_val:   {X_val.shape}, y_val:   {y_val.shape}")
print(f"X_test:  {X_test.shape}, y_test:  {y_test.shape}")

Take your time to read through this code and its comments. The logic directly follows our discussion: shuffle indices, calculate boundaries, and then use those indices to slice the X and y arrays. Setting a random_state is also good practice, as it ensures your splits are the same every time you run the code, making your experiments reproducible.

Conclusion

In this lesson, we moved from the abstract concept of supervised learning to the first practical step in any modeling project: preparing the data for training and evaluation.

Key Takeaways:

• Overfitting is when a model memorizes the training data instead of learning general patterns.
• To get a reliable estimate of model performance, we split data into training, validation, and test sets.
• The training set is for teaching the model.
• The validation set is for tuning hyperparameters and comparing models.
• The test set is used only once for a final, unbiased performance report.
• Shuffling the data before splitting is essential to ensure each set is a random and representative sample of the overall dataset.

Preview of the Next Lesson:

A single validation set works well, but what if our dataset is small? A random 15% split might unfortunately contain particularly easy or hard examples, giving us a misleading signal. To make our validation process even more robust, especially with limited data, we can use a technique called k-fold cross-validation. In the next lesson, you will implement k-fold cross-validation from scratch to use your training data even more effectively.