K-Fold Cross-Validation From Scratch

What is K-Fold Cross-Validation?

K-fold cross-validation is a procedure used to evaluate a model by training and testing it on different subsets of the data. Instead of one validation split, we create k of them.

Here's the process:

1. Split the Data (Once): First, set aside your test set just as we did before. The test set is not used in cross-validation.
2. Partition the Training Data: Take the remaining training data and divide it into k equal-sized, non-overlapping subsets. These subsets are called "folds". A common choice for k is 5 or 10.
3. Iterate and Evaluate: Loop k times. In each iteration:
   • Select one fold to be the validation set.
   • Use the remaining k-1 folds as the training set.
   • Train your model on the training set and evaluate its performance on the validation set.
   • Record the performance score (e.g., accuracy, MSE).
4. Aggregate the Results: After k iterations, you will have k performance scores. The average of these scores is your cross-validation performance estimate. You can also look at the standard deviation to understand the variance in the model's performance across different data subsets.

The following diagram illustrates this process for 5-fold cross-validation.

The main advantages of this approach are:

• More Robust Evaluation: By averaging over k different validation sets, the final metric is less sensitive to the randomness of a single split.
• Efficient Data Usage: Every data point in your original training set is used for validation exactly once. This is especially valuable when you don't have a lot of data to spare.

Implementing K-Fold Cross-Validation from Scratch

As you prefer, we will now build this mechanism from the ground up using Python and NumPy. Our implementation will consist of two main parts: a function to generate the indices for each fold, and a loop to perform the training and evaluation.

Part 1: Splitting Data into Folds

Our first task is to write a function that takes our data and the number of folds, k, and returns a list of index pairs, where each pair contains the indices for one train/validation split.

The article below provides a clear and concise from-scratch implementation that we can adapt.

The core idea from the article is to:

1. Calculate the size of each fold.
2. Create an array of indices from 0 to n_samples - 1.
3. Loop k times. In each loop, slice out the indices for the validation fold and concatenate the parts before and after it to create the training indices.

Part 2: The Full Workflow from Scratch

Now let's put it all together into a reusable function. This function will perform the entire k-fold cross-validation process. For demonstration, we'll use a simple LogisticRegression model from scikit-learn, as our focus here is on the cross-validation logic, not the model itself.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

def k_fold_cross_validation(X, y, k=5, shuffle=True, random_state=42):
    """
    Performs k-fold cross-validation from scratch.

    Args:
        X (np.ndarray): Feature data.
        y (np.ndarray): Label data.
        k (int): The number of folds.
        shuffle (bool): Whether to shuffle the data before splitting.
        random_state (int): Seed for reproducibility of the shuffle.

    Returns:
        tuple: A tuple containing a list of scores and the mean score.
    """
    # Create an array of indices
    indices = np.arange(X.shape[0])

    if shuffle:
        # For reproducibility, we seed the random number generator
        np.random.seed(random_state)
        np.random.shuffle(indices)

    fold_size = len(indices) // k
    scores = []

    print(f"Starting {k}-fold cross-validation...")
    for i in range(k):
        # Determine the start and end index for the validation fold
        start, end = i * fold_size, (i + 1) * fold_size
        
        # Define validation and training indices
        val_indices = indices[start:end]
        train_indices = np.concatenate([indices[:start], indices[end:]])
        
        # Split the data into training and validation sets for this fold
        X_train_fold, y_train_fold = X[train_indices], y[train_indices]
        X_val_fold, y_val_fold = X[val_indices], y[val_indices]
        
        # --- Model Training and Evaluation ---
        # In future lessons, you will place your own from-scratch models here.
        model = LogisticRegression(solver='liblinear') 
        model.fit(X_train_fold, y_train_fold)
        
        # Make predictions on the validation set
        y_pred = model.predict(X_val_fold)
        
        # Calculate score and store it
        fold_score = accuracy_score(y_val_fold, y_pred)
        scores.append(fold_score)
        
        print(f"  Fold {i+1}/{k}: Validation Accuracy = {fold_score:.4f}")

    mean_score = np.mean(scores)
    std_dev = np.std(scores)
    print(f"\nAverage Validation Accuracy: {mean_score:.4f} (+/- {std_dev:.4f})")
    
    return scores, mean_score

# --- Example Usage ---
# Use scikit-learn to generate a sample classification dataset
from sklearn.datasets import make_classification
X_dummy, y_dummy = make_classification(n_samples=500, n_features=10, n_informative=5, n_redundant=0, random_state=42)

# Run our from-scratch k-fold cross-validation
k_fold_cross_validation(X_dummy, y_dummy, k=5)

Take a moment to read through this code. It directly implements the logic we've discussed. We first shuffle the indices (a crucial step, just like in the last lesson), then iterate through k folds, splitting the data, training the model, and collecting scores.

A Critical Warning: Data Leakage and Preprocessing

There's a common and subtle mistake that can invalidate cross-validation results: data leakage. This happens when information from the validation set unintentionally "leaks" into the training process.

A prime example is feature scaling (e.g., standardizing your data to have a mean of 0 and a standard deviation of 1). If you scale the entire training dataset before performing cross-validation, the minimum, maximum, or mean values used for scaling are calculated from all the data, including the samples that will end up in your validation fold. This gives the model a hint about the data it's supposed to be evaluated on, leading to an artificially inflated score.

The Correct Way: Any data preprocessing step that learns parameters from the data (like a scaler) must be done inside the cross-validation loop.

1. Fit the scaler only on the training fold (X_train_fold).
2. Transform both the training fold and the validation fold using the fitted scaler.

The following video explains this concept clearly. While it uses scikit-learn's Pipeline to solve the problem elegantly, the core principle is what matters for our from-scratch understanding.

Conclusion

In this lesson, we have leveled up our model evaluation toolkit significantly. By moving from a single validation set to k-fold cross-validation, we can get a far more trustworthy estimate of how our model will perform on unseen data.

Key Takeaways:

• K-fold cross-validation provides a more robust model evaluation by averaging performance across k different train/validation splits of your data.
• It's a data-efficient technique, making it ideal for situations where data is limited.
• The entire cross-validation process is performed on the training portion of your data, leaving the test set untouched for a final, unbiased evaluation.
• Implementation from scratch involves shuffling indices, partitioning them into k folds, and looping through them to train and evaluate a model.
• To prevent data leakage, any preprocessing steps (like feature scaling) must be fitted only on the training data within each fold of the cross-validation loop.

Preview of the Next Lesson:

So far, we have been using "accuracy" as our performance metric. It's simple and intuitive, but it can be misleading, especially for classification problems with imbalanced classes (e.g., a dataset with 99% non-fraudulent and 1% fraudulent transactions). In the next lesson, we will dive deeper into classification metrics. You will implement and interpret a confusion matrix, precision, recall, and the F1-score, giving you a much more nuanced understanding of a classifier's performance.