Leakage-Free Train, Validation, and Test Splitting

Hello. Last lesson, you saw why a predicted probability needs checking against outcomes on data the model did not learn from. That check is only meaningful if the data really represents an unseen prediction situation. A model can produce excellent-looking results by exploiting information that would not exist when a real prediction is made.

In this lesson, you’ll construct training, validation, and test splits and check them for two distinct problems: features that reveal the target too early, and future information that slips into a past prediction. Allow about 40 minutes for the lesson, video, and readings.

Three splits, three jobs

Suppose you want to predict whether a new request will need specialist review. Each row contains information about a request, and its target records whether review was needed. Before dividing rows, decide when the prediction would actually be made—for example, at the moment the request arrives. That moment determines which information the model may use.

The three splits have different jobs:

Validation data is not used to fit the model, but repeated decisions based on validation results do make it part of development. That is why you keep the test set separate until you have settled on an approach. There is no universally correct percentage for each split. More important than a particular ratio is that each split reflects the question you need it to answer.

Read the short guideline list below. Notice its two separate safeguards: the test set is reserved for final evaluation, and preprocessing statistics are learned without consulting validation or test rows.

A useful principle is to design the split around the deployment question. If the model will predict requests arriving next month, a test set drawn from a later month is more convincing than a random sample of requests from the same period as training.

Target leakage: a feature that knows too much

A target is supposed to be known when you create training examples: you need it to teach the model. Target leakage occurs when an input feature gives the model information about that target that would not be available at prediction time.

Imagine the request-review model uses a column called review_completed. It will be strongly associated with whether specialist review was needed. But if the prediction is made when a request arrives, that column does not yet exist. Splitting the dataset perfectly will not fix the problem: the invalid column could be present in training, validation, and test rows, producing an impressive score that cannot be reproduced in use.

The following reading uses a chargeback in fraud detection to make the same point. It also introduces train–test contamination, a different problem in which information crosses a split boundary.

The practical test for a feature is not merely “Is it in my dataset?” Ask instead: Could the system obtain this value, in this form, at the exact time it must make the prediction? Apply that question to derived features too. A “number of prior reviews” feature may be valid if it counts only reviews already recorded when the request arrives; a count recomputed later from the complete database may silently include future activity.

Target leakage can involve information collected after the outcome, a field directly copied from the target, or a derived feature that incorporates the outcome. It does not require an obviously named “target” column, and a clean train–test boundary alone cannot prevent it.

Temporal leakage: construct a split as time would unfold

When deployment means predicting the future from the past, randomly assigning dated rows to splits can let the model learn from events that occur after the events on which it is evaluated. This is temporal leakage. A chronological split makes training precede validation and validation precede test, using each row’s prediction time to place it.

Here is a concrete design. Suppose a request arrives at 09:00, and you predict whether it will need specialist review within the following seven days. Its target is not fully known until those seven days have passed.

Assume targets become available exactly seven days after arrival. A request arriving on March 24 has a known target by March 31, before the first validation prediction. A request arriving on April 30 has a known target by May 7, before the test period begins. You cannot finish scoring the whole test period until the May 31 targets have matured, on June 7.

This is an illustrative, conservative design, not a rule that every time-based dataset needs two seven-day gaps. The necessary boundaries depend on the prediction horizon, label-reporting delays, how often you retrain, and whether features use overlapping time windows. Write those assumptions down before choosing dates.

Chronology also applies inside each row. A feature such as “requests submitted in the previous 30 days” must be computed using records available before that row’s 09:00 prediction time. Sorting rows by date does not repair a feature that was calculated retrospectively from a database containing later events.

Random splitting has another weakness when rows are related. Watch this example of flights whose delays are connected, then consider which relationship a split needs to preserve.

Grouping and chronology answer different questions. If deployment involves entirely new entities—such as predicting for aircraft never represented in training—keeping all rows for an entity in one split may be essential. If deployment involves future observations of existing entities, requiring entirely new entities in the test set could instead answer the wrong question. Choose the holdout rule that matches the real prediction setting; when both time and entity independence matter, check both boundaries.

Keep information from crossing the boundaries

Even well-chosen rows can be contaminated by work done before or during model fitting. Suppose you calculate the mean and standard deviation of a numerical feature across the full dataset, then standardize each row and finally split the rows. The resulting training features already reflect information from validation and test data.

The safe pattern is: establish the splits first; fit the scaler, missing-value imputer, or other learned preprocessing step on training data; then use that fitted step to transform validation and test data. Do not fit a separate scaler on validation or test data—that would give each split its own learned transformation rather than applying the one the model will use. The same boundary matters for feature selection informed by targets: choosing features after examining all rows lets held-out outcomes influence development.

For the request example, you could document the design in this order:

1. Define the prediction contract. At 09:00 when a request arrives, predict whether specialist review will be needed within seven days.
2. Audit each proposed feature. Record where it comes from and when its value becomes available. Reject or reconstruct anything that uses later information.
3. Assign rows by prediction date. Use the chronological training, validation, and test periods above; allow targets to mature before they are used.
4. Fit only on permitted development data. Learn preprocessing and model parameters from training rows; use validation results to select an approach.
5. Check the selected approach on test once. Preserve the test set as evidence about a decision already made, rather than using it to make that decision.

You may later refit the selected approach on both training and validation rows if their targets would be available by the simulated deployment time. In that case, refit its preprocessing on that combined development data as well. The test rows remain untouched. If a disappointing test result leads to substantial changes and repeated testing, that test set is no longer an independent final check; a fresh future holdout is needed to assess the revised approach.

Key takeaways

A reliable split is more than three non-overlapping files. Training fits the model, validation guides development, and test checks the chosen approach. To prevent target leakage, verify that every feature would exist when the prediction is made. To prevent temporal leakage, place rows by prediction time, account for when their targets become known, and calculate historical features using only information available at that time. Finally, fit preprocessing on development data—not on held-out rows.

In the next lesson, you’ll use training and validation results to diagnose underfitting and overfitting. The split you designed here makes that comparison worth trusting.