Diagnosing Sequence-Model Overfitting Across Time and Instruments

Hello! Last lesson turned your stateful GRU into a trainable model using truncated backpropagation through time, masked loss, and gradient clipping. A falling training loss now tells you that optimization is working—but not whether the model will predict well on later streams.

In this lesson, you will diagnose that distinction. You will compare checkpoints on past and future data, then break the results down by time period and instrument. The aim is to tell ordinary overfitting apart from a difficult new period, an instrument-specific failure, or an evaluation mistake.

First make “unseen” mean future

Nearby observations in a sequence often share information. If you randomly assign steps to training and validation, the model may train on observations that occur after the ones you ask it to predict. It may also see nearly identical conditions on both sides of the split. A good random-split score would then answer the wrong question.

Watch the chronological-split explanation in Cross-Validation for Time Series Forecasting, by Egor Howell:

A practical split for the two-output GRU is to choose global time boundaries across all instruments: train on earlier periods, use a later period for checkpoint selection, and keep a still-later period untouched for a final assessment. A simple instrument-by-instrument random split is not a substitute; the model could still learn from future market conditions through other instruments.

Rolling-origin evaluation repeats that idea at several boundaries. Fit a fresh model for each fold using only information available before that fold’s validation period. An earlier fold’s validation period may become training data in a later fold, but checkpoints and hidden states must not be carried between these independently fitted models. This gives you evidence about performance across changing periods rather than one fortunate boundary.

The boundary must apply to information, not just feature rows. If a training label at time $t$ is computed using a future outcome that extends into validation, it is not a label you would have had at the training cutoff. Likewise, fit preprocessing statistics using only the training portion of each fold. A chronological split cannot rescue a feature or normalization rule that already incorporates future information.

Compare checkpoints on the same basis

The loss printed inside last lesson’s training loop is useful for debugging, but it is not automatically comparable with validation loss. It comes from training-mode chunks, possibly with different numbers of eligible targets and different total weights. For diagnosis, periodically replay both the training and validation sequences under a consistent evaluation procedure:

1. Select a saved checkpoint and call model.eval() with gradient recording disabled.
2. Traverse each sequence in chronological order. Reset hidden state at true independent-sequence boundaries, not at arbitrary chunks; include unscored inputs that affect later state.
3. Collect predictions, targets, eligibility masks, and weights. Compute masked weighted MSE from the total weighted squared error divided by total eligible weight, rather than averaging chunk losses equally.
4. Apply the exact global weighted-Pearson scorer established earlier to the collected eligible predictions for each split.

The training-set Pearson score is an in-sample diagnostic, not a second validation score. Applying the same inference behavior and scorer to both splits makes their difference more interpretable. Retain the masked MSE as a diagnostic because it is the optimization surrogate; use the competition metric to judge the forecasting objective. They need not rank checkpoints identically: correlation measures association after centering and scaling, whereas MSE penalizes prediction magnitude and offset.

Google’s A scientific approach to improving model performance explains why inspecting curves is more informative than retaining only each run’s best validation number:

For example, suppose comparable checkpoint evaluations produce:

These illustrative numbers are evidence of problematic overfitting after epoch 6: the model keeps improving on data used to fit it while worsening on future data, including on the metric that matters. If your selection rule is weighted Pearson, epoch 6 is the better checkpoint of these three. Do not select an epoch merely because training loss is lowest.

A gap by itself is less decisive. Training and validation periods can have different target variability, weights, or difficulty. Look for a persistent divergence across checkpoints, then check whether it survives examination of the underlying periods and instruments.

Locate where generalization fails

A single global score can conceal a model that works for one part of the stream and fails elsewhere. For each checkpoint worth inspecting, retain enough prediction records to group eligible entries by time period and instrument. Report, at minimum, the number of eligible entries, total eligible weight, masked MSE, and the relevant weighted-Pearson diagnostic for each group. If the two output channels behave differently, report them separately too.

Do not average period-level or instrument-level Pearson scores and call the result the competition’s global score. Pearson correlation depends on global weighted means and variation; it is not an additive chunk statistic. Recompute the official score over all eligible entries for the official result, and use group scores only to locate failures. A small group, or one with nearly constant targets or predictions, may have an undefined or unstable correlation; its count, weight, and variability are part of the diagnosis.

Consider this illustrative pattern from one checkpoint:

Both instruments weaken in the later validation period. That points first toward a time-dependent change or a problem affecting the whole validation pipeline, rather than one instrument’s quirks. By contrast, if A remained stable while B failed in both validation periods, inspect B’s coverage, feature distributions, masks, and sequence boundaries before changing the entire architecture. Neither pattern alone proves its cause.

Grouped evaluation answers a different question: does the model generalize to instruments absent from training? The scikit-learn guide describes why dependent observations from the same group should be kept together when that is the question:

For this challenge, a future period containing familiar instruments is usually the primary diagnostic. Holding out entire instruments is an additional stress test if success on unseen instruments matters. To combine both questions, you must explicitly enforce both constraints: validation instruments are excluded from training, and training information precedes the validation period. Plain GroupKFold by itself does not provide that chronological guarantee.

Turn the pattern into a controlled next step

Before changing the model, audit the evaluation path. Check that masks and weights match the scored entries, targets remain aligned with predictions, input features contain no unavailable future information, and hidden state resets at real boundaries. In particular, validation should replay state using the selected checkpoint—not reuse a state produced during an earlier training epoch with older parameters.

Then let the observed pattern guide a small change:

Regularization and early stopping can address a genuine train–validation divergence, but they do not fix leakage, incorrect state handling, or a changed data-generating process. Change one factor at a time and keep the split and scoring code fixed so that the comparison has meaning.

Finally, remember that repeatedly choosing architectures, epochs, and preprocessing from the same validation period gradually adapts your decisions to that period. Use forward-time folds to check whether a promising choice is consistent, and reserve an untouched later period—if the available data permits—for the final estimate. The eventual competition evaluation remains unseen; a local validation result is evidence, not a guarantee.

Takeaways

Diagnosing sequence-model overfitting requires more than watching training loss fall. Evaluate saved GRU checkpoints by chronological replay, compare training and future validation under the same masks and scorer, and inspect when and for which instruments performance breaks down. A rising validation error alongside improving training error supports an overfitting diagnosis; a failure isolated to a period or instrument calls for a more specific investigation.

Next, you will take a selected stateful model toward deployment by exporting it to ONNX and checking that ONNX Runtime reproduces its PyTorch predictions.