Benchmark and Validate Resource-Constrained Submissions

Hello! The last lesson established that your selected GRU checkpoint and its ONNX export agree across evolving hidden states. This final lesson in the sequence-model sprint asks a different question: will the complete submission produce the right outputs, repeatedly, within the scorer’s limits?

You will wrap the verified graph in the competition interface, package it so it works outside your development directory, and measure the whole inference run—not just an isolated ONNX call. The result should be a submission you can audit before uploading.

Treat the submission guide as a runtime contract

Read the competition’s Wunder Fund RNN challenge submission guide first. Its interface and limits determine what you need to test; an accurate model behind an incorrect wrapper will not be a valid submission.

The scorer calls a no-argument PredictionModel() and then predict(data_point) in row order. Each data_point.state has 112 float32 features. You must update the sequence state on every call, return None when need_prediction is false, and otherwise return two finite float32-compatible values in [t0, t1] order. A changed seq_ix requires a reset.

There are four resource constraints to design around: one vCPU, 16 GB RAM, 60 minutes for the entire test set, and a ZIP smaller than 20 MB. Inference has no GPU or internet access. These are properties of the submitted program and its environment, not properties you can establish from an ONNX graph alone.

Turn the verified graph into PredictionModel

Keep solution.py small and explicit. The following is a reference wrapper for the fixed one-step ONNX interface from the previous lesson: inputs x and h_in, outputs prediction and h_out. It assumes the exported checkpoint was trained on the raw 112 features in their supplied order. If your selected model used scaling, imputation, feature selection, or a different input width, replace _prepare with exactly the preprocessing used in validation, include its fitted parameters in the ZIP, and adjust the shape check. Do not refit preprocessing on test rows.

# solution.py
from pathlib import Path

import numpy as np
import onnxruntime as ort


class PredictionModel:
    def __init__(self):
        model_path = Path(__file__).resolve().parent / "model.onnx"

        options = ort.SessionOptions()
        options.intra_op_num_threads = 1
        options.inter_op_num_threads = 1
        options.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL

        self.session = ort.InferenceSession(
            str(model_path),
            sess_options=options,
            providers=["CPUExecutionProvider"],
        )

        assert [v.name for v in self.session.get_inputs()] == ["x", "h_in"]
        assert [v.name for v in self.session.get_outputs()] == [
            "prediction", "h_out"
        ]

        # This example expects fixed ONNX dimensions [L, 1, H].
        state_shape = tuple(self.session.get_inputs()[1].shape)
        assert len(state_shape) == 3
        assert all(isinstance(d, int) and d > 0 for d in state_shape)
        self.state_shape = state_shape

        self.seq_ix = None
        self.h = np.zeros(self.state_shape, dtype=np.float32)

    def _prepare(self, state):
        # Identity preprocessing is correct ONLY if training used raw features.
        row = np.asarray(state, dtype=np.float32)
        if row.shape != (112,):
            raise ValueError(f"Unexpected feature shape: {row.shape}")
        return np.ascontiguousarray(row.reshape(1, 1, 112))

    def predict(self, data_point):
        if data_point.seq_ix != self.seq_ix:
            self.seq_ix = data_point.seq_ix
            self.h = np.zeros(self.state_shape, dtype=np.float32)

        x = self._prepare(data_point.state)
        prediction, next_h = self.session.run(
            ["prediction", "h_out"],
            {"x": x, "h_in": self.h},
        )
        self.h = next_h

        # The graph ran even if this row does not request an output.
        if not data_point.need_prediction:
            return None

        result = np.asarray(prediction[0], dtype=np.float32).copy()
        if result.shape != (2,) or not np.isfinite(result).all():
            raise ValueError("Invalid prediction shape or nonfinite value")
        return result

If the prior export is named gru_step.onnx, package that same verified file as model.onnx; do not export a different checkpoint for submission. Check that the saved ONNX file contains its weights, or include every external-data file it requires. The previous lesson’s external_data=False save is intended to produce a self-contained model file.

Notice the placement of the need_prediction check. Returning early would skip the GRU update on unscored steps, so the next scored prediction would use the wrong history. Equally, self.h belongs to one sequence at a time: even when two sequences begin with identical feature rows, the second must start from zeros rather than inherit the first sequence’s final state.

The guide lists ONNX Runtime among the installed packages, so this wrapper does not need to ship a runtime wheel or import PyTorch for inference. That saves ZIP space and startup work. The package must still contain every model-specific file that _prepare or the graph reads, addressed relative to solution.py, not relative to your shell’s working directory.

Validate behavior before measuring speed

A useful smoke test covers more than one scored row. Replay a short sequence containing an unscored row, then start a second sequence and compare its first prediction with the prediction from a fresh PredictionModel given that second sequence alone. They should agree. Compare the scored outputs of the complete wrapper against the already-verified ONNX inference path using the same prepared rows; this catches changes in feature order, preprocessing, or reset policy outside the graph.

Run that replay in two separate processes and compare the resulting output arrays or their saved bytes. A fixed random seed for generating test inputs makes the test reproducible; it does not, by itself, make inference deterministic. For identical packaged files, ordered inputs, and runtime environment, demand identical predictions across repeated runs. If they differ, inspect mutable state, preprocessing, threading, and any random operations before profiling further. Numerical closeness to PyTorch, tested last lesson, is a different check from repeatability of the submitted runtime.

Use representative streams for this test, including a long sequence and real held-out feature distributions. Record which rows required predictions, verify every other row returned None, and assert that every requested output has shape (2,), dtype compatible with float32, and finite values. Keep the validation targets and scoring code outside the ZIP: the submission only needs to predict.

Profile the complete workload, then investigate hotspots

The limit is on end-to-end inference. Session creation, feature preparation, Python-to-ONNX calls, hidden-state updates, and scorer-facing output handling all matter. Timing only session.run understates the work, especially when it is called once per row.

The Linux time(1) manual distinguishes elapsed wall time from CPU time and documents GNU time’s peak resident-memory measurement:

Build a streaming local harness that creates PredictionModel, iterates through representative DataPoint objects in their original order, calls predict on every row, and counts or checks outputs without accumulating the entire dataset in a list. Then measure an unprofiled run:

OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 \
/usr/bin/time -f 'elapsed_s=%e user_s=%U sys_s=%S peak_rss_kb=%M' \
python harness.py

elapsed_s is the relevant wall-clock quantity for a 60-minute deadline; peak_rss_kb reports the process’s maximum resident set size. Include model loading and startup in the measured command. Report the number of processed rows and sequences alongside the measurements so that a faster run is not accidentally just a shorter run.

If you cannot reproduce the entire hidden test set locally, measure enough realistically mixed sequences to include startup and steady-state behavior. With measured steady-state cost $t_{\mathrm{row}}$, startup time $T_{\mathrm{start}}$, and a defensible upper bound $N_{\mathrm{test}}$ on test rows, a first estimate is

Do not invent $N_{\mathrm{test}}$ from the 20,000-step maximum for a sequence; you also need the number and lengths of sequences. Until you have a trustworthy bound or a full-size dry run, this is a planning estimate, not proof of compliance. Leave substantial headroom below 3,600 seconds and 16 GB for machine differences and scorer overhead. Likewise, a low average memory figure cannot substitute for peak RSS.

One local CPU-affinity setting or intra_op_num_threads=1 does not make a many-core laptop equivalent to the scorer’s one-vCPU environment. If you have a prepared container matching its Python and dependencies, an offline run with a one-CPU quota and memory limit is a stronger check. Otherwise, label local results as estimates, constrain library threads as above, and avoid claiming that a fast development-machine benchmark guarantees scorer performance.

If wall time is poor, find out where it goes before changing the model. NeuralNine’s Optimize Your Python Programs: Code Profiling with cProfile demonstrates a quick function-level profile:

For example, python -m cProfile -s cumulative harness.py can show whether _prepare or harness I/O consumes surprising Python time. First optimize a demonstrated hotspot, then rerun the correctness tests and the unprofiled benchmark. An optimization that changes feature order, drops unscored steps, or makes outputs unstable is not an improvement.

Package and test the ZIP itself

Prepare a staging directory containing only files needed at inference:

submission/
├── solution.py
├── model.onnx
└── preprocessing_parameters.npz   # Only if your trained pipeline needs it

Create the ZIP from inside that directory so solution.py lands at the archive root:

cd submission
zip -r ../submission.zip .
cd ..
unzip -l submission.zip

Avoid an extra enclosing submission/ directory inside the archive. Inspect the listing for missing ONNX sidecars or preprocessing files, as well as accidental checkpoints, datasets, caches, or development environments. For a conservative decimal interpretation of the upload limit, check:

python - <<'PY'
from pathlib import Path
from zipfile import ZipFile

path = Path("submission.zip")
with ZipFile(path) as archive:
    names = set(archive.namelist())
    assert "solution.py" in names
    assert "model.onnx" in names
print("ZIP bytes:", path.stat().st_size)
assert path.stat().st_size < 20_000_000
PY

A small ZIP does not establish low RAM use: model loading and libraries occupy memory after extraction. Conversely, compressing a large model does not fix slow per-row inference. Treat archive bytes, peak RSS, and elapsed seconds as three independent measurements.

Finally, extract the ZIP into a clean directory and run the smoke test and timing harness against its extracted solution.py, not your original source tree. Change the working directory during at least one test to confirm that artifact paths really are relative to the module file. A successful run from your development directory can otherwise conceal an absolute path or an undeclared local dependency. If you make any edit after these checks, rebuild and retest the final ZIP; it is the uploaded artifact, not the source directory, that needs to pass.

Takeaways

A submission is ready when the packaged model preserves the verified ONNX predictions and state behavior, processes every row, resets on new sequences, returns the required two-value output only when requested, and repeats its results across runs. Measure end-to-end wall time and peak RSS under the closest practical approximation to one CPU, then separately verify the ZIP’s root layout and size. Where test-set size or scorer hardware prevents a definitive local guarantee, record that uncertainty and keep headroom.

This completes the time-critical RNN sprint. The next module turns to production C++ fundamentals—object lifetime, ownership, and tooling—which will support the larger systems projects ahead.