Reproducible Numerical Experiments: Seeds, Configuration, and Environment Metadata

Hello! Your last lesson put the plotting code and its toy input under Git, so you can identify which version produced a figure. A numerical experiment needs a few more facts: the random seed, the settings used for that run, and the software environment. Without them, the same commit may produce a different result—or leave someone guessing how to rerun it.

This lesson takes about 40–50 minutes. You’ll add a small simulated experiment to the signal-study repository, save a run record, and check that a second execution reproduces its result. The five input readings remain illustrative; nothing here is a measured machine-learning result.

A seed fixes a starting point, not an entire experiment

A computer’s pseudorandom-number generator produces a sequence from an initial state. Supplying the same seed lets you restart that sequence. It does not, by itself, fix changes to the code, the number or order of random draws, the input data, or the library implementation.

NumPy recommends creating a local Generator with default_rng, rather than relying on a shared global random state. Read its documentation with that distinction in mind.

In this lesson, rng = np.random.default_rng(2025) will create a fresh generator each time the script starts. Draws within one execution still consume its state. That is why adding an extra random draw near the top of a script can change later values even when the seed stays fixed.

Put run choices in a configuration file

Instead of editing numbers scattered through Python code, keep the run choices together in config.yaml. YAML is a human-readable format for settings. The short video introduces its key–value syntax and shows how to load it into a Python dictionary.

Open your existing signal-study repository and activate its isolated Python environment. Install the packages needed by both the earlier plot and this experiment:

python -m pip install numpy pandas matplotlib PyYAML

If this is a dedicated project environment, record its installed package versions:

python -m pip freeze > requirements.txt

The resulting requirements.txt may include packages installed as dependencies of those four. On a fresh environment, python -m pip install -r requirements.txt installs the recorded versions. A freeze from a shared environment may contain unrelated packages, which is another reason to use an isolated environment.

Create config.yaml beside plot.py:

seed: 2025
trials: 20
noise_std: 0.05

Here, one trial adds freshly generated noise to each of the five illustrative readings and calculates their mean. noise_std controls the size of that simulated noise. Moving these values into a file makes the run’s choices inspectable without searching the script.

Save the result and the conditions that produced it

Create experiment.py in the same directory. It reads the existing data/measurements.csv, generates 20 simulated trial means, and writes two files under outputs/numerical-run/: the means themselves and a JSON run record.

import csv
import hashlib
import json
import platform
from pathlib import Path

import numpy as np
import yaml


def sha256(path):
    return hashlib.sha256(path.read_bytes()).hexdigest()


root = Path(__file__).resolve().parent
config_path = root / "config.yaml"
data_path = root / "data" / "measurements.csv"
requirements_path = root / "requirements.txt"

with config_path.open(encoding="utf-8") as file:
    config = yaml.safe_load(file)

seed = int(config["seed"])
trials = int(config["trials"])
noise_std = float(config["noise_std"])

if seed < 0 or trials < 1 or noise_std < 0:
    raise ValueError("Use a nonnegative seed and noise_std, and at least one trial")

readings = np.loadtxt(data_path, delimiter=",", skiprows=1, ndmin=1)
rng = np.random.default_rng(seed)

noise = rng.normal(0.0, noise_std, size=(trials, readings.size))
trial_means = (readings[None, :] + noise).mean(axis=1)

run_dir = root / "outputs" / "numerical-run"
run_dir.mkdir(parents=True, exist_ok=True)
results_path = run_dir / "trial_means.csv"

with results_path.open("w", encoding="utf-8", newline="") as file:
    writer = csv.writer(file)
    writer.writerow(["trial", "mean"])
    for trial, mean in enumerate(trial_means, start=1):
        writer.writerow([trial, repr(float(mean))])

record = {
    "config": {
        "seed": seed,
        "trials": trials,
        "noise_std": noise_std,
    },
    "environment": {
        "python": platform.python_version(),
        "os": platform.system(),
        "architecture": platform.machine(),
        "numpy": np.__version__,
        "pyyaml": yaml.__version__,
        "bit_generator": type(rng.bit_generator).__name__,
    },
    "sha256": {
        "experiment_py": sha256(root / "experiment.py"),
        "config_yaml": sha256(config_path),
        "input_csv": sha256(data_path),
        "requirements_txt": sha256(requirements_path),
        "trial_means_csv": sha256(results_path),
    },
    "summary": {
        "average_trial_mean": float(trial_means.mean()),
    },
}

record_path = run_dir / "run.json"
record_path.write_text(
    json.dumps(record, indent=2, sort_keys=True) + "\n",
    encoding="utf-8",
)

print(f"Average trial mean: {trial_means.mean():.6f}")
print(f"Results SHA-256: {sha256(results_path)}")
print(f"Run record: {record_path}")

The expression readings[None, :] gives the five readings a new leading dimension, so NumPy can add them to the array of noise with one row per trial. You met this kind of broadcasting earlier in the module. The script then averages across each row.

Run it from the project directory:

python experiment.py

Inspect outputs/numerical-run/trial_means.csv and run.json. The CSV holds the result for each trial. The JSON holds the settings, relevant environment versions, a summary, and SHA-256 hashes of the code, configuration, input, dependency list, and result file. A hash acts as a compact fingerprint of file contents: it helps you detect that a file has changed, but it cannot recreate a missing file.

You do not need MLflow for this exercise. The important practice is to keep a result connected to the conditions that produced it.

Test the claim of reproducibility

Run python experiment.py a second time without changing any files. Compare the printed average and results SHA-256 hash with the first run. On the same setup, both should match. If they do not, check that you started a new execution of the script, did not edit config.yaml or the CSV, and are using the intended Python environment.

Now change only seed: 2025 to seed: 2026 in config.yaml and run it again. The generated trial means—and normally their hash—should change. Restore seed: 2025 and rerun: the original result hash should return. This distinguishes a controlled change from unexplained variation.

Before committing, add a short description to README.md stating that python experiment.py reads config.yaml, uses illustrative input data, and writes its CSV and run record to outputs/numerical-run/. Then inspect and commit the reproducible recipe:

git status
git add README.md requirements.txt config.yaml experiment.py
git diff --staged
git commit -m "Record configuration and environment for numerical run"

Your earlier .gitignore excludes outputs/, so the generated files are not in this commit. Another person can regenerate them from the committed recipe. If you later present a particular result in a report, retain or publish its CSV and run record too; code that could regenerate a result is not a substitute for preserving the result you actually reported. This demonstration also overwrites the same output files on each run, so use distinct run directories when you need to retain multiple experiments.

A matching seed is strongest as a repeatability check within the recorded setup, not a promise of identical bytes on every computer. Library versions, numerical implementations, and hardware can affect some computations. Recording those conditions makes a discrepancy investigable rather than mysterious.

What to take forward

A reproducible numerical run has more than a saved seed. You now have a Git-tracked recipe—code, input, configuration, and dependency versions—plus a run record connecting that recipe to an output. You also tested it by rerunning unchanged settings and by deliberately changing one setting.

This completes the reproducible-workflows module. The next module develops the mathematics you’ll use to reason about classical models and, later, quantum states and circuits; the same recording habits will matter when those computations become more complex.