Reproducible PyTorch Experiments: Seeds and Determinism

Welcome back. Last lesson, you saved model and optimizer state so training could continue after a stop. But even with the same code and checkpoint, random choices can change what happens next. This final lesson in the PyTorch essentials module tackles a related, simpler goal: run a small experiment from the beginning twice and obtain the same result.

You’ll control random number generators, ask PyTorch to use deterministic operations, and check reproducibility by comparing two complete training runs. That gives you a dependable starting point for the larger training experiments ahead.

What can change between runs?

A training run may use randomness when it initializes weights, shuffles examples, or applies dropout. Python, NumPy, and PyTorch have separate random number generators (RNGs). Giving each the same seed value makes its sequence repeat on a fresh run; seeding PyTorch does not also seed NumPy or Python.

There is a second issue: an operation can produce different results even when no new random number is drawn. Some hardware implementations may execute work in an order that varies. Seeding controls random choices; deterministic settings control the implementations of operations. Both matter, but neither guarantees identical results across different hardware or PyTorch versions.

The image is a useful checklist, not an all-or-nothing guarantee. For example, a DataLoader without worker processes does not need worker seeding, while an experiment that uses NumPy’s newer, separately created Generator objects must seed those objects explicitly.

Read the relevant parts of PyTorch’s Reproducibility note before applying the settings. It distinguishes what each control does and states the limits of reproducibility.

For CUDA convolutions, torch.backends.cudnn.benchmark = False prevents a timing-based search from selecting different algorithms on different runs. It does not ensure that the selected algorithm itself is deterministic. torch.use_deterministic_algorithms(True) requests deterministic implementations more broadly and raises an error when PyTorch knows an operation has no deterministic alternative. We will set torch.backends.cudnn.deterministic = True as an explicit cuDNN setting too, although the broader PyTorch setting also covers cuDNN convolution determinism.

Run the same small experiment twice

The following experiment trains an MLP on six labeled examples. It deliberately uses all three RNGs: NumPy adds a little input noise, Python shuffles the example order, and PyTorch initializes the model and applies dropout. The data and model are small so you can concentrate on the reproducibility check, not classifier quality.

Run the entire block on CPU. Calling run(42) starts a fresh model and optimizer each time; importantly, configure_reproducibility() runs before data creation and model initialization.

import random
import numpy as np
import torch
from torch import nn

def configure_reproducibility(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)  # Seeds PyTorch RNGs for CPU and CUDA.

    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True
    torch.use_deterministic_algorithms(True)

def run(seed):
    configure_reproducibility(seed)

    base_x = np.array([
        [1., 0., 0., 0.],
        [0., 1., 0., 0.],
        [0., 0., 1., 0.],
        [2., 0., 0., 0.],
        [0., 2., 0., 0.],
        [0., 0., 2., 0.],
    ], dtype=np.float32)
    noise = np.random.normal(
        loc=0.0, scale=0.02, size=base_x.shape
    ).astype(np.float32)
    X = torch.from_numpy(base_x + noise)
    y = torch.tensor([0, 1, 2, 0, 1, 2], dtype=torch.long)

    model = nn.Sequential(
        nn.Linear(4, 8),
        nn.ReLU(),
        nn.Dropout(p=0.2),
        nn.Linear(8, 3),
    )
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
    loss_fn = nn.CrossEntropyLoss()

    model.train()
    losses = []

    for epoch in range(5):
        order = list(range(len(y)))
        random.shuffle(order)

        for start in range(0, len(order), 3):
            indices = order[start:start + 3]
            optimizer.zero_grad()
            loss = loss_fn(model(X[indices]), y[indices])
            loss.backward()
            optimizer.step()
            losses.append(loss.item())

    final_state = {
        name: tensor.detach().clone()
        for name, tensor in model.state_dict().items()
    }
    return losses, final_state

losses_a, state_a = run(42)
losses_b, state_b = run(42)

same_losses = losses_a == losses_b
same_parameters = all(
    torch.equal(state_a[name], state_b[name])
    for name in state_a
)

print("Identical loss histories:", same_losses)
print("Identical final parameters:", same_parameters)
assert same_losses and same_parameters

Both printed results should be True in this CPU experiment. The comparison checks the whole sequence of recorded losses and every final parameter, rather than relying on two rounded loss values that merely look alike.

Notice that the seed is set once at the beginning of each complete run, not before every batch. Dropout should still make different random choices as training proceeds; we want that sequence of choices to repeat when the entire experiment is rerun. Moving the seed call below model construction would leave initial weights uncontrolled.

When data loading has its own RNGs

The example used random.shuffle() to keep its data order visible. A PyTorch DataLoader can shuffle for you using a seeded torch.Generator. With multiple loader workers, each worker can also run Python or NumPy randomness while preparing examples. PyTorch’s worker initialization pattern seeds those libraries inside each worker.

Here is the same pattern adapted to this lesson’s variables. It belongs inside run() after X and y are created, if you choose to replace manual shuffling with a DataLoader:

from torch.utils.data import DataLoader, TensorDataset

def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % (2**32)
    np.random.seed(worker_seed)
    random.seed(worker_seed)

loader_generator = torch.Generator()
loader_generator.manual_seed(seed)

loader = DataLoader(
    TensorDataset(X, y),
    batch_size=3,
    shuffle=True,
    generator=loader_generator,
    num_workers=0,
    worker_init_fn=seed_worker,
)

With num_workers=0, loading happens in the main process, so seed_worker is not called. If you later use worker processes, the function becomes relevant; keep it at the top level of your script, as shown. A separate loader generator also makes the intended source of shuffle randomness explicit. There is no need to introduce worker processes just to make this six-example experiment reproducible.

Know what the check proves

Two identical outputs show that this experiment repeated under these conditions. They do not promise identical results after changing device, hardware, library versions, or every operation in a larger model. Deterministic algorithms can also be slower. If PyTorch raises a nondeterministic-operation error, investigate the operation and available deterministic alternative rather than silently disabling the setting and assuming the run is still reproducible.

This is also different from resuming the checkpoint in the previous lesson. Reseeding and starting from scratch replays an experiment. Resuming at a checkpoint boundary may require saving and restoring RNG states—and the data loader’s position or state—as well as model and optimizer state to match an uninterrupted run exactly.

Conclusion

To make a small PyTorch experiment repeatable, seed every RNG the experiment uses before creating data or a model, request deterministic operations, and test the claim by repeating the full run and comparing its results. Keep the device and software environment fixed when interpreting that comparison.

You have now completed the PyTorch essentials module. Next, we’ll examine how the affine layers and nonlinear activations you used in an MLP compose into a feedforward neural network.