Bridging Training Corruption and Iterative Inference with Remasking

Welcome back. In the last lesson, you counted denoiser calls for full-sequence and blockwise generation. Both samplers repeatedly fed partially completed text back into the model. Today we’ll ask a different question: are those intermediate inputs like the ones the denoiser saw during training?

The answer is “not necessarily.” Understanding why will make the role of remasking much clearer—and help you interpret what your small model can and cannot fix during generation.

Two ways to get a partially masked sequence

During training, you start with a clean example from the corpus. For each example, you choose a noise level and independently decide which eligible positions to mask. The denoiser sees the resulting sequence and predicts the original characters at its masked positions. A new training example gets a fresh masking decision; it does not depend on the model’s previous predictions.

During generation, you have no original answer to corrupt. You start with masks and let the model supply candidate characters. On the next call, the model sees a sequence containing some of its own earlier choices.

Watch this short sampling explanation for a visual account of why predicting every blank is different from deciding which predictions to retain:

In your notebook’s full-sequence sampler, the selection is greedy: it commits the highest-confidence predictions for some still-masked positions. The blockwise sampler adds another restriction: once a block is complete, its characters stay fixed. Neither sampler knows whether a committed character matches any particular corpus example.

Where the training–inference gap appears

The gap is not simply that training uses partial text and inference uses partial text. How that partial text was obtained matters.

At training time, a visible character is a character from the clean example. If the clean sequence is $x$ and its masked version is $z$, every visible position satisfies $z_i=x_i$. At inference time, a visible character may instead be an earlier model prediction. It can be wrong, and the model may have chosen to expose that position precisely because its confidence was high.

The opening overview of the following article names this distinction:

Imagine an early pass on a mostly masked phrase predicts “dog” in a position where “cat” would ultimately fit the surrounding text better. Once “dog” is visible, later predictions condition on it. The model might produce a locally sensible continuation for dog rather than reconsidering the earlier choice. More generally, several individually plausible predictions can form a poor combination.

That does not mean every generated intermediate state is impossible under the training procedure. A state produced by the model might also resemble a masked version of some clean corpus sentence. The narrower, important point is that training does not deliberately present the model with its own mistaken commitments paired with the correction it should make. Moreover, confidence-based selection makes the visible positions model-dependent, whereas training masks were sampled independently of what the denoiser would predict. A low masked fraction alone does not erase that distinction.

What exactly does “remasking” fix?

There are two related actions that are easy to conflate:

1. Discard a fresh candidate. A model call predicts characters for all current masks, but the sampler retains only some. The other positions remain masked for the next call. This gives those positions another prediction with more visible context; it does not revise a character that was previously committed.
2. Reopen a committed position. The sampler replaces an already visible, model-generated character with a mask. A later call can then predict a different character there.

Your current irreversible-commitment samplers do the first kind of selection: they leave unselected positions masked. They do not do the second. Once “dog” is committed, further calls may fill the blanks around it, but they cannot change “dog.”

Read the discussion of frozen tokens to see why the second kind can be useful:

If an earlier “dog” is reopened, a later prediction can use the surrounding characters that have since appeared and choose “cat” instead. This is the practical promise of remasking: a tentative decision need not be permanent.

It is not a guarantee of self-correction. Confidence can be misleading, so a sampler may keep a confident mistake or remask a correct character. Reopening positions also takes additional model calls, and the denoiser was still trained primarily to fill masks in independently corrupted clean text—not explicitly to repair every kind of state its sampler might create. Remasking gives the sampler an opportunity to recover; it does not by itself remove the entire training–inference gap.

Takeaways

Training presents fresh, independently masked versions of clean text. Iterative inference presents states shaped by earlier model predictions and reveal decisions. That difference can let mistakes influence later predictions, especially when committed tokens are frozen. Leaving unselected fresh candidates masked supports gradual filling; remasking previously committed tokens additionally permits revision.

Next, you’ll look at what the masked denoising objective trains the model to predict, and why it is not the usual left-to-right likelihood calculation.