Bidirectional Attention and Explicit Noise-Level Conditioning in Text Denoising

Welcome to the denoiser-building module. In the first module, you treated a sentence as a sequence of tokens, hid some valid tokens with [MASK], and kept padding separate from real text. Now the question is: what should a model be allowed to look at when it guesses the hidden tokens?

This lesson explains why a masked-text denoiser normally uses bidirectional attention, unlike a next-token language model. You’ll also see why we might tell the denoiser the current noise level—and why a simple mask-only model can sometimes do without that extra input. Allow about 15–20 minutes.

The clue might be on the right

Consider a corrupted sentence:

The [MASK] sat on the mat.

To predict the missing word, “The” offers little help. “Sat on the mat” provides much more. A denoiser’s job is to use the visible tokens on either side of a masked position to predict what was there before corruption.

A next-token model has a different job: predict what comes after the text seen so far. During training, causal attention prevents each position from looking at tokens to its right; otherwise it could see information that would not exist yet during left-to-right generation. That restriction is appropriate for next-token prediction but counterproductive for filling a gap in the middle of text.

Watch these short portions of Transformers & Diffusion LLMs: What’s the Connection? by Julia Turc. They distinguish full attention from causal attention, then show how masked-token prediction uses the former.

For our denoiser, “bidirectional” means a position can attend to all valid positions, including those on its right. It does not mean it can see the missing answer. At a masked position, the input contains [MASK], not the original token. Other masked positions likewise supply placeholders rather than their hidden answers. The model must combine whatever genuine context remains visible.

This also separates two kinds of attention restriction that will matter in PyTorch:

• A causal mask blocks future positions, even if they contain valid tokens. We do not want one here.
• A padding mask blocks [PAD] positions, which are not part of the text. We do want one.

The same reasoning holds for the character-level model you will build: a hidden character may be easier to recover using characters both before and after it.

One denoiser, many degrees of hiding

The first module’s corruption function can present the model with quite different inputs. Under a simple schedule where the noise level $t$ is also the probability of masking each eligible token, $t=0.2$ tends to leave plenty of context visible, while $t=0.8$ tends to leave very little. These are probabilities, not guarantees about the exact number of masks in a particular short sentence.

Watch how the video distinguishes a fixed masking rate from training on varying noise levels:

We can describe an explicitly noise-conditioned denoiser as producing token predictions from both the corrupted sequence $x_t$ and its noise level $t$:

Here, $x_t$ tells the model which tokens are actually visible; $t$ tells it which corruption level was sampled. Providing $t$ can be useful when you want one network to handle a wide range of corruption levels, or when the intended noise level is not obvious from a short, randomly masked sequence. Architectures typically turn that scalar into a vector before combining it with token representations—a step you’ll meet in the next lesson.

There is an important qualification: explicit noise conditioning is optional for a simple mask-only denoiser. The pattern of visible tokens and [MASK] symbols already tells it a great deal about how much of this particular input is missing. If masking is random and visible tokens are always genuine, giving $t$ does not reveal the hidden answers or make the visible tokens more trustworthy. Some models therefore infer the degree of corruption from $x_t$ alone.

The practical choice is to be clear about which model you are implementing. If you pass $t$, use the same noise-level convention in training and generation. If you omit it, the denoiser must work from the corrupted sequence itself. Neither choice changes the central attention rule: let valid tokens on both sides contribute; exclude padding.

What to carry forward

A next-token model uses causal attention because its predictions must depend only on the prefix. A masked denoiser predicts missing content within a corrupted sequence, so right-hand context is legitimate and often essential. Full attention does not leak the answer: masked positions contain placeholders, while padding is excluded separately.

The denoiser may also receive the sampled noise level as an explicit input, but mask-only models do not always need it. Next, you’ll put these ideas into a small PyTorch module by combining token and position representations with a noise-level representation.