Absorbing-Mask Diffusion: Forward Trajectories and Direct Sampling

Recommended resources

The freely available resources this lesson will direct you to.

Hello! In the previous lesson, you used the signal level $α_{t}$ to find the chance that a clean token is still visible at noise level $t$ . Now there is a subtle question: if you already have a partly masked sentence, how do you move it to a noisier level without ever uncovering a mask? We will trace one sentence through that process, then see why you can also sample its state at a chosen level directly from the clean sentence.

Set aside about 15–20 minutes for the video, reading, and worked example.

A mask is an absorbing state

In this forward process, a valid position can contain either its original token or [MASK]. Once it becomes [MASK], it stays masked for the rest of the forward trajectory. That is what absorbing-mask means. The learned reverse process will eventually try to uncover tokens, but it is not involved in today’s trajectory.

Watch this short segment of Diffusion Language Models: The Next Big Shift in GenAI by Jia-Bin Huang for a visual explanation.

Diffusion Language Models: The Next Big Shift in GenAI

Watch how a special mask token serves as the noise state for discrete text, rather than adding numerical noise to a token.

Watch the mask illustration. Focus on the claim that, during forward corruption, a position that becomes masked remains masked at later noise levels. You can leave the discussion of one-hot vectors aside for now.

The picture below shows the other view we will need: at a single chosen level $t$ , each token in the clean sequence is masked with probability $1 - α_{t}$ .

A clean sequence $x$ is sampled into a partially masked state $z_t$ at signal level $\alpha_t$; each valid token is retained with probability $\alpha_t$. The bottom row represents the fully masked endpoint.

A snapshot such as $z_{t}$ does not, by itself, tell us which intermediate states it passed through. To construct one consistent trajectory, we need a rule for moving between levels.

Moving between two noise levels

Read the forward-process explanation in How to Build a Diffusion Language Model by the Kuleshov Group. The key distinction is between masking from clean text and masking from an already corrupted state.

How to Build a Diffusion Language Model | Kuleshov Group

Read the forward-process description to connect the single-level masking rule with a progressive trajectory.

In “Understanding Masked Diffusion: A Probabilistic Perspective,” under “Forward process,” start at “The goal of the forward process is to generate training data for the reverse process.” Read through the forward explanation, ending after the paragraph that begins “We implement this process as a Markov chain.” Focus especially on what happens to tokens that are still unmasked when moving from level s to a noisier level t.

Suppose $s < t$ , so $α_{s} > α_{t}$ . At level $s$ , a token has probability $α_{s}$ of still being visible. At level $t$ , we want its probability of still being visible to be $α_{t}$ . We only make a new random decision for positions that are still visible at $s$ :

P (stay visible at t ∣ visible at s) = \frac{α _{t}}{α _{s}} .

Consequently, the chance of masking one of those surviving positions is

P (mask between s and t ∣ visible at s) = \frac{α _{s} - α _{t}}{α _{s}}, α_{s} > 0.

Already masked positions remain masked; we never draw a new clean token for them. Notice why the denominator matters: $α_{s} - α_{t}$ is the drop in visibility among all original positions. The transition probability applies only to the smaller group that survived to $s$ .

Trace one sentence all the way down

Take four valid tokens, with no padding, and use the linear schedule $α_{t} = 1 - t$ :

the  cat  naps  .

The table shows one possible random trajectory, not a requirement that exactly one token be masked at each step.

Noise level $t$	Signal level $α_{t}$	State of this one trajectory
$0$	$1$	`the cat naps .`
$0.25$	$0.75$	`[MASK] cat naps .`
$0.5$	$0.5$	`[MASK] cat [MASK] .`
$0.75$	$0.25$	`[MASK] [MASK] [MASK] .`
$1$	$0$	`[MASK] [MASK] [MASK] [MASK]`

From $t = 0.25$ to $t = 0.5$ , for example, only cat, naps, and . are eligible for a new masking decision. Each has probability

\frac{0.75 - 0.5}{0.75} = \frac{1}{3}

of being masked during that transition. In the trajectory shown, naps was masked; the was already masked and could not reappear. The next transition masks each remaining visible position with probability $(0.5 - 0.25) /0.5 = 1/2$ . At the final level, all remaining visible positions are masked.

The table is deliberately concrete: the schedule specifies probabilities, while the displayed sequence is just one outcome of the random draws.

How can a direct draw give the same level?

Return to $t = 0.5$ . The previous lesson gives a shortcut: start with the clean sentence and independently keep each valid token with probability $α_{0.5} = 0.5$ , masking it otherwise. That direct draw could produce [MASK] cat [MASK] ., just like the trajectory above. A fresh draw might produce a different pattern.

Why is the shortcut legitimate? For any one token to remain visible through both $t = 0.25$ and $t = 0.5$ , it must survive both transitions. Its probability is

P (visible at 0.5) = survive to 0.25 0.75 survive from 0.25 to 0.5 \frac{0.5}{0.75} = 0.5.

That is exactly the retention probability in a direct draw at $t = 0.5$ . The same reasoning holds between any two levels with $α_{s} > 0$ : a position visible at $s$ remains visible at $t$ with probability $α_{t} / α_{s}$ , giving an overall probability of $α_{t}$ . If positions are sampled independently, this also gives the same distribution over entire masked sequences at level $t$ .

Same distribution does not mean the same realized sample. A progressive trajectory keeps its earlier masks; a separate direct draw does not know what happened on that trajectory. You would trace the chain to see a consistent history, but you can draw directly from clean text when all you need is a corrupted example at one chosen noise level.

Takeaway

An absorbing-mask trajectory only loses visible tokens as noise increases. To move between levels, mask each still-visible position with probability $(α_{s} - α_{t}) / α_{s}$ ; never change an existing [MASK]. To sample the state at level $t$ without tracing the history, retain each valid clean token with probability $α_{t}$ . Both methods give the same distribution at that level, even though independent draws need not match.

Next, you will give clean characters, [MASK], and [PAD] their own IDs in a tiny vocabulary—the representation needed to turn these masking rules into code.