Recommended resources
The freely available resources this lesson will direct you to.
Hello! This course will take you from a mental model of masked text diffusion to a small implementation you can train and sample from. Today, there is no code or diffusion math. The goal is to see how two kinds of language model can construct the same short sentence in different ways.
We will use words as toy tokens for readability. Real tokenizers may split words into smaller pieces, and your eventual model will work at the character level; the generation contrast is the same.
One position at a time: autoregressive generation
Imagine a model generating “the cat sleeps.” An autoregressive model predicts the next token from the tokens it has already produced. Its working sentence grows at the right-hand edge:
| Prediction round | Text available afterward |
|---|---|
| Start | nothing yet |
| 1 | the |
| 2 | the cat |
| 3 | the cat sleeps |
| 4 | the cat sleeps . |
At round 3, the model can use the cat to predict sleeps, but it cannot use the eventual period as context: that token does not exist yet. Once it emits cat, ordinary left-to-right generation does not go back and replace it. Later tokens must work with that choice.
This is a dependency between generation rounds, not a claim that every calculation inside a model runs one at a time. The model may process many positions during one evaluation, but it cannot decide the next generated token until the preceding generated tokens are known.
Several positions in view: masked diffusion
Watch this brief comparison before tracing the alternative. Focus on what the model starts with and whether positions must be filled from left to right.
Diffusion Language Models: The Next Big Shift in GenAI
Watch “Diffusion Language Models: The Next Big Shift in GenAI” by Jia-Bin Huang for a visual contrast between left-to-right generation and progressive unmasking.
Watch the comparison. Notice the qualification at the end: in the simple unmasking procedure shown, a revealed token stays fixed, so revealing several positions does not automatically provide error correction.
A simple masked diffusion generator starts differently: it reserves slots for a sequence and puts a special [MASK] token in every slot. A mask means “the content here is not yet visible.” It is not a word in the final sentence.
Here is one possible generation path for our four-token sentence:
| Denoising round | Current sequence |
|---|---|
| Start | [MASK] [MASK] [MASK] [MASK] |
| 1 | [MASK] cat [MASK] . |
| 2 | the cat [MASK] . |
| 3 | the cat sleeps . |
The striking difference is round 1: two positions become visible together, including the last position before the first. When predicting sleeps in round 3, the model can use both cat on its left and . on its right. That is bidirectional context.
Do not read the table as a model secretly knowing the answer “the cat sleeps.” before generation starts. We chose a completed sentence to make the two traces comparable. During actual generation, the model makes predictions from the partially masked sequence it currently sees. It could produce a different sentence or make a poor choice.
Also, “predicting across the sequence” does not mean that one model call necessarily finishes the sentence. At a round, the model can propose tokens for multiple masked positions; the sampling procedure decides which proposals become visible before the next round. That choice is why generation can take several denoising rounds.
What is learned, and what can change?
The Kuleshov Group’s How to Build a Diffusion Language Model connects this generation picture to its training setup. Read only the short subsection below; you do not need its equations or later model variants yet.
How to Build a Diffusion Language Model | Kuleshov Group
Read the Kuleshov Group’s explanation of masked diffusion to connect partially hidden training text with generation from an entirely hidden sequence.
In “Simple Masked Diffusion Models,” find the subsection “Masked Diffusion in a Nutshell.” Start with the training idea, then continue from the paragraph beginning “Once we trained the unmasking transformer” through the sampling description. Focus on what the model predicts versus which predictions the sampling procedure keeps visible. You can stop at the paragraph ending “arbitrary order”; the later examples are optional.
Here is the distinction behind that reading. During training, we have clean text, hide some of its tokens, and teach the model to predict the hidden originals. During generation, we do not have an original sentence to recover. We start with masks and use what the model learned to create a new sentence. The next lesson will examine this corruption-and-denoising relationship more carefully.
There is more than one way to manage tentative predictions. In the simple trace above, once cat is revealed, it stays fixed. In the reading’s infill-and-remask procedure, a round can produce a rough guess for every blank, then hide some fresh guesses again before the next round. Those guesses are proposals, not necessarily permanent commitments. Later, you will implement and compare such choices.
Keep the core contrast compact:
- Autoregressive: the visible prefix grows from left to right; the next token depends on that prefix.
- Masked diffusion: a sequence of slots exists from the start; repeated predictions make more of it visible, potentially in a non-left-to-right order and with context on both sides.
- Neither method has a free speed advantage: diffusion may reveal multiple positions per round, but it may need multiple model evaluations. We will measure that trade-off later.
Takeaway
If you can picture the two tables, you have the foundation for the course: one model extends a prefix, while the other denoises a sequence of masked positions. “Diffusion” here refers to this learned reversal of corruption, not to writing words left to right in disguise.
Next, we will look at the two processes behind that phrase: how clean text is deliberately corrupted with masks and how a model learns to predict what was hidden.