Diffusion Models for Beginners

Contrast autoregressive generation with masked diffusion generation by tracing how each method constructs a short sentence.

Explain the forward corruption and learned reverse denoising processes for text, contrasting discrete mask corruption with additive Gaussian noising.

Given a masking schedule and noise level, calculate the masking and retention probabilities for each valid token.

Trace an absorbing-mask forward trajectory for a toy sequence and show how its state at a chosen noise level can also be sampled directly from the clean sequence.

Build a tiny character-level vocabulary with padding and mask symbols, then encode and decode short text examples.

Create fixed-length padded PyTorch batches in Colab and construct a Boolean mask identifying valid, non-padding tokens.

Implement a PyTorch corruption function that samples noise levels, masks only eligible tokens, and returns both corrupted token IDs and the corruption mask.

Explain why a text denoiser uses bidirectional rather than causal attention, and when explicitly providing the current noise level is useful.

Complete a provided PyTorch module that combines token, position, and noise-level embeddings for corrupted sequences.

Configure a small Transformer encoder to use a padding mask without applying an autoregressive causal mask.

Project the encoder's hidden states into vocabulary logits at every sequence position and verify the resulting tensor shapes.

Compute a safely normalized cross-entropy denoising loss over only corrupted, non-padding positions.

Implement one optimizer step for the denoising objective, including device placement, gradient reset, backpropagation, and parameter updates.

Verify the training pipeline by deliberately overfitting the denoiser on one tiny batch.

Initialize a fully masked sequence and obtain candidate-token distributions from the trained denoiser.

Compute token confidence scores from the denoiser's vocabulary distributions.

Given a reveal schedule, calculate how many additional positions should be revealed at each denoising step.

Implement one reverse-denoising update that commits selected token predictions to a sequence.

Compose reverse updates into a complete sampler that can generate unconditionally or preserve an optional fixed prompt prefix.

Add temperature-controlled categorical sampling and reproducible random seeds to the sampler.

Implement confidence-based remasking and compare its trajectory with irreversible token commitment.

Run a provided Colab training loop on a small character-level corpus and diagnose learning from its loss curve and periodic generated samples.

Complete a provided blockwise semi-autoregressive decoding scaffold that denoises positions in parallel within each block and freezes completed blocks.

Measure the number of model evaluations and wall-clock generation time for full-sequence and blockwise decoding.

Explain the mismatch between independently corrupted training examples and iterative inference trajectories, including why remasking can help.

Explain why masked diffusion language models lack the usual left-to-right likelihood factorization and how masked denoising loss, including optional schedule weighting, serves as a training surrogate.

Run a controlled sweep over the number of denoising steps and explain the resulting sample-quality versus computation trade-off.

Translate the clean sequence, corrupted state, noise level, denoiser prediction, and reverse-update notation in Section 3.1 of the linked paper into corresponding tensors and operations in the Colab implementation.