Diffusion Models for Beginners

Explain the roles of the forward corruption process and the learned reverse denoising process in a diffusion language model.

Distinguish Gaussian diffusion for continuous data from mask-based diffusion for discrete text.

Calculate the probability that a token is masked at a given noise level under a simple masking schedule.

Trace a complete forward masking trajectory for a toy token sequence by hand.

Represent token sequences, padding, and a special mask token as PyTorch tensors in Colab.

Implement a PyTorch function that samples noise levels and corrupts a batch of token sequences.

Explain why a masked text denoiser uses bidirectional attention and receives the current noise level as input.

Construct token, position, and noise-level embeddings for a batch of corrupted sequences in PyTorch.

Configure a small PyTorch Transformer encoder with the correct padding mask for denoising.

Project the denoiser's hidden states into vocabulary logits at every sequence position.

Compute cross-entropy loss only at corrupted token positions.

Implement one optimizer step for the masked denoising objective.

Select which masked positions to reveal according to a denoising schedule.

Compose reverse-denoising updates into a complete iterative text sampler.

Train the tiny masked diffusion language model on a small text corpus in Colab and inspect its generated samples.

Implement blockwise semi-autoregressive decoding that denoises several tokens in parallel within each block.

Explain why masked diffusion language models lack the usual autoregressive sequence likelihood and how the denoising loss acts as a training surrogate.

Translate the diffusion-language-model notation and equations in Section 3.1 of the linked paper into operations in the Colab implementation.

Run a controlled experiment varying denoising steps, reveal schedule, and temperature, then explain the observed quality–speed trade-offs.