Diffusion Models for ML Engineers

Derive the closed-form token transition kernel for a uniform-replacement continuous-time Markov chain and compute its marginal distribution at a selected noise time.

Implement vectorized multinomial token corruption in PyTorch and validate its empirical frequencies against the derived marginals.

Derive a categorical reverse transition by combining the tractable posterior q(x_s | x_t, x_0) with a model prediction of the clean-token distribution p_theta(x_0 | x_t).

Distinguish the exact reverse-process likelihood objective from the sampled-time clean-token cross-entropy objective used to train a clean-token predictor.

Train a small PyTorch denoiser on synthetic token sequences and diagnose its clean-token accuracy at different noise levels.

Implement an iterative reverse-time sampler that starts from the terminal uniform distribution and uses the trained clean-token predictor with the derived reverse transition.

Construct and test an attention mask that combines causal attention over prompt and history tokens with bidirectional attention inside the active denoising canvas.

Trace one DiffusionGemma response through prompt encoding, KV-cache reuse, self-conditioning, canvas denoising, and block-autoregressive continuation.

Extend the baseline sampler with DiffusionGemma’s entropy-ranked acceptance, uniform re-noising, temperature schedule, and adaptive stopping rules, then compare its behavior with fixed-step sampling.

Differentiate the roles of supervised denoising fine-tuning, sampler distillation, and reinforcement learning in converting an autoregressive Gemma checkpoint into a useful diffusion model.

Given an inference trace, calculate neural-network evaluations, accepted tokens per forward pass, and the break-even conditions under which diffusion decoding can outperform autoregressive decoding.