Math for Deep Learning

State the Markov property and frame the DDPM forward process as a discrete-time Markov chain.

Characterize Brownian motion via its independent, normally distributed increments and scaling properties.

Interpret the drift and diffusion coefficients of an Itô stochastic differential equation (SDE).

Analyze the Ornstein-Uhlenbeck process as a continuous-time mean-reverting stochastic process.

Define the score function (gradient of the log-probability density) and explain its role in reversing a diffusion process.

Derive the DDPM forward noising process as a discretization of a variance-preserving SDE.

Derive scaled dot-product attention from first principles and justify the 1/√dₖ scaling factor.

Implement multi-head attention in PyTorch using efficient batched matrix multiplications.

Implement sinusoidal positional encodings and analyze their properties for representing sequence order.

Assemble a complete Transformer encoder block, including residual connections and layer normalization.

Implement causal masking for a decoder's self-attention module to enforce autoregressive generation.

Assemble a complete Transformer decoder block, integrating self-attention, cross-attention, and the feed-forward network.

Train a small Transformer on a toy sequence-to-sequence task (e.g., reversal) and validate its convergence.

Implement a key-value cache to optimize inference speed and memory usage during autoregressive decoding.

Implement a VAE's encoder and decoder networks in PyTorch, including the reparameterization trick for the latent space.

Implement the ELBO loss function, combining a reconstruction term (e.g., binary cross-entropy) with the analytical KL divergence for Gaussian distributions.

Train a VAE on MNIST and diagnose potential posterior collapse by monitoring the KL divergence or active units.

Implement KL cost annealing to schedule the weight of the KL term, improving reconstruction before enforcing a strict prior.

Generate new images by sampling from the prior and decoding, and visualize the latent space manifold.

Evaluate model performance by comparing ELBO values across different architectural choices and interpreting the results in bits/dim.

Derive the change of variables formula for probability densities and identify the log-determinant-Jacobian term as the objective for training flows.

Implement an affine coupling layer in PyTorch, using a neural network to predict scaling and translation parameters from un-transformed inputs.

Stack coupling layers with alternating masks to ensure the entire data vector can be transformed, and verify invertibility.

Implement the multi-scale architecture using squeeze and factor-out operations to create a hierarchy of latent variables.

Train RealNVP on a 2D toy dataset by maximizing the exact log-likelihood and visualize the learned density transformation.

Generate high-fidelity samples by passing noise from the base distribution through the inverted flow.

Derive the closed-form expression for the forward process posterior q(xₜ | x₀), enabling single-step noising.

Derive the simplified noise-prediction training objective from the full variational bound.

Implement the forward noising process and variance schedules (e.g., linear, cosine) in PyTorch.

Implement a U-Net architecture with attention and sinusoidal time embeddings to serve as the noise predictor.

Train the DDPM on a small image dataset by minimizing the noise prediction error.

Implement the ancestral sampling loop (DDPM algorithm) to generate images by iteratively denoising from pure noise.

Implement the DDIM deterministic sampling method and compare its sample quality versus computational cost against the ancestral sampler.