Engineering for Deep Learning & AI

Define occupancy and calculate it based on a kernel's register usage, shared memory usage, and block size.

Identify and resolve uncoalesced global memory accesses to maximize memory throughput.

Apply shared memory tiling to reduce redundant global memory loads in a matrix multiplication kernel.

Detect and mitigate shared memory bank conflicts using padding or data layout swizzling.

Use loop unrolling to expose instruction-level parallelism and improve warp throughput.

Implement register tiling within a compute-intensive loop to improve arithmetic intensity.

Use Nsight Compute to analyze SM occupancy, memory throughput, and warp-stall reasons, classifying a kernel as latency-, bandwidth-, or compute-bound.

Implement a warp-level reduction using warp-level primitives (e.g., shuffle instructions).

Implement a block-level parallel reduction kernel and analyze its bandwidth efficiency.

Implement a naive matrix multiplication kernel for GPT-2's QKV projection shapes and measure its FLOPs/s against the roofline limit.

Optimize the matmul kernel with shared-memory tiling and measure the reduction in global memory traffic.

Refine the tiled matmul with register blocking and loop unrolling to approach a significant fraction of peak FLOPs.

Describe the Tensor Core programming model and warp-level matrix fragments using WMMA/MMA intrinsics.

Implement a half-precision tiled WMMA matrix multiplication kernel and benchmark it against cuBLAS.

Implement a two-pass fused softmax kernel (as in GPT-2 attention) that ensures numerical stability.

Implement a fused BatchNorm+ReLU kernel for a ResNet-50 residual block, measuring the speedup from fusion.

Explain Triton's programming model, contrasting its block-level abstractions and autotuning capabilities with CUDA C++.

Write a Triton kernel for an elementwise op (e.g., GELU) using block-level pointer arithmetic and masking.

Implement a tiled matrix multiplication in Triton and use `@triton.autotune` to find optimal block sizes for GPT-2's hidden dimension.

Implement a fused softmax kernel in Triton and replace the PyTorch native softmax inside GPT-2's attention module.

Describe the online softmax algorithm and explain how FlashAttention's tiling reduces HBM traffic, changing memory complexity from O(N^2) to O(N).

Implement the outer tiling loop of a FlashAttention forward pass in Triton, loading blocks of Q, K, V.

Implement the online softmax update logic inside the FlashAttention inner loop, maintaining running statistics for numerical stability.

Integrate the FlashAttention Triton kernel into GPT-2 small and measure the end-to-end forward-pass speedup.

Trace a tensor operation from Python through the PyTorch dispatcher to a backend kernel.

Explain the roles of ATen, c10, and the device/dtype dispatch mechanism in PyTorch.

Implement a custom `torch.autograd.Function` with `forward` and `backward` methods and verify gradients using `gradcheck`.

Bind a custom CUDA kernel to PyTorch using the modern `torch.library` custom op API.

Explain how `torch.compile` lowers a model through Dynamo, AOTAutograd, and Inductor.

Use the PyTorch profiler with Kineto to decompose a training step into Python overhead, CUDA API calls, kernels, and data transfers.

Apply `torch.compile` to the case study models, inspect the generated Triton kernels, and quantify the speedup.

Diagnose CUDA out-of-memory errors using `torch.cuda.memory_snapshot` and associated tooling.

Choose between bfloat16 and fp16 for mixed-precision training based on hardware support and numerical stability considerations.

Apply automatic mixed-precision (AMP) with `autocast` and `GradScaler` to both models and measure the throughput and accuracy impact.

Estimate the theoretical training-step memory footprint of GPT-2 small (weights, gradients, optimizer states, activations).

Measure the peak training-step memory of GPT-2 small using PyTorch memory profiling tools and reconcile it with the theoretical estimate.

Apply activation checkpointing to GPT-2 small to trade recomputation for reduced memory usage.

Replace a standard optimizer with a fused optimizer (e.g., FusedAdamW) and measure the reduction in kernel launch overhead.

Tune the data loading pipeline (e.g., `num_workers`, `pin_memory`, `prefetch_factor`) to eliminate input bottlenecks.

Apply CUDA Graphs to the training loops and measure the speedup from reduced CPU launch overhead.

Compare interconnect technologies (PCIe, NVLink, InfiniBand/RoCE) and their impact on multi-GPU and multi-node training.

Explain the concepts of RDMA and GPUDirect RDMA and their importance for low-latency communication.

Define the semantics of core collective communication primitives: broadcast, reduce, all-reduce, all-gather, reduce-scatter, and all-to-all.

Derive the bandwidth and latency costs of ring all-reduce and tree all-reduce algorithms for N workers.

Use the NCCL backend in PyTorch to perform an all-reduce and benchmark the achieved bus bandwidth against the theoretical maximum.

Inspect NCCL's chosen algorithms and communication channels using `NCCL_DEBUG` and topology files.

Explain how network topology (e.g., fat-tree, rail-optimized) influences collective performance at scale.

Diagnose communication bottlenecks and stragglers in a distributed job using timeline traces from a profiler.

Explain the synchronous data parallelism algorithm and the role of gradient all-reduction.

Configure and run PyTorch DistributedDataParallel (DDP) for both ResNet-50 and GPT-2 small on multiple GPUs.

Measure the scaling efficiency of DDP relative to the single-GPU baseline.

Enable gradient bucketing in DDP to overlap backward pass computation with gradient communication and measure the speedup.

Explain the three stages of the ZeRO optimizer and their respective memory and communication trade-offs.

Wrap the GPT-2 small model in PyTorch Fully Sharded Data Parallel (FSDP) with appropriate auto-wrap policies.

Experiment with FSDP sharding strategies (e.g., `FULL_SHARD` vs. `SHARD_GRAD_OP`) and measure the impact on memory and throughput.

Benchmark the throughput and memory usage of DDP vs. FSDP for GPT-2 small, explaining the observed performance differences.

Derive the Megatron-style tensor parallelism algorithm for a transformer MLP block, identifying the required collective operations.

Derive the tensor parallelism algorithm for a transformer attention block, noting the different communication patterns.

Implement a basic tensor-parallel linear layer using `torch.distributed` primitives and verify its correctness.

Integrate the tensor-parallel layer into the GPT-2 small model and benchmark its memory and performance on a single node.

Explain how sequence parallelism reduces activation memory by splitting tensors along the sequence dimension.

Diagram how tensor parallelism and sequence parallelism can be combined within a transformer layer to further reduce activation memory.

Analyze the communication costs per layer for tensor parallelism and sequence parallelism in contrast to data parallelism.

Explain naive pipeline parallelism and quantify the 'pipeline bubble' for a given number of stages and microbatches.

Describe the GPipe and 1F1B (PipeDream-style) scheduling algorithms and compare their memory and bubble characteristics.

Describe interleaved 1F1B schedules and how they can achieve near-zero bubble.

Calculate the per-layer parameter, activation, and FLOPs costs for a scaled-up GPT-style model.

Partition the layers of the scaled-up model into balanced pipeline stages, considering both FLOPs and peak activation memory.

Combine data, tensor, and pipeline parallelism (3D Parallelism) to define a configuration for a multi-billion-parameter model.

Compute the per-GPU memory and communication budget for a 3D-parallel configuration.

Diagnose performance regimes (bubble-bound, communication-bound, compute-bound) in a pipeline run from profiler traces.

Compare synchronous, asynchronous, and hierarchical distributed checkpointing strategies for sharded models.

Implement fault-tolerant training using `torch.distributed.elastic` with automated restart from checkpoints.

Describe strategies for detecting and mitigating silent data corruption (bit flips) in long training runs.

Describe methods for detecting and mitigating the impact of slow or faulty hardware (stragglers) in a distributed job.

Plan a multi-node training job on a Slurm-based cluster, considering topology-aware process placement and CPU/GPU affinity.

Calculate Model FLOPs Utilization (MFU) and Hardware FLOPs Utilization (HFU) for a training run and explain the gap.

Design an observability stack for distributed training, combining loss dashboards, per-rank hardware metrics, and communication traces.

Integrate optimizations from the course into final training scripts for the case studies and produce a report attributing the cumulative speedup to each technique.