Foundations: GPU Architecture and Performance Metrics

Contrast CPU and GPU design philosophies in terms of latency vs throughput, control logic, and arithmetic intensity.

Describe the hierarchical structure of a modern NVIDIA GPU: Streaming Multiprocessors (SMs), warp schedulers, CUDA cores, and Tensor Cores.

Explain the Single Instruction, Multiple Thread (SIMT) execution model, including warps, warp divergence, and predication.

Diagram the GPU memory hierarchy (registers, shared memory/L1, L2, HBM) with their typical bandwidth and latency characteristics.

Compute peak FP32/FP16/BF16 FLOPs and HBM bandwidth for a target GPU (e.g., A100/H100) to establish a hardware performance budget.

Estimate the per-layer FLOPs and parameter counts for the GPT-2 small and ResNet-50 case studies.

Use the roofline model to predict whether key layers in GPT-2 (attention, MLP) and ResNet-50 (convolutions) are compute- or memory-bound.

Compute effective host-device bandwidth over PCIe and estimate the transfer cost of model parameters for a single training step.