GPU Hardware and Systems Expertise

Describe the fundamental architectural differences between CPUs and GPUs in terms of transistor allocation (control, cache, ALUs).

Explain the SIMD (Single Instruction, Multiple Data) and SIMT (Single Instruction, Multiple Threads) execution models.

Compare latency-oriented (CPU) vs. throughput-oriented (GPU) design philosophies.

Describe how GPU thread scheduling works using warps or wavefronts.

Identify the key components of a modern GPU, such as Streaming Multiprocessors (SMs), and their roles.

Explain why the parallelism and arithmetic intensity of GPUs are well-suited for deep learning workloads.

Describe the role of specialized hardware like Tensor Cores in accelerating mixed-precision matrix operations.

Diagram the GPU memory hierarchy, including registers, shared memory, L1/L2 caches, and global memory (HBM/GDDR).

Explain the concept of memory coalescing and its impact on effective bandwidth.

Analyze the relationship between arithmetic intensity and memory bandwidth using the Roofline model.

Explain how memory bandwidth can become a bottleneck in data-intensive ML workloads.

Compare High Bandwidth Memory (HBM) and GDDR memory technologies in terms of their performance, power, and cost tradeoffs.

Describe the core concepts of the CUDA programming model: grids, blocks, and threads, and their mapping to the hardware.

Explain the significance of kernel launches, thread divergence, and occupancy in GPU program performance.

Compare the goals and ecosystems of major GPU programming frameworks: CUDA, OpenCL, and ROCm/HIP.

Explain how to use shared memory and synchronization primitives to facilitate efficient on-chip data sharing.

Describe the parallel reduction primitive and a common GPU implementation pattern.

Describe the parallel scan (prefix sum) primitive and its application in data-parallel algorithms.

Analyze how thread block size and grid dimensions can be tuned to maximize occupancy and performance.

Explain how matrix multiplication is parallelized on GPUs using tiling and shared memory.

Describe how convolution operations are commonly mapped to matrix multiplication (im2col/implicit GEMM) for GPU efficiency.

Analyze the computational and memory access patterns of the attention mechanism in Transformers.

Describe how batching strategies affect GPU utilization and throughput in both training and inference.

Explain the role of operator fusion in reducing memory traffic and improving kernel performance.

Explain how mixed-precision training (FP16/BF16) with dynamic loss scaling reduces memory footprint and leverages Tensor Cores.

Compare intra-node interconnects like PCIe and NVLink/NVSwitch in terms of bandwidth, latency, and topology.

Explain how inter-node technologies like InfiniBand and RDMA enable low-latency communication in a cluster.

Describe the role of GPU-Direct technologies in enabling data transfers that bypass the CPU.

Explain the role of collective communication libraries (e.g., NCCL) in abstracting hardware communication details.

Describe the AllReduce collective operation and its function in synchronizing gradients for data-parallel training.

Analyze the Ring-AllReduce and Tree-based algorithms in terms of their bandwidth usage, latency, and message size sensitivity.

Describe data parallelism, including data partitioning, gradient synchronization, and its implementation in frameworks like PyTorch DDP.

Explain tensor parallelism (intra-layer model parallelism) and how it partitions individual model layers across GPUs.

Describe pipeline parallelism (inter-layer model parallelism) and how it partitions a model across a pipeline of devices.

Analyze the 'bubble overhead' in pipeline parallelism and techniques to mitigate it, such as micro-batching and interleaved schedules.

Compare data, tensor, and pipeline parallelism in terms of their communication patterns, memory usage, and ideal use cases.

Describe how hybrid parallelism strategies (e.g., 3D parallelism) combine multiple approaches for training very large models.

Explain synchronous vs. asynchronous SGD paradigms and their tradeoffs in terms of convergence and hardware utilization.

Analyze the impact of large-batch training on convergence and the corresponding need for learning rate scaling and adaptive optimizers (e.g., LARS/LAMB).

Describe techniques for mitigating communication overhead, such as gradient accumulation and gradient compression.

Explain the concept of communication-computation overlap and its implementation via CUDA streams and asynchronous collectives.

Describe the parameter server architecture and contrast it with decentralized (AllReduce-based) approaches.

Explain how the ZeRO (Zero Redundancy Optimizer) technique partitions optimizer states, gradients, and parameters to reduce memory per device.

Compare the different stages of ZeRO and their corresponding reductions in memory consumption.

Describe how memory sharding is implemented in frameworks like DeepSpeed (ZeRO) and PyTorch FSDP.

Explain activation checkpointing (or gradient checkpointing) as a technique to trade re-computation for memory savings.

Describe the architecture of a large-scale ML training cluster, including its compute, storage, and networking tiers.

Explain how job schedulers (e.g., Slurm, Kubernetes) manage resource allocation, gang scheduling, and priorities in a multi-tenant GPU cluster.

Explain the role of containerization (e.g., Docker, Apptainer) in managing dependencies and ensuring reproducibility in GPU environments.

Analyze data loading bottlenecks and compare the suitability of parallel file systems vs. object stores for ML workloads.

Describe techniques for mitigating data pipeline bottlenecks, such as prefetching, caching, and GPUDirect Storage.

Explain strategies for fault tolerance in long-running training jobs, such as distributed checkpointing and restart.

Describe the key power and cooling considerations in designing and operating large-scale GPU deployments.

Define key performance metrics for distributed training, including throughput (e.g., TFLOPs/s), hardware utilization (MFU/HFU), and scaling efficiency.

Explain how to use profiling tools to identify if a workload is compute-bound, memory-bound, or communication-bound.

Describe the capabilities of system-level profilers (e.g., Nsight Systems) vs. framework-level profilers (e.g., PyTorch Profiler).

Analyze a profiler trace to identify performance limiters like kernel execution time, data movement overhead, and idle time.

Explain how to reason about strong vs. weak scaling to evaluate the efficiency of a distributed system.

Describe how topology-aware task placement can optimize collective communication performance.

Contrast the system requirements and optimization goals of training vs. inference workloads (latency, throughput, cost).

Describe inference batching strategies (static, dynamic, continuous) and their impact on latency and throughput.

Explain model quantization (e.g., INT8/INT4) and its effect on performance, memory usage, and accuracy.

Describe the KV-cache mechanism in Transformer inference and its memory implications for long-context models.

Explain how specialized serving libraries (e.g., vLLM, TGI) optimize memory management for LLM inference using techniques like PagedAttention.

Describe the role of a dedicated inference server (e.g., Triton) in managing models, requests, and hardware resources.

Describe how systolic array-based accelerators (e.g., Google TPUs) differ from GPUs in their architecture and programming model.

Explain the chiplet-based approach to processor design and its implications for cost, yield, and specialization.

Describe emerging interconnect standards (e.g., CXL, UCIe) and their potential to create more heterogeneous and composable systems.

Analyze the memory capacity and bandwidth challenges ('the memory wall') when scaling to trillion-parameter models.

Analyze the interconnect and communication bottlenecks that arise when training models at extreme scale.

Describe the power, cooling, and sustainability challenges of operating exascale AI hardware.