High-Performance AI Computing

Perform pointer arithmetic and explain the relationship between pointers and arrays in C/C++.

Explain stack vs. heap memory allocation and their performance characteristics.

Manage dynamic memory using malloc/free and new/delete, identifying common pitfalls like memory leaks and dangling pointers.

Explain memory layout for C structs and C++ classes, including data alignment and padding.

Perform bitwise operations (AND, OR, XOR, shifts) and apply them to common tasks like bit masking.

Use C++ templates to write generic functions capable of operating on different data types.

Write and compile a multi-file C++ program, managing dependencies between source files and headers.

Describe the fetch-decode-execute cycle and the role of the modern CPU pipeline.

Explain the CPU cache hierarchy (L1/L2/L3) and the concept of a cache line.

Analyze how data access patterns (e.g., spatial and temporal locality) affect cache performance.

Define instruction-level parallelism (ILP) and describe how pipelining and superscalar execution exploit it.

Explain SIMD (Single Instruction, Multiple Data) and its role in modern data-parallel computation.

State Amdahl's Law and calculate the theoretical speedup for a given parallelizable workload.

Distinguish between latency-oriented (CPU) and throughput-oriented (GPU) processor design philosophies.

Explain the GPU thread hierarchy: threads, warps, thread blocks, and grids.

Describe the GPU memory hierarchy: registers, shared memory, L1/L2 caches, and global memory (HBM).

Compare the bandwidth and latency characteristics of each level of the GPU memory hierarchy.

Explain how concurrent execution of warps on an SM is used to hide memory latency.

Define SM occupancy and explain its relationship to latency hiding and resource limitations (registers, shared memory).

Compile and execute a basic CUDA C++ program using the NVIDIA C++ compiler (nvcc).

Transfer data between host (CPU) and device (GPU) memory using `cudaMemcpy`.

Write and launch a CUDA kernel to perform a simple computation, such as element-wise vector addition.

Use CUDA's built-in variables (threadIdx, blockIdx, blockDim) to compute unique global thread indices for 1D and 2D grids.

Implement bounds checking within a kernel to safely handle input sizes not evenly divisible by the block size.

Implement robust error handling for CUDA API calls and kernel launches.

Write a kernel to demonstrate coalesced vs. uncoalesced memory access and measure the performance impact.

Declare and use shared memory within a kernel to facilitate data reuse and reduce global memory traffic.

Use `__syncthreads()` to correctly synchronize threads within a thread block.

Implement a tiled algorithm using shared memory for an operation like matrix transposition.

Explain what a shared memory bank conflict is and how to organize data access to avoid them.

Describe the roles of the read-only data cache and constant memory in optimizing data fetch patterns.

Explain how threads within a warp execute in a Single Instruction, Multiple Thread (SIMT) fashion.

Describe thread divergence caused by conditional branching and analyze its performance impact.

Refactor conditional code to minimize divergence, for example by using predicated execution.

Use atomic operations (e.g., `atomicAdd`, `atomicCAS`) to safely update memory locations from multiple threads.

Use warp-level primitives (e.g., `__shfl_down_sync`) for efficient intra-warp data exchange.

Use warp vote functions (`__any_sync`, `__all_sync`) for collective warp-level decisions.

Implement a naive parallel reduction and analyze its inefficiencies from divergence and non-coalesced access.

Optimize a parallel reduction by using sequential addressing and shared memory to improve performance.

Further optimize the reduction using warp shuffle instructions for efficient intra-warp communication.

Implement a parallel histogram kernel using atomic operations to handle contention.

Implement the up-sweep (reduction) phase of the Blelloch scan algorithm within a single thread block.

Implement the down-sweep phase to complete a single-block exclusive scan algorithm.

Implement gather and scatter parallel data movement patterns in CUDA kernels.

Define arithmetic intensity and use it to classify a kernel as compute-bound or memory-bound.

Use NVIDIA Nsight Compute to profile a kernel and identify primary performance bottlenecks.

Calculate a kernel's achieved memory bandwidth and compute throughput from profiler metrics.

Construct a roofline model for a given GPU and plot a kernel's performance to visualize its bottleneck.

Apply an iterative optimization methodology: profile, identify bottleneck, formulate a hypothesis, implement an optimization, and re-profile.

Use CUDA streams to manage asynchronous operations and overlap kernel execution with data transfers.

Use CUDA events to accurately time kernel execution and measure the effectiveness of stream overlap.

Implement a naive, thread-per-element matrix multiplication (GEMM) CUDA kernel.

Implement tiled matrix multiplication using shared memory to reduce global memory accesses.

Analyze the reduction in global memory traffic and performance improvement from shared memory tiling.

Implement register-level tiling within a shared memory tile to maximize data reuse from registers.

Use vectorized memory loads (e.g., `float4`) to increase memory bandwidth utilization.

Handle boundary conditions for matrices whose dimensions are not integer multiples of the tile size.

Benchmark each GEMM optimization stage against a library implementation (cuBLAS) and analyze remaining performance gaps.

Implement a fused element-wise kernel (e.g., bias-add and ReLU activation) and explain the performance benefits of kernel fusion.

Implement the forward pass for a Layer Normalization kernel, including parallel reduction for mean and variance.

Derive the gradients for Layer Normalization with respect to its input, scale, and bias.

Implement the backward pass for the Layer Normalization kernel in CUDA.

Explain the numerical instability of a naive softmax and implement the max-subtraction trick for stabilization.

Implement a numerically stable softmax kernel using a two-pass parallel reduction algorithm.

Explain the principles of FP16/BFloat16 mixed-precision and its impact on memory bandwidth and compute throughput.

Describe the architecture of NVIDIA Tensor Cores and the `mma.sync` instruction for matrix multiply-accumulate operations.

Use the `nvcuda::wmma` C++ API to load matrices into fragments, perform warp-level MMA, and store results.

Construct a complete GEMM kernel using `wmma` intrinsics, managing the data pipeline from global memory to shared memory and finally to registers.

Explain the quadratic memory complexity of standard self-attention that FlashAttention addresses.

Describe the core concepts of FlashAttention: tiling and online softmax computation.

Implement the inner loop of a simplified FlashAttention kernel, computing a block of the output matrix.

Combine components into a simplified FlashAttention forward pass kernel and verify its output against a naive baseline.

Write a custom CUDA kernel for a PyTorch operator using the C++ extension API.

Build and link the PyTorch extension using a `setup.py` file and call the custom kernel from Python.

Explain the high-level programming model of Triton and its domain-specific abstractions.

Implement a common AI kernel, such as element-wise addition or a simplified GEMM, in Triton.

Compare the Triton implementation with an equivalent raw CUDA kernel, analyzing the trade-offs between developer productivity and performance.

Describe the role of collective communication libraries (e.g., NCCL) and key operations like All-Reduce in multi-GPU training.

Explain the concept of peer-to-peer (P2P) memory access for enabling direct communication between multiple GPUs.