Core AI Kernels: Fusion, Normalization, and Backpropagation

Implement a fused element-wise kernel (e.g., bias-add and ReLU activation) and explain the performance benefits of kernel fusion.

Implement the forward pass for a Layer Normalization kernel, including parallel reduction for mean and variance.

Derive the gradients for Layer Normalization with respect to its input, scale, and bias.

Implement the backward pass for the Layer Normalization kernel in CUDA.

Explain the numerical instability of a naive softmax and implement the max-subtraction trick for stabilization.

Implement a numerically stable softmax kernel using a two-pass parallel reduction algorithm.

Explain the principles of FP16/BFloat16 mixed-precision and its impact on memory bandwidth and compute throughput.