GPU Performance: Kernel Launches, Divergence, and Occupancy

Introduction

Welcome back. In our previous lesson, we established the foundational concepts of the CUDA programming model: the hierarchy of grids, blocks, and threads, and their direct mapping to the GPU's hardware architecture of devices, Streaming Multiprocessors (SMs), and warps. You now understand what these abstractions are and where they execute.

Today, we build on that foundation to explore the performance implications of this model. This lesson addresses the critical "how" and "why" of GPU performance. We will examine how the decisions you make when launching a kernel—specifically, the number and arrangement of threads—directly influence the hardware's efficiency.

In line with your learning outcome, this lesson will explain the significance of kernel launches, thread divergence, and occupancy in GPU program performance. These three concepts are deeply intertwined and form the basis for reasoning about, and ultimately optimizing, any GPU-accelerated code.

1. Kernel Launches and the Art of Latency Hiding

In the previous lesson, we saw that a kernel launch, using the <<<...>>> syntax, is how the host CPU dispatches work to the GPU. This launch configuration—the number of blocks in the grid and threads in each block—is the primary lever you have to control the amount of parallelism. The first and most important goal of a kernel launch is simple: give the GPU enough work to stay busy.

Why is this so crucial? Because GPU performance is built on a principle called latency hiding. Accessing data from global memory (DRAM) is slow, taking hundreds of clock cycles. Even floating-point arithmetic has a latency of several cycles. If a single thread (or even a single warp) had to wait for each of these operations to complete, the GPU would spend most of its time idle.

The GPU's solution is to oversubscribe the hardware with a massive number of threads, grouped into warps. When one warp stalls (e.g., waiting for a memory read), the SM's warp scheduler simply switches to another ready warp and executes its instructions. By rapidly context-switching between a large pool of available warps, the scheduler can "hide" the latency of memory and arithmetic operations, ensuring that the SM's execution units are almost always doing useful work.

To understand this mechanism in detail, we'll watch a key segment from a CUDA optimization tutorial.

The key takeaway is that your kernel launch must create enough total threads—and therefore warps—to provide the SM schedulers with sufficient work to hide latency. As a rule of thumb, you should aim to launch many more blocks than you have SMs and configure your blocks with a healthy number of threads.

2. Thread Divergence: The Penalty for Branching

While latency hiding is about keeping the hardware busy, thread divergence is about using that hardware efficiently. As we learned, the SM executes threads in warps of 32, and all threads in a warp execute the same instruction at the same time (the SIMT model).

Thread divergence occurs when threads within the same warp follow different execution paths due to conditional logic (e.g., an if-else statement). Since the hardware can only issue one instruction stream per warp, it handles this by serializing the different paths:

1. Threads taking the if path execute their instructions while the threads taking the else path are disabled (masked off).
2. Then, threads that took the else path execute their instructions while the if-path threads are disabled.

The total time taken is the sum of the time for both branches. For that portion of the code, you are effectively using only a fraction of the warp's 32 lanes, reducing your computational throughput.

To solidify this concept, let's read a concise explanation.

The significance of divergence is that it can silently cripple performance. A key aspect of designing efficient kernels is to structure your code and data access to minimize data-dependent branching within a warp. When possible, you want all 32 threads in a warp to agree on which path to take.

3. Occupancy: A Measure of Latency Hiding Potential

We've established that we need a large pool of active warps on each SM to hide latency. Occupancy is the metric used to quantify this. It is defined as the ratio of active warps on an SM to the maximum number of warps that SM can theoretically support.

For example, a Volta V100 SM can support a maximum of 64 active warps. If your kernel's launch configuration and resource requirements result in only 32 warps being active on each SM, the occupancy is 50%.

High occupancy is generally desirable because it gives the warp scheduler more choices to find a ready warp, increasing the probability of hiding latency effectively. However, occupancy is not simply a function of launching more threads. It is constrained by the finite resources on an SM, primarily:

• Registers: Each SM has a fixed-size register file (e.g., 65,536 32-bit registers on a V100). This pool is partitioned among all active threads on the SM. If your kernel requires many registers per thread, it will limit the total number of threads (and thus warps) that can be resident on the SM.
• Shared Memory: Each SM has a fixed amount of on-chip shared memory. Thread blocks can request a portion of this memory. If a block requests a large amount, it can limit the number of blocks that can run concurrently on that SM.
• Thread Block Limits: The hardware also has a maximum number of thread blocks and threads that can be co-resident on an SM.

The following video, while based on AMD's architecture (which uses "wavefronts" instead of "warps" and "WGP/CU" instead of "SM"), provides an excellent and architecturally-neutral explanation of how occupancy works and what limits it. The principles are identical.

As the video illustrates, achieving high occupancy is a balancing act. Your kernel launch and resource usage directly determine the theoretical maximum occupancy you can achieve.

The Nuance: Occupancy is a Means, Not an End

A common mistake is to treat maximizing occupancy as the ultimate goal of optimization. This is incorrect. High occupancy is a means to an end—the end being latency hiding.

Consider these scenarios:

• Compute-Bound Kernel: If a kernel spends all its time performing arithmetic and very little time waiting for memory, there is no latency to hide. In this case, even low occupancy can achieve peak performance. Having more warps sitting around idle doesn't help.
• Register Spilling: If you aggressively reduce register usage per thread just to increase occupancy, the compiler might be forced to "spill" variables to the much slower local memory (which resides in global DRAM). This can dramatically increase memory latency, and the performance loss from these spills often outweighs any benefit from higher occupancy.

The goal is not 100% occupancy; it's to achieve sufficient occupancy to hide the latencies present in your specific kernel.

Conclusion

In this lesson, we moved from the structure of a CUDA program to the dynamics of its performance. You've seen how the choices made at the software level have profound and direct consequences on the hardware's efficiency.

Key Takeaways:

• Kernel Launches are your primary tool for feeding the GPU's massively parallel engine. The goal is to launch enough threads (organized into blocks) to create a large pool of warps for each SM.
• The GPU achieves high performance by using this pool of warps for latency hiding, switching to ready warps whenever one stalls on a long-latency operation like a memory access.
• Thread Divergence is a significant performance hazard. When threads in a warp branch, the hardware serializes the different paths, underutilizing its parallel execution units.
• Occupancy measures the number of active warps relative to the hardware maximum. It indicates the potential for latency hiding but is limited by on-chip resources like registers and shared memory.
• High occupancy is a tool for hiding latency, not a goal in itself. The optimal level of occupancy depends on whether a kernel is limited by memory latency or computation.

Preview of the Next Lesson:

So far, our entire discussion has been within the NVIDIA CUDA ecosystem. However, GPUs from other vendors like AMD and Intel are also major players in high-performance and AI computing. In our next lesson, we will broaden our perspective to compare the goals and ecosystems of major GPU programming frameworks: CUDA, OpenCL, and ROCm/HIP, examining their similarities, differences, and strategic positions in the industry.