GPU Thread Scheduling: Warps and Wavefronts

Introduction

In our last lesson, we established the fundamental design philosophies that separate CPUs and GPUs. We saw that CPUs are latency-oriented, designed to execute a single task as quickly as possible, while GPUs are throughput-oriented, designed to maximize the total work done. The GPU's primary strategy for achieving this is latency hiding—performing other useful work while waiting for long-latency operations like memory reads to complete.

This lesson addresses the core mechanism behind that strategy, fulfilling the learning outcome: Describe how GPU thread scheduling works using warps or wavefronts. We will dissect the role of the hardware scheduler and explore how it leverages groups of threads, called warps, to keep the GPU's many execution units constantly busy. Understanding this process is key to grasping how GPUs translate massive parallelism into high performance.

The Unit of Scheduling: Warps and Wavefronts

On a CPU, the operating system schedules individual threads. On a GPU, scheduling is handled directly by hardware at a much finer granularity. The fundamental unit of scheduling is not a single thread, but a group of threads.

In NVIDIA terminology, this group is called a warp. In AMD and other contexts, it's known as a wavefront. While their sizes can differ between architectures, the concept is the same: a fixed-size group of threads that are scheduled and executed together, in lockstep.

As the video explains, a thread block is partitioned by the hardware into these warps. For example, a block of 256 threads is divided into 8 warps (256 / 32 = 8). The first 32 threads (thread IDs 0-31) form the first warp, threads 32-63 form the second, and so on.

This concept is not exclusive to NVIDIA. Other GPU architectures use the same principle with different terminology.

For the rest of this course, we will primarily use the term "warp," but it's important to recognize that "wavefront" refers to the same underlying scheduling concept on other platforms. The key takeaway is that threads are not independent entities for scheduling; they are managed in these fixed-size groups.

The Warp Scheduler and the Latency Hiding Mechanism

Now that we've defined the warp, let's see how the hardware uses it to implement latency hiding.

Within each Streaming Multiprocessor (SM), there is a hardware unit called a warp scheduler. Its job is to select which warp to execute at any given clock cycle. An SM can have dozens of warps "resident" at once, meaning their entire execution context (program counters, registers for all 32 threads) is stored in on-chip memory.

The scheduling process works as follows:

1. The warp scheduler picks a resident warp that is ready to execute (i.e., not waiting for anything).
2. It issues the warp's next instruction to the SM's execution units (the CUDA cores/ALUs). Since execution is "warp-wide," all 32 threads in the warp execute this same instruction in lockstep.
3. If this instruction is a long-latency operation, such as reading from global memory, the warp becomes stalled (or "waiting"). It cannot proceed until the data returns.
4. The warp scheduler doesn't wait. It marks the current warp as stalled and, on the very next clock cycle, picks another ready warp from its pool of resident warps and issues an instruction for it.

This rapid, zero-cost context switching is the engine of latency hiding. The hundreds of clock cycles that a warp spends waiting for memory are filled with useful computations from other warps.

The following video provides a detailed, clock-by-clock visualization of this process. It's the most critical resource for understanding this mechanism.

The key reason this is possible is that the "context switch" between warps is essentially free. Unlike a CPU, which must save a thread's state to main memory, a GPU's SM keeps the full context for all resident warps on-chip. The scheduler simply has to change a pointer to the next ready warp.

Consequences of Warp-Based Scheduling

This elegant scheduling model has profound implications for performance. Two are particularly important: control divergence and occupancy.

1. Control Divergence

Because all threads in a warp execute the same instruction in lockstep (the 'SI' in SIMT), what happens when the code contains a branch, like an if-else statement, and threads in the same warp need to take different paths?

This situation is called control divergence. The hardware handles it by serializing the execution paths.

1. All threads in the warp travel down the if path. Threads for which the condition was false are masked off and do nothing.
2. Next, all threads travel down the else path. This time, threads for which the condition was true are masked off.
3. The paths reconverge, and all threads become active again.

The result is that the total execution time is the sum of both paths, and for the duration of the divergence, some execution units are idle. This can significantly reduce performance.

Minimizing control divergence is a critical aspect of GPU performance tuning. Ideally, threads within a warp should follow the same execution path. Divergence between different warps is not a problem, as they are scheduled independently.

2. Occupancy

The latency-hiding mechanism relies on the warp scheduler having a pool of ready warps to choose from. If all resident warps are stalled waiting for memory, the SM's execution units will go idle, and throughput will drop.

Therefore, it's crucial to have enough active warps on the SM to hide the instruction latencies. This concept is measured by occupancy.

Occupancy is the ratio of active warps on an SM to the maximum number of warps that the SM can support. For example, if an SM can support a maximum of 64 warps and your kernel is running with 32 active warps on it, the occupancy is 50%.

A higher occupancy generally gives the scheduler more flexibility to find a ready warp, leading to better latency hiding and higher performance.

As the video explains, occupancy can be limited by various factors, including the number of registers or the amount of shared memory a thread block requires. We will explore these resource limitations in more detail in later lessons. For now, the key idea is that the number of active warps is the fuel for the latency-hiding engine.

Conclusion

In this lesson, we unpacked the mechanics of GPU thread scheduling. We saw that it is a hardware-driven process that operates on groups of threads called warps or wavefronts. This model is the cornerstone of the GPU's throughput-oriented design.

Key Takeaways:

• GPUs schedule threads in fixed-size groups: warps (typically 32 threads on NVIDIA) or wavefronts (typically 64 on AMD).
• The warp scheduler, a hardware unit on the SM, issues one instruction at a time for an entire warp to execute in lockstep.
• The scheduler implements latency hiding by performing a zero-cost context switch to another ready warp whenever the current warp stalls (e.g., on a memory access).
• This lockstep execution model creates a performance hazard called control divergence, where threads in a warp taking different branches must execute them serially.
• Occupancy—the ratio of active warps to the maximum possible on an SM—is a key metric. Higher occupancy provides the scheduler with more warps to hide latency effectively.

Preview of the Next Lesson:

We have now discussed the SM's role as the GPU's core computational engine and seen how its warp scheduler orchestrates thread execution. In the next lesson, we will zoom in further to identify the key components of a modern GPU, such as Streaming Multiprocessors (SMs), and their roles. We'll build a more complete picture of the SM, looking at its CUDA cores, register file, caches, and other functional units, providing a more detailed hardware context for the scheduling and execution models we've covered.