Welcome to the third lesson in our foundational module on GPU architecture. In our last session, we dissected the physical hardware of a modern GPU, mapping out its hierarchical structure from the entire chip down to the Streaming Multiprocessors (SMs) and their specialized execution units like CUDA Cores and Tensor Cores. We established what the components are and where they reside.
Today, we shift our focus from the static hardware layout to the dynamic process of execution. We will explore how a GPU actually runs the thousands of threads required for parallel computation. This lesson addresses the Single Instruction, Multiple Thread (SIMT) execution model. By the end, you'll understand the concept of a warp, the critical performance pitfall of warp divergence, and the hardware mechanism of predication used to manage it. This knowledge is fundamental to writing efficient GPU code, as it clarifies the constraints and behaviors of the underlying execution environment.
The SIMT Execution Model
At the heart of the GPU's massive parallelism lies the Single Instruction, Multiple Thread (SIMT) execution model. As the name suggests, a single instruction is issued by the hardware, but it is executed by multiple threads simultaneously. This provides a programmable and flexible abstraction over the raw hardware.
To understand SIMT, it's useful to contrast it with the more traditional Single Instruction, Multiple Data (SIMD) model common in CPU vector units. In a pure SIMD model, a single instruction operates on a vector of data. SIMT extends this by giving each processing lane the characteristics of a full-fledged thread, with its own instruction pointer and state. However, these threads are not fully independent. The hardware groups them together for execution.
This short video provides an excellent visual intuition for how SIMT builds upon SIMD to manage massive parallelism.
GPU Warps Explained: How SIMT Really Works Under the Hood (Visual Deep Dive) | M2L3
This video from the Parallel Routines channel animates the transition from simple scalar execution to SIMD and finally to the GPU's way of organizing work.
Please watch from the beginning to the end of the section on the programming model. Pay close attention to how a large problem is broken into batches of 32 elements, which then execute as a unit. This is the core concept we'll be building on.
Warps: The Fundamental Unit of Execution
As the video hinted, the GPU hardware doesn't manage individual threads one by one. Instead, it groups them into bundles of 32, known as a warp. A warp is the fundamental unit of scheduling and execution on an NVIDIA GPU. All 32 threads in a warp are dispatched and executed together by an SM's warp scheduler.
At any given moment, the warp scheduler selects a warp that is ready to run and issues a single instruction from its program. This instruction is then executed by all 32 threads in the warp in lock-step.
Tutorial: Understanding GPU Architecture and Execution Model
This tutorial from the eunomia.dev blog provides a clear and concise explanation of the warp concept.
Please read the section titled "Understanding Warps". It directly states the "one instruction at the same time" rule and introduces the idea that this has significant performance implications, which we'll explore next.
The concept of a warp is a hardware reality that sits just below the CUDA programming model's abstraction of a thread block. You define the behavior of a single thread, but the SM executes 32 of them at a time. The following video segment brilliantly visualizes this process, tracing a warp's execution through the SM's hardware pipeline.
GPU Warps Explained: How SIMT Really Works Under the Hood (Visual Deep Dive) | M2L3
This is a continuation of the previous video. It now dives into the hardware level, showing how a warp moves through the SM.
Watch from the hardware view until the end. As you watch, focus on these key moments: Lock-step Execution: Notice how all 32 threads execute the same arithmetic instruction together. Memory Stalls: Crucially, observe what happens when the warp issues a memory load (at this point). The entire warp stalls, waiting for data. Latency Hiding: The narrator explains that this stall is why GPUs need so many threads. While one warp is stalled waiting for memory, the SM's warp scheduler can switch to another ready warp and continue executing useful work, thus "hiding" the memory latency.
Warp Divergence and Predication
The lock-step execution of a warp leads to a critical question: what happens if the code contains a data-dependent conditional, like an if-else statement, where different threads in the same warp need to follow different code paths?
This situation is called warp divergence. Since the warp can only execute one instruction at a time, it cannot execute both the if and else blocks simultaneously. Instead, the hardware serializes the execution.
Here's how it works:
- The hardware first executes the
ifblock. During this time, all threads that should take theelsepath are disabled. - Once the
ifblock is complete, the hardware executes theelseblock. Now, the threads that took theifpath are disabled. - The total time taken is the sum of the execution times for both paths.
The mechanism used to disable threads is called predication. Each thread in the warp has an associated activity bit in a hardware "active mask". When an instruction is issued, only the threads whose corresponding bit in the mask is set will actually execute it and write back results. For divergent branches, the hardware manipulates this mask to ensure only threads on the correct path are active.
Tutorial: Understanding GPU Architecture and Execution Model
The same tutorial provides a great summary of this performance-critical issue.
Please read the section on Warp Divergence. The code example and explanation clearly illustrate the performance penalty.
Visualizing Divergence and Reconvergence
Let's analyze this process with a concrete example. The following image is a classic illustration of how a GPU handles a divergent branch using an active mask and a program counter (PC) stack.
Let's break down this diagram, assuming a warp of 4 threads for simplicity (in reality, it's 32). The Active Mask is a 4-bit number where each bit corresponds to a thread. 1 means active, 0 means inactive.
- Initial State (c): The warp starts at block A. All 4 threads are active (
Active Mask: 1111). The PC stack is shown with return/reconvergence points; initially, it points to the end of the program. - The Branch: At block B, a conditional branch occurs. Let's say Thread 0 evaluates the condition as false, while Threads 1, 2, and 3 evaluate it as true. This creates divergence.
- Divergence (d): The hardware must choose a path to take first. In this example, it chooses the
elsepath (block D).- To do this, it pushes information onto a hardware stack. It saves the reconvergence point (E), the next PC for the other path (C), and the active mask for that other path (
1000, for Thread 0). This is entry(iii). - It then proceeds to block D with an active mask of
0110, representing threads 2 and 3 that need to execute this path. (The diagram shows a slightly more complex scenario where the branch at B also leads to a nested branch, creating multiple stack entries, but the principle is the same).
- To do this, it pushes information onto a hardware stack. It saves the reconvergence point (E), the next PC for the other path (C), and the active mask for that other path (
- Serialization: The timeline in (b) shows this serialization. The hardware executes blocks C, D, and E for different subsets of threads. All threads must wait until the full sequence is complete.
- Reconvergence (e): The control flow graph shows that all paths (from C, D, and F) eventually meet at block G. This is the immediate post-dominator of the branch at B. This is where the hardware can guarantee all divergent paths have rejoined. After popping the saved states off the stack and executing all paths, the hardware reaches block G, and the active mask is restored to
1111. All threads are once again executing in lock-step.
This automatic handling of divergence through predication and a hardware stack is what makes SIMT more flexible than classic SIMD. However, it's not free. Warp divergence effectively reduces the number of active threads, lowering the utilization of the SM's execution units and thus harming performance. A key goal in GPU programming is to structure data and code to minimize divergence.
Conclusion
In this lesson, we've moved from the static hardware diagram to the dynamic execution model that brings it to life. Understanding these dynamics is non-negotiable for anyone serious about GPU performance engineering.
Key Takeaways:
- SIMT (Single Instruction, Multiple Thread): The GPU execution model where one instruction stream controls many threads, providing a flexible programming model for massive parallelism.
- Warp: The fundamental unit of scheduling and execution, consisting of 32 threads that execute in lock-step.
- Latency Hiding: The SM's ability to switch between warps is the primary mechanism for tolerating high memory latency and keeping the execution units busy.
- Warp Divergence: A major performance inhibitor that occurs when threads in a warp follow different paths in a conditional branch. The hardware serializes the paths, reducing effective parallelism.
- Predication: The hardware mechanism, using an active mask, that enables or disables individual threads within a warp to handle divergence.
We've seen that the entire SIMT model is architected to deal with memory latency. In our next lesson, we will confront this latency head-on by taking a detailed tour of the GPU memory hierarchy, from the ultra-fast registers to the vast but slow High-Bandwidth Memory (HBM).