Warp Execution: Understanding SIMT

The Warp: The Fundamental Unit of GPU Execution

You'll recall from our early look at GPU architecture that threads are organized into a hierarchy of grids, blocks, and threads. While this is the programming model, the hardware introduces another crucial grouping: the warp.

A warp is a group of 32 threads that the GPU's warp scheduler treats as a single, indivisible unit for execution. When a thread block is scheduled onto a Streaming Multiprocessor (SM), it is partitioned into these 32-thread warps. For instance, a block of 256 threads will be divided into 8 warps.

The NVIDIA documentation provides the official definition.

The key takeaway is that you don't program warps directly; you program threads. However, the hardware executes them in warps. A deep understanding of this execution model is what separates a novice from an expert CUDA programmer.

SIMT: Single Instruction, Multiple Thread

The execution model for a warp is called SIMT (Single Instruction, Multiple Thread). This concept is at the heart of the GPU's design philosophy.

At any given moment, all 32 threads within a warp execute the exact same instruction in lockstep. The differentiation comes from the data they operate on; each thread has its own private set of registers and can therefore work on unique data.

Imagine a simple kernel instruction like c[i] = a[i] + b[i]. In a warp executing this code:

• All 32 threads fetch the instruction load a[i].
• All 32 threads fetch the instruction load b[i].
• All 32 threads execute the add instruction.
• All 32 threads execute the store c[i] instruction.

Each thread i will have a different value for a[i] and b[i], but the sequence of operations is identical and synchronized across the entire warp.

The image below provides a great mental model for this lockstep execution.

This model allows for tremendous hardware efficiency. Instead of needing 32 separate, complex instruction fetch and decode units (like a 32-core CPU), the SM only needs one that can serve the entire warp. This is a primary reason why GPUs can pack thousands of "cores" onto a single chip.

The following reading provides an excellent breakdown of SIMT and contrasts it with the traditional CPU model.

Distinguishing SIMT from SIMD

Your computer science background has likely introduced you to SIMD (Single Instruction, Multiple Data), a feature of modern CPUs that uses special vector registers and instructions (like AVX or SSE) to perform one operation on a small vector of data. While the names are similar, the distinction between SIMD and SIMT is critical for a systems programmer.

• SIMD: A single thread executes a special instruction that operates on a vector data type. The parallelism is explicit in the data type. The programmer manages vector registers directly.
• SIMT: Multiple threads (e.g., 32) execute a standard scalar instruction. The parallelism is implicit in the threading model. Each thread has its own state (program counter, registers) and thinks it's running independently.

The following video and diagram provide an excellent clarification of this difference.

The diagram below visualizes this contrast. On the top, a CPU's SIMD unit applies one operation (op) to multiple data elements (X0, Y0, etc.) within wide registers. On the bottom, a GPU's SIMT architecture has an instruction decoder feeding the same instruction to multiple independent CUDA cores, each with its own registers and state.

SIMT gives you the programming ease of writing simple, scalar thread code, while the hardware translates it into highly efficient, wide vector execution.

The "Why" of SIMT: Latency Hiding and Memory Coalescing

The SIMT model isn't just for hardware simplification; it is the enabler of two of the most important GPU performance principles.

1. Latency Hiding: As we've discussed, global memory access has high latency. If a warp issues a load instruction, it may have to wait hundreds of cycles for the data to arrive. Instead of stalling, the SM's warp scheduler can instantly switch to another warp that is ready to execute. Because the state for an entire warp is kept on-chip, this context switch is nearly free. The SM can keep dozens of warps in flight, ensuring its execution units are always busy.

2. Memory Coalescing: The SIMT model is what makes coalesced memory access possible. Since all 32 threads in a warp execute a load or store instruction at the exact same time, the memory controller can look at all 32 requested addresses. If they fall within a single, aligned memory segment (e.g., 128 bytes), it can satisfy all 32 requests with a single transaction. This is a massive bandwidth multiplier that would be impossible if threads were executing asynchronously.

The following video segment revisits our memory discussion, but this time through the lens of the SIMT model, connecting warp execution directly to achieving peak memory bandwidth.

The Programming Model vs. Hardware Reality

The programming model promises that all threads in a warp execute in lockstep. For older GPU architectures, this was a strict hardware reality. This strictness, however, could lead to a subtle but deadly problem called SIMT deadlock in certain synchronization scenarios.

Imagine a thread in a warp acquires a lock and then waits for other threads to catch up at a reconvergence point. If the thread that needs to release the lock is in the same warp but on a different code path, the first thread might wait forever for a lock that can never be released, because the hardware is waiting to reconverge the warp before proceeding.

To solve this, modern GPUs (Volta architecture and newer) introduced Independent Thread Scheduling. The hardware still issues instructions to warps, but it has more flexibility to allow threads within a warp to progress independently. This breaks the strict lockstep model to prevent deadlocks and offers other scheduling benefits.

The following video provides a deep dive into this evolution, which will be fascinating given your interest in systems-level details.

As a CUDA developer, you should generally adhere to the simpler, stricter SIMT programming model. However, knowing that the underlying hardware is more sophisticated provides a more accurate mental model and explains why NVIDIA now provides explicit warp-synchronization functions, which we will cover later.

Conclusion

This lesson has established the fundamental execution model of the GPU. We've moved from the abstract hierarchy of threads and blocks to the concrete hardware reality of the warp and SIMT execution.

Key Takeaways:

• The fundamental unit of execution on a GPU is the warp, a group of 32 threads.
• Warps execute in a Single Instruction, Multiple Thread (SIMT) fashion, where all 32 threads execute the same instruction in lockstep, but on their own private data.
• SIMT enables immense hardware efficiency by sharing instruction fetch/decode logic and is the key to performance features like latency hiding and memory coalescing.
• SIMT is distinct from CPU SIMD. SIMT uses multiple independent threads executing scalar code, while SIMD uses a single thread executing explicit vector instructions.
• While the programming model assumes strict lockstep, modern GPU hardware implements a more flexible form of independent thread scheduling to prevent deadlocks and improve efficiency.

We have focused on the "happy path" where all threads in a warp follow the same code. In our next lesson, we will explore what happens when they don't. We'll investigate thread divergence—the performance penalty incurred when if-else statements cause threads within a warp to take different paths—and learn strategies to mitigate its impact.