SIMD vs. SIMT Execution Models

Introduction

Welcome to your second lesson. In our previous discussion, we established the fundamental architectural distinction between CPUs and GPUs based on their transistor allocation. We saw that CPUs dedicate their resources to large caches and complex control logic for low-latency execution of a few threads, while GPUs prioritize a massive number of ALUs for high-throughput parallel computation.

This lesson addresses the logical next question: How does a GPU efficiently manage and issue instructions to thousands of ALUs at once? The answer lies in specialized execution models. Today, you will learn to explain the SIMD (Single Instruction, Multiple Data) and SIMT (Single Instruction, Multiple Threads) execution models. Understanding these models is critical as they form the bridge between the GPU's hardware structure and the parallel programming paradigms used in AI.

The Foundation: Single Instruction, Multiple Data (SIMD)

The core principle behind the GPU's efficiency is to minimize the overhead of instruction processing. If you need to perform the same operation on a large dataset, it's wasteful to fetch, decode, and issue a separate instruction for each piece of data. The SIMD model solves this by allowing a single instruction to operate on multiple data elements simultaneously.

To begin, let's watch a segment of a computer architecture lecture from Prof. Onur Mutlu at Carnegie Mellon University. It will formally introduce SIMD within the broader context of computer architectures.

As the video explains, SIMD is a form of data parallelism. The concurrency arises from performing the same operation on different data. This is distinct from the control-flow parallelism (thread-level parallelism) common in multi-core CPUs, where different threads can execute entirely different instruction streams.

Benefits of SIMD

The primary benefit of the SIMD model is its efficiency. By fetching and decoding a single instruction that triggers many parallel operations, the processor amortizes the cost of control logic over many ALUs. This allows a much larger fraction of the chip's power and transistor budget to be dedicated to computation.

Prof. Mutlu elaborates on these benefits in the next segment.

How SIMD is Implemented

Historically, SIMD has been implemented in two main ways:

1. Array Processors: Use multiple processing elements to execute the same instruction on different data elements at the same time (parallelism in space).
2. Vector Processors: Use a single, deeply pipelined functional unit to execute the same instruction on different data elements in consecutive time steps (parallelism in time).

Modern GPUs are effectively a hybrid, combining both spatial and temporal parallelism. They feature multiple parallel "lanes" of execution, and each lane is pipelined.

To handle data-parallel code, a vector processor needs specific hardware features. Prof. Mutlu provides a clear overview of these components. This foundational knowledge is crucial because, as we'll see, the SIMT model builds directly upon these hardware concepts.

To summarize the key ideas from the video:

• Vector Registers: Hold multiple data elements (a vector) instead of a single scalar value.
• Strided Access: A programmable stride allows efficient loading of data that is stored contiguously (e.g., a row of a matrix) or at regular intervals (e.g., a column of a matrix).
• Gather/Scatter: For truly irregular data patterns, an index vector is used to "gather" data from arbitrary memory locations into a vector register and "scatter" it back after computation.
• Masking (Predication): A bitmask determines which data elements (lanes) are active for a given instruction. This is the fundamental mechanism for handling if statements in a SIMD context.

The major drawback of the pure SIMD/vector model is its reliance on highly regular, vectorizable code. It works brilliantly for graphics and scientific computing but is less effective for tasks with complex, data-dependent control flow.

The Evolution: Single Instruction, Multiple Threads (SIMT)

Modern GPUs use an execution model called Single Instruction, Multiple Threads (SIMT). SIMT is a powerful abstraction layer built on top of SIMD hardware. It combines the efficiency of SIMD with a more flexible and familiar programming model.

Instead of writing explicit vector instructions, a programmer writes code for a single scalar thread. The GPU hardware then takes a large group of these threads—called a warp (NVIDIA) or wavefront (AMD)—and executes them in lockstep on the SIMD execution units. Each execution unit within the SIMD datapath is called a lane.

The article "SIMD < SIMT < SMT" provides an excellent conceptual framework for understanding the key differences that make SIMT more flexible than traditional SIMD.

Let's break down these three key differences:

1. Single Instruction, Multiple Register Sets (Scalar Programming Model):

   • In a pure SIMD model, the programmer deals with explicit vector registers and instructions (e.g., _mm_mul_ps in SSE).
   • In the SIMT model, you write a standard scalar program (e.g., c[i] = a[i] * b[i];). The hardware transparently maps the variable i for each thread to a different SIMD lane. Each thread has its own private register state, giving the illusion of independent execution. This is a far more intuitive programming model.

2. Single Instruction, Multiple Addresses (Flexible Memory Access):

   • While vector processors introduced gather/scatter, SIMT elevates this to a core feature. Each thread in a warp can calculate its own memory address independently.
   • The hardware then coalesces these memory requests. If threads access nearby locations, the requests can be served efficiently in a single transaction. If they access disparate locations (the gather/scatter case), the hardware handles it correctly, albeit with a performance penalty. This makes implementing algorithms like histogramming or traversing linked data structures possible.

3. Single Instruction, Multiple Flow Paths (Control Flow Divergence):

   • This is the most significant departure from the classic SIMD model. What happens when threads in the same warp encounter an if-else statement and need to take different paths? This is called control flow divergence.
   • Pure SIMD hardware can't execute two different instruction pointers at once. SIMT hardware solves this by serializing the paths.
     1. The hardware first executes the if block, using a hardware active mask (like the vector mask register we saw earlier) to disable the threads that should take the else path.
     2. Then, it inverts the mask and executes the else block for the other set of threads.
     3. Finally, the threads reconverge at the end of the conditional block to continue executing in lockstep.

The performance penalty of divergence is that the total execution time is the sum of both the if and else branches.

To see exactly how this works at a hardware level, the following video provides a detailed walkthrough of divergence management using a SIMT stack.

Synthesis and Comparison

We've seen that SIMD provides efficiency through shared instruction processing, while SIMT adds a layer of flexibility for programming and handling more complex code patterns.

The move from explicit vector instructions to the scalar SIMT model was a critical step in making GPUs more general-purpose and programmable, paving the way for GPGPU and the AI revolution.

Conclusion

This lesson explored the execution models that enable GPUs to perform massive parallel computation efficiently.

Key Takeaways:

• SIMD (Single Instruction, Multiple Data) is an execution model that improves efficiency by using a single instruction to perform the same operation on multiple data elements. It relies on hardware like vector registers and mask registers.
• SIMT (Single Instruction, Multiple Threads) is the model used by modern GPUs. It builds on SIMD hardware but provides a more flexible programming abstraction where the programmer writes code for a single, scalar thread.
• SIMT's key advantages over pure SIMD are a more intuitive scalar programming model, native support for irregular memory access (gather/scatter), and a hardware mechanism to handle control flow divergence by serializing paths and using an active mask.
• The trade-off for SIMT's flexibility is a performance penalty when threads within a warp diverge, as the total execution time becomes the sum of all divergent paths.

Preview of the Next Lesson:

We have now covered the core architectural differences (Lesson 1) and the primary execution models (Lesson 2). In the next lesson, we will synthesize these concepts to formally compare latency-oriented (CPU) vs. throughput-oriented (GPU) design philosophies. We will delve into the concept of latency hiding—the GPU's primary strategy for dealing with memory stalls—and see how the SIMT model with its thousands of threads is the key to making this strategy work.