CPU vs. GPU: Architectural Design Philosophies

The Great Divide: Latency vs. Throughput

At the heart of the CPU vs. GPU distinction is a trade-off between two performance goals: latency and throughput.

• Latency is the time it takes to complete a single task from start to finish. A low-latency system is optimized to make individual tasks very fast.
• Throughput is the total number of tasks that can be completed in a given amount of time. A high-throughput system is optimized to maximize the total work done, even if individual tasks take longer.

A CPU is a latency-optimized device. Think of it as a master artisan. It has a few extremely powerful and versatile cores, equipped with large caches and sophisticated control logic like branch prediction and out-of-order execution. These features are all designed to minimize the time for a single, complex instruction stream to run. This is ideal for general-purpose computing—running your operating system, web browser, or a Python script—where tasks are often serial and unpredictable.

A GPU, on the other hand, is a throughput-optimized device. Think of it as a massive, specialized factory assembly line. It contains thousands of simpler, less powerful cores. The entire architecture is geared towards executing a massive number of parallel, similar computations simultaneously. It sacrifices single-thread performance to achieve enormous aggregate computational power. This is exactly what's needed for tasks like rendering graphics (computing the color of millions of pixels) and, as we'll see, training neural networks (performing millions of matrix multiplications).

This short video provides an excellent high-level overview of this fundamental difference.

Control Logic and the Execution Model

The latency vs. throughput trade-off is a direct result of different approaches to control logic and execution.

CPUs follow a Multiple Instruction, Multiple Data (MIMD) model. Each core is an independent processor that fetches its own instructions and operates on its own data. The operating system is responsible for scheduling different programs or threads onto these cores. This provides maximum flexibility but requires significant overhead in silicon for control logic on each core.

GPUs evolved a different model born from the needs of computer graphics. The original task was to run the same small program (a "shader") for every single pixel on the screen—a perfectly parallel problem. This led to the Single Instruction, Multiple Thread (SIMT) execution model, which is a more flexible and powerful variant of the classic Single Instruction, Multiple Data (SIMD) paradigm.

In the SIMT model:

1. A program, called a kernel, is launched on the GPU.
2. The GPU creates thousands or millions of threads to execute this kernel.
3. These threads are grouped into small batches, typically of 32, called warps.
4. All threads within a warp execute the same instruction at the same time, but on different data.

This lockstep execution allows the GPU to dedicate the vast majority of its transistors to arithmetic logic units (ALUs) for computation, rather than complex control logic and large caches. While this design is incredibly efficient for data-parallel tasks, it comes with a constraint: if threads within a warp need to take different paths (e.g., due to an if-else statement on data-dependent conditions), some threads will be idle while the others execute. This phenomenon is called warp divergence.

The following video clips provide more detail on this crucial concept.

This next video from Stanford's parallel computing course provides a deeper historical and architectural context for why GPUs are designed this way. Given your computing background, the details on how the hardware implements the programming model should be particularly interesting.

Quantifying the Difference: Arithmetic Intensity

We now have a qualitative understanding of the design differences. But how do we quantify whether a specific workload, like a layer from ResNet-50, is better suited for a CPU or a GPU? The key metric is Arithmetic Intensity.

Arithmetic Intensity (AI) is the ratio of arithmetic operations (FLOPs) to the amount of data moved from memory (bytes).

It answers the question: "For every byte of data I fetch, how much computation do I perform?"

Workloads can be classified based on their AI:

• Memory-Bound: These are low-AI workloads. The limiting factor is the speed at which data can be fetched from memory (memory bandwidth). The processor spends much of its time idle, waiting for data. Examples include vector addition or streaming large amounts of data.
• Compute-Bound: These are high-AI workloads. The limiting factor is the processor's peak computational speed (peak FLOPs). The processor is kept busy with calculations because there's enough work to do on the data it already has. Dense matrix multiplication is a classic example.

Let's explore this concept through a short reading.

This brings us to the Roofline Model, a simple but powerful visual tool for understanding hardware performance.

In the graph above:

• The y-axis is Performance, measured in Floating Point Operations per Second (FLOPs/sec). Higher is better.
• The x-axis is the Arithmetic Intensity of a given computation (FLOPs/byte).
• The slanted line represents the memory-bound region. Performance is limited by memory bandwidth. For any given AI, the performance is AI * Memory_Bandwidth.
• The horizontal line represents the compute-bound region. Performance is capped by the processor's peak arithmetic bandwidth (its maximum FLOPs/sec).

The "Ridge Point" is the crucial intersection. Workloads with an AI to the left of this point are memory-bound; those to the right are compute-bound. GPUs, with their massive parallelism and high memory bandwidth, are designed to push both the slanted line and the horizontal roof as high as possible, enabling them to achieve high performance on workloads with sufficient arithmetic intensity. CPUs, with their sophisticated caches, are designed to effectively increase the AI of memory-bound tasks by keeping data closer to the processor, but they cannot match the raw throughput of a GPU on truly compute-bound problems.

Conclusion

In this lesson, we've established the fundamental philosophical differences between CPUs and GPUs, which are essential for understanding everything that follows in this course.

Key Takeaways:

• CPU (Latency-Optimized): Designed as a general-purpose "jack of all trades." It uses a few powerful cores with complex control logic and large caches to execute single instruction streams as fast as possible.
• GPU (Throughput-Optimized): Designed as a specialized "master of one trade"—massive parallelism. It uses thousands of simple cores with minimal control logic to execute many similar tasks concurrently, maximizing total work done.
• Execution Model: CPUs use a flexible MIMD model managed by the OS. GPUs use a more restrictive but highly efficient SIMT model, where threads are executed in lockstep groups called warps.
• Arithmetic Intensity (FLOPs/Byte): This crucial metric quantifies how much computation a workload performs relative to its memory traffic. It helps us classify tasks as memory-bound or compute-bound and predict their performance on a given piece of hardware using the Roofline Model.

In our next lesson, we will dissect the architecture of a modern NVIDIA GPU. We will move from these high-level concepts to the concrete building blocks: Streaming Multiprocessors (SMs), CUDA cores, Tensor Cores, and the GPU's memory hierarchy. This will give us a more detailed map of the hardware we aim to master.