Introduction
In our last lesson on Amdahl's Law, we established a crucial principle: the inherently serial part of any task creates a hard ceiling on the performance gains achievable through parallelization. This insight naturally leads to the question of architectural design. If we can't solve every problem by simply adding more processors, perhaps we need different kinds of processors designed for different kinds of problems.
This lesson delves into the fundamental design philosophies that separate the two main processors in your computer: the CPU and the GPU. We will explore why one is a master of executing single, complex tasks with minimal delay, while the other is a powerhouse for executing millions of simple, independent tasks simultaneously.
Your learning objective for today is to distinguish between latency-oriented (CPU) and throughput-oriented (GPU) processor design philosophies. Understanding this distinction is the bedrock upon which all high-performance GPU programming is built. It explains why GPUs are structured the way they are and is essential for anyone aspiring to write efficient AI kernels.
Latency vs. Throughput: Two Measures of Speed
Before we compare architectures, we must be precise about what we mean by "fast." The terms latency and throughput are central to performance engineering.
- Latency is the time it takes to complete a single, discrete task from start to finish. It's about minimizing the time-to-result for one unit of work. Think of it as the travel time of a single package from source to destination.
- Throughput is the total number of tasks that can be completed in a given period. It's about maximizing the overall work rate of the system. Think of it as the total number of packages delivered per day.
A system can have high latency but also high throughput. The following video provides an excellent analogy to make this concrete.
How do Graphics Cards Work? Exploring GPU Architecture
This segment from Branch Education's video, "How do Graphics Cards Work?", introduces the core difference between CPU and GPU design philosophies using the analogy of a jumbo jet versus a cargo ship.
Please watch the section from the start of the comparison. Focus on how the jumbo jet (CPU) is optimized for speed and flexibility (low latency), while the cargo ship (GPU) is optimized for massive capacity (high throughput), even though its individual journey is slower.
The CPU: A Latency-Oriented Generalist
The Central Processing Unit (CPU) is the "brain" of a computer. It runs the operating system, manages system resources, and executes the vast majority of general-purpose applications. These tasks are often complex, interdependent, and require a response as quickly as possible. Therefore, the CPU is architected to minimize latency.
A CPU achieves low latency through several key design choices:
- A few powerful cores: Modern CPUs have a handful of cores (e.g., 8, 16, 32), but each one is incredibly complex and designed to execute a single thread of instructions as fast as possible.
- Large, sophisticated caches: As we discussed in a previous lesson, the CPU cache hierarchy (L1, L2, L3) is designed to hide the high latency of accessing main memory (DRAM). By keeping frequently used data and instructions close to the core, the CPU avoids long stalls.
- Complex control logic: A significant portion of a CPU core's silicon is dedicated to logic that extracts Instruction-Level Parallelism (ILP) from a single thread. This includes techniques like:
- Pipelining: Overlapping the fetch-decode-execute stages of multiple instructions.
- Superscalar Execution: Having multiple execution units to dispatch several instructions per clock cycle.
- Out-of-Order Execution: Reordering instructions to keep execution units busy while waiting for data dependencies to resolve.
- Speculative Execution & Branch Prediction: Guessing the outcome of a conditional branch and executing instructions along that path, rolling back only if the guess was wrong.
GPU programming part 2: architecture details and when to ...
This article from the JDriven blog provides a concise comparison of the hardware philosophies of CPUs and GPUs.
Read the section titled "The hardware". Pay close attention to the description of a CPU as a "complex beast" that uses tricks like out-of-order and speculative execution to hide latency and maximize the throughput of a single instruction stream.
In essence, a CPU throws a vast amount of hardware complexity at the problem of making a single task finish faster.
The GPU: A Throughput-Oriented Specialist
The Graphics Processing Unit (GPU) has a different mission. Originally designed to render millions of pixels in parallel for computer graphics, its architecture is fundamentally built to maximize throughput. This makes it exceptionally well-suited for the "embarrassingly parallel" workloads common in scientific computing and AI, where the same operation must be applied to massive datasets.
A GPU achieves high throughput with an almost opposite design philosophy to a CPU:
- Thousands of simple cores: A GPU contains a massive number of smaller, simpler cores (often called CUDA cores or Stream Processors). Individually, they are much less powerful than a CPU core.
- Massive Thread-Level Parallelism (TLP): Instead of using complex logic to speed up one thread, a GPU's strategy is to run tens of thousands of threads concurrently.
- Latency Hiding, Not Hiding Latency: This is a critical distinction. While a CPU tries to hide latency for a single thread using caches and prediction, a GPU tolerates latency by finding other work to do. When a group of threads (a warp) stalls—for example, waiting for a memory read—the GPU's scheduler simply switches to another resident warp that is ready to execute. Since there are always many warps ready to go, the execution units remain busy, achieving high overall throughput even if individual threads experience long latencies.
This latency hiding mechanism is the cornerstone of the GPU's throughput-oriented design. The diagram below illustrates this concept. While one warp is stalled waiting for a result, the scheduler can issue instructions from other warps to keep the execution hardware busy.
GPU programming part 2: architecture details and when to ...
Let's return to the JDriven blog post to see how this latency-hiding mechanism works.
Read the section titled "Scheduling". This explains exactly what it means for a GPU to be a "latency-hiding architecture" by using massive data parallelism to keep its hardware busy.
Synthesis: The ILP vs. TLP Trade-off
We can now synthesize the core distinction: CPUs prioritize Instruction-Level Parallelism (ILP), while GPUs prioritize Thread-Level Parallelism (TLP).
- CPU (ILP): Spends silicon real estate on complex control units and large caches to extract as much parallelism as possible from a single thread. This is ideal for latency-sensitive, serial, or irregularly structured tasks.
- GPU (TLP): Spends silicon real estate on a massive number of simple arithmetic logic units (ALUs) and registers to execute many thousands of threads at once. This is ideal for throughput-sensitive, massively parallel tasks.
The following academic paper provides a deep, technical look at this trade-off, which is fundamental to processor design.
[PDF] Revisiting ILP Designs for Throughput-Oriented GPGPU Architecture
This paper delves into the architectural trade-offs between ILP and TLP in GPU design. Reading technical papers is a key skill for a kernel engineer, as they often introduce the foundational ideas behind new hardware and software techniques.
Please read the following three excerpts: In the Introduction, focus on the paragraphs that establish the core distinction between CPU and GPU design philosophies regarding TLP and ILP. In Section II. BACKGROUND, read the paragraph describing the SIMT execution model. This explains how GPUs organize threads into warps and use the concurrency of many warps to overcome pipeline hazards (i.e., hide latency). Finally, in Section 4.4.3, read the subsection titled "Interaction between ILP and TLP". This is the key section. It explicitly discusses how both ILP and TLP can be used to overcome pipeline hazards and the resource trade-offs involved (e.g., complex logic for ILP vs. large register files for TLP).
To summarize the key differences:
| Feature | CPU (Latency-Oriented) | GPU (Throughput-Oriented) |
|---|---|---|
| Primary Goal | Minimize time for a single task (low latency) | Maximize total tasks completed per second (high throughput) |
| Core Design | Few, powerful, complex cores | Many, simple cores |
| Parallelism Strategy | Instruction-Level Parallelism (ILP) | Thread-Level Parallelism (TLP) |
| Control Logic | Large & complex (branch prediction, out-of-order exec) | Small & simple (in-order execution) |
| Cache | Large caches to reduce memory latency for a single thread | Smaller caches; relies on thread switching to hide latency |
| Best-suited Workloads | Serial tasks, operating systems, complex conditional logic | "Embarrassingly parallel" tasks, scientific computing, AI |
Conclusion
In this lesson, we've dissected the fundamental philosophies that make CPUs and GPUs two very different, yet complementary, types of processors. CPUs are latency-oriented generalists, built to execute complex, sequential code as quickly as possible. They achieve this with a few powerful cores rich in complex control logic and large caches, maximizing Instruction-Level Parallelism. GPUs, in contrast, are throughput-oriented specialists. They use thousands of simple cores to execute a massive number of parallel threads, tolerating memory latency by switching between them. This reliance on Thread-Level Parallelism makes them computational workhorses for AI and scientific computing.
Key Takeaways:
- Latency-oriented design (CPU) aims to minimize the execution time of a single task using complex cores, large caches, and Instruction-Level Parallelism (ILP).
- Throughput-oriented design (GPU) aims to maximize the total work done per unit time using thousands of simple cores and massive Thread-Level Parallelism (TLP).
- GPUs achieve high throughput by hiding latency: when one group of threads stalls waiting for memory, the hardware scheduler instantly switches to another group, keeping the execution units busy.
- Neither architecture is universally "better"; they are specialized tools designed for different computational problems. Modern high-performance systems rely on their synergy.
This lesson concludes our module on CPU and parallelism fundamentals. You now have the necessary context to appreciate the specific architectural choices made in modern GPUs. In our next module, we will begin our deep dive into the GPU itself, exploring the hierarchy of threads, the layout of its memory systems, and the function of its core components like the Streaming Multiprocessor.