Welcome back. In our previous lesson, we mapped out the GPU memory hierarchy, identifying its distinct levels—registers, shared memory, caches, and global memory—and linking their scope to the CUDA thread hierarchy. You now have a clear "floor plan" of where data can live on the GPU.
This lesson moves from the what and where to the how fast and how much. Our goal is to quantify the performance of each memory level by comparing their latency and bandwidth. Understanding these two metrics is not just an academic exercise; it's the foundation of performance engineering. For a kernel developer, these numbers represent the fundamental "speed limits" of the hardware. They dictate which operations are expensive, which are cheap, and why nearly all high-performance GPU programming revolves around intelligently managing data movement.
Latency: The Cost of a Single Trip
Latency is the time delay between requesting a piece of data and receiving it. Think of it as the round-trip time for a single fetch. In the context of a GPU, this is typically measured in clock cycles. As you might expect, this delay varies dramatically depending on how far the data has to travel from the core that needs it.
Let's start with a video that provides a concise overview of typical latency values.
HetSys Course: Lecture 4: GPU Memory Hierarchy (Spring 2023)
This segment from Dr. Onur Mutlu's "HetSys Course" lecture gives an excellent first approximation of the latency costs for different memory spaces in a modern GPU.
Watch the clip from the um Cindy units. Pay close attention to the approximate number of clock cycles he quotes for accessing registers, shared memory, and global memory.
The video establishes a clear hierarchy of latency:
- Registers: Access is practically instantaneous, on the order of a single clock cycle. The compiler maps variables here for the fastest possible access.
- L1 Cache / Shared Memory: These on-chip memories are very fast, but introduce a small delay, perhaps around 5 to 30 cycles.
- Global Memory: This off-chip DRAM/HBM is vastly slower, costing hundreds of clock cycles for a single access.
To refine these numbers and add more detail, especially for a modern architecture like NVIDIA's H100, we can consult performance analyses. The following article provides a well-researched summary.
GPU Memory Hierarchy — How AI Training Actually Works - Medium
This article breaks down the memory hierarchy into tiers and provides specific latency figures for the H100 and A100 GPUs, giving us concrete, up-to-date numbers.
Please read the first part of the article, focusing on the section titled The Six-Tier Memory Hierarchy. Study the latency values (in cycles) for Tiers 1, 2, and 3.
Combining these sources, we can build a more detailed picture. For a state-of-the-art GPU like the H100:
| Memory Tier | Typical Latency (Clock Cycles) | Analogy |
|---|---|---|
| Registers | ~1 | Picking up a tool from your own toolbelt |
| Shared Memory | ~30 | Grabbing a tool from a shared workbench |
| L1 Cache (hit) | ~30-35 | A well-organized local supply cabinet |
| L2 Cache (hit) | ~200-260 (near) / ~750 (far) | Walking to a central warehouse on-site |
| Global Memory | ~600+ | Driving off-site to a supplier |
Notice the "near" and "far" latencies for the L2 cache. This is a fascinating detail for someone interested in low-level hardware. Modern GPU dies are so large that a signal's physical travel time across the chip becomes significant. The L2 cache is distributed across the die, and an SM accessing a distant L2 partition will experience higher latency than one accessing a partition physically closer to it. The Chips and Cheese article on the H100 provides a great analysis of this "split L2 configuration," noting it feels more like a two-level cache.
Also, it's important to distinguish between the L1 cache and Shared Memory. While they may share the same physical silicon on the SM, Shared Memory access is often slightly faster and more predictable. This is because it is a software-managed scratchpad. Your code explicitly places data there, so the hardware doesn't need to perform tag lookups and hit/miss checks, as it does for the L1 cache.
The key takeaway from latency is the staggering orders-of-magnitude difference. An access that misses the on-chip caches and goes all the way to global memory can be over 20 times slower than an access to shared memory.
Bandwidth: The Width of the Data Highway
While latency measures the time for one piece of data, bandwidth measures the total amount of data that can be moved per unit of time. Think of it as the number of lanes on a highway. Latency is how long it takes one car to travel, while bandwidth is how many cars can pass a point in one second. For throughput-oriented processors like GPUs, which operate on massive datasets, bandwidth is often the most critical performance limiter.
Let's begin with a video that highlights the vast difference in bandwidth between on-chip and off-chip memory.
Lecture 4 Compute and Memory Basics
This clip from the "GPU MODE" lecture series uses a figure from the original FlashAttention paper to visually and numerically contrast the bandwidth of shared memory and high-bandwidth global memory (HBM).
Watch the short segment from figures in the flesh. Note the comparison he makes between the speed of HBM and shared memory.
The video points out that shared memory bandwidth can be more than 10 times greater than that of global memory. This is a colossal difference. To get precise numbers, we can again turn to hardware analysis.
Nvidia's H100: Funny L2, and Tons of Bandwidth - Chips and Cheese
This "Chips and Cheese" article provides an in-depth, quantitative analysis of the H100's memory system, with a specific focus on bandwidth.
Read the section titled Bandwidth. As you read, extract the peak read bandwidth figures for the H100's L2 cache and its VRAM (HBM). Then, read the Final Words section, which summarizes the design philosophy this bandwidth-focused approach enables.
Synthesizing the data from these resources gives us a clear bandwidth hierarchy for a GPU like the H100:
| Memory Tier | Typical Bandwidth | Analogy |
|---|---|---|
| Global Memory | ~3 TB/s | A two-lane road |
| L2 Cache | ~5-6 TB/s | A six-lane highway |
| L1/Shared Mem | >20 TB/s | A massive, 20-lane superhighway |
| Registers | >100 TB/s | Instant teleportation within your workshop |
Again, the trade-off is stark: the memory with the highest bandwidth is also the most limited in size. You have terabytes per second of access to a few hundred kilobytes of shared memory per SM, but only a fraction of that bandwidth to the tens of gigabytes of global memory.
Performance Implications: Memory-Bound vs. Compute-Bound
Now we can connect these numbers to your goal of AI kernel optimization. The performance of a kernel is ultimately limited by its bottleneck. A kernel can be either:
- Compute-Bound: The kernel performs so many mathematical operations for each byte of data it reads that the memory system can easily keep up. The performance is limited by the raw floating-point calculation speed (FLOPS) of the CUDA cores.
- Memory-Bound: The kernel performs relatively few calculations for each byte of data. The powerful CUDA cores spend most of their time idle, waiting for data to arrive from a slower part of the memory hierarchy. Performance is limited by memory bandwidth.
As noted in the Medium article and a common observation in the field, most large-scale AI workloads, like training and inferring Transformers, are memory-bound. The bottleneck is not how fast the GPU can multiply matrices, but how fast it can feed the data for those matrices to the arithmetic units.
This is precisely why understanding the memory hierarchy is so critical. The art of GPU optimization is the art of managing data movement to overcome memory bottlenecks. The entire strategy is to:
- Minimize expensive trips to global memory.
- When you do fetch data from global memory, bring it into a fast on-chip memory like shared memory.
- Reuse the data in that fast memory as many times as possible before discarding it.
A classic example is tiled matrix multiplication, which you will implement in a later module. Instead of having each thread read its required data directly from global memory, a block of threads collaborates to load small "tiles" of the input matrices into shared memory. They then perform all necessary calculations using fast shared memory accesses before moving to the next tile. This drastically reduces global memory bandwidth consumption and is a cornerstone of high-performance computing.
Conclusion
In this lesson, we have quantified the staggering performance differences within the GPU memory hierarchy. These are not minor distinctions; they are orders-of-magnitude gaps that define the performance landscape.
Key Takeaways:
- Latency: Accessing global memory is exceptionally slow (~600+ cycles) compared to on-chip memory like L1 cache or shared memory (~30 cycles). Registers are effectively zero-latency.
- Bandwidth: The data transfer rate from on-chip memory can be over 10 times higher than from global memory (~20+ TB/s vs. ~3 TB/s).
- The Fundamental Trade-off: The memory with the highest performance (low latency, high bandwidth) is also the smallest in capacity.
- Performance Bottlenecks: Because of these characteristics, many large-scale compute problems, especially in AI, are memory bandwidth-bound.
- The Goal of Optimization: All optimization roads lead to minimizing data movement. The primary strategy is to keep data in the fastest, highest-bandwidth memory tiers (registers and shared memory) for as long as possible.
We have established that accessing global memory incurs a massive latency penalty. A single thread waiting 600 cycles for data would bring the GPU's processing power to a grinding halt. So, how does a GPU, a machine built for throughput, remain productive in the face of such delays? In our next lesson, we will explore the GPU's main architectural trick for dealing with this problem: hiding memory latency through the massively parallel execution of warps.