GPU Memory Hierarchy Diagram

Introduction

Welcome to the first lesson of our second module, "GPU Memory Hierarchy and Data Movement." In the previous module, we established that a GPU's massive parallelism is only effective if we can keep its thousands of cores busy. We saw with the Roofline model that performance is often constrained not by computation, but by the rate at which we can feed the processors with data. This challenge is famously known as the "Memory Wall."

This lesson directly addresses the learning outcome: Diagram the GPU memory hierarchy, including registers, shared memory, L1/L2 caches, and global memory (HBM/GDDR).

We will explore the GPU's architectural solution to the memory wall: a sophisticated, multi-tiered memory hierarchy. By the end of this lesson, you will be able to:

1. Visualize the different levels of GPU memory, from the fastest on-core registers to the vast off-chip global memory.
2. Understand the characteristics of each memory level: its location, size, speed, and scope.
3. Distinguish between the physical implementations of global memory, specifically GDDR and HBM.

1. The Architectural Big Picture: Latency Hiding

To appreciate the GPU's memory design, we must first recall its core philosophy. Unlike a CPU, which uses large caches and complex logic to minimize the latency of a single thread, a GPU is a throughput-oriented machine. It hides memory latency by having thousands of threads active at once. When one group of threads (a warp) is stalled waiting for data from slow memory, the hardware scheduler instantly switches to another warp that is ready to compute.

This "latency hiding" model only works if there are enough active warps to keep the compute units busy. The memory hierarchy is what makes this possible, providing different tiers of storage with varying trade-offs between speed, size, and cost.

Let's begin by reading an introduction to this concept.

The key takeaway is that the memory system isn't just a passive storage pool; it is an active and essential component of the GPU's execution model. The choice of where to place data directly impacts the machine's ability to hide latency and achieve high throughput.

Here is a typical diagram of a modern GPU's memory hierarchy, using the NVIDIA A100 as a concrete example.

This diagram illustrates a fundamental principle: as you move further from the processing cores (the SMs), memory becomes larger, slower, and has a wider scope (i.e., it's visible to more threads).

2. On-SM Memory: The Inner Sanctum

The fastest memory is located directly on the Streaming Multiprocessor (SM). This is where threads execute, so providing them with low-latency storage is critical.

Registers

• Location: Physically on the SM.
• Size: Smallest component per thread, but large in aggregate (e.g., 256 KB per SM on an A100).
• Speed: The fastest memory, with effectively zero latency (~1 cycle).
• Scope: Private to a single thread. No other thread can access another's registers.
• Purpose: To hold private variables for a thread and intermediate results of calculations. The compiler automatically manages register allocation. As noted in the article, register usage is a primary constraint on occupancy, the number of active warps an SM can support.

Shared Memory and L1 Data Cache

The next level on the SM is a unified block of fast SRAM that serves two roles. Modern architectures (NVIDIA Volta and newer) allow this memory to be dynamically partitioned between a programmer-managed Shared Memory and a hardware-managed L1 Data Cache.

• Location: Physically on the SM.
• Size: Typically 192 KB on an A100, configurable (e.g., 164KB for Shared Memory and 28KB for L1, or other combinations).
• Speed: Very fast, low-latency access (around 20-50 cycles).
• Scope:
  • Shared Memory: Accessible by all threads within a single thread block.
  • L1 Cache: Transparent to the programmer; caches data from Global Memory for the threads running on that SM.
• Purpose:
  • Shared Memory: Acts as a programmer-managed "scratchpad." Its primary use is to facilitate explicit data sharing and communication between threads in a block, dramatically reducing trips to slow global memory. This is fundamental for algorithms like matrix multiplication and convolutions.
  • L1 Cache: Improves performance by automatically caching frequently accessed global memory data with good spatial or temporal locality.

The following video provides a good overview of these different memory spaces from a programming perspective.

3. On-Chip and Off-Chip Memory: The Global View

Data that doesn't fit in the fast on-SM memory, or needs to be shared across all thread blocks, resides in the larger, slower tiers of the hierarchy.

L2 Cache

• Location: On the GPU chip, but outside the SMs.
• Size: Significantly larger than the on-SM memory (e.g., 40 MB on A100, and even larger on newer consumer cards).
• Speed: Slower than L1/Shared Memory (~200 cycles), but much faster than Global Memory.
• Scope: Shared by all SMs across the entire GPU.
• Purpose: The L2 cache is the last line of defense before a memory request must go off-chip to the main DRAM. It serves all requests to global memory, helping to reduce DRAM traffic. A large L2 is particularly effective at caching the working set of many AI workloads, which is why its size has grown dramatically in recent architectures.

Global Memory

• Location: Off-chip, in DRAM packages soldered to the main circuit board.
• Size: The largest memory space by far (e.g., 40-80 GB on datacenter GPUs).
• Speed: The slowest tier, with high latency (400-800+ cycles). However, it offers immense bandwidth.
• Scope: Accessible by all threads on the GPU, and also by the host (CPU) via the PCIe or NVLink bus.
• Purpose: This is the main data reservoir for the GPU. It stores the input datasets, neural network weights, and final outputs. All data must originate here before being moved into the faster on-chip memories for processing.

Physical Implementation: GDDR vs. HBM

"Global Memory" is a logical concept; physically, it's implemented using DRAM chips. The two dominant technologies are GDDR and HBM. Their architectural differences have a profound impact on performance.

As the video illustrates:

• GDDR (Graphics Double Data Rate) Memory: Achieves high bandwidth with very high clock speeds over a relatively narrow data bus (e.g., 384-bit). This is common in consumer and professional graphics cards.
• HBM (High Bandwidth Memory): Achieves massive bandwidth by using an extremely wide data bus (e.g., 4096-bit or more) at lower clock speeds. This is accomplished by stacking DRAM dies vertically and connecting them directly to the GPU package. HBM is the standard for high-end datacenter accelerators where memory bandwidth is a critical performance driver for large-scale AI training.

4. Clarifying Logical vs. Physical Memory Spaces

Your CS background makes you familiar with the distinction between logical addresses and physical memory. This concept is crucial on the GPU, and misunderstanding it is a common source of performance bugs.

Let's use the detailed article from earlier to clarify three special memory spaces.

To summarize these important distinctions:

• Local Memory: This is the most confusing term. It is logically private to a thread, but physically resides in slow, off-chip Global Memory. The compiler uses it when a thread's private data (e.g., a large array or spilled registers) is too large to fit in the fast on-chip registers. Accessing local memory is as slow as accessing global memory.
• Texture and Constant Memory: These are not separate physical memory chips. They are logical, read-only views of data that resides in Global Memory, but accessed through their own dedicated, specialized cache paths.
  • Constant Cache: Optimized for broadcasting a single value to all threads in a warp simultaneously.
  • Texture Cache: Optimized for spatial locality in 2D or 3D, useful in image processing and scientific simulations.

Conclusion

We have now mapped out the GPU memory hierarchy, a critical piece of architecture designed to feed the GPU's massively parallel compute engine. Understanding the properties of each level is the first step toward writing high-performance code.

Key Takeaways:

• The GPU memory hierarchy is a tiered system with a fundamental trade-off: speed and cost vs. size and scope.
• Memory closest to the SM (Registers, Shared Memory/L1 Cache) is fastest, smallest, and has limited scope (thread or block). It is essential for low-latency computation and inter-thread communication.
• Memory further from the SM (L2 Cache, Global Memory) is larger, slower, and has a wider scope (device-wide). The L2 cache mitigates the high latency of off-chip global memory.
• Global Memory is physically implemented as GDDR (fast clocks, narrow bus) or HBM (slower clocks, ultra-wide bus), with HBM providing superior bandwidth for data-hungry AI workloads.
• Logical abstractions like Local Memory do not always map to fast physical hardware; local memory is slow because it's stored in off-chip DRAM.

The following table, adapted from the article we've been using, provides a concise summary:

Memory Type	Scope	Location	Managed By	Latency (Cycles)	Primary Purpose
Register	Thread	On-Chip (SM)	Compiler	~1	Storing a thread's private state.
Shared	Block	On-Chip (SM)	Programmer	20-50	Fast inter-thread communication within a block.
L1 Cache	Block	On-Chip (SM)	Hardware	30-50	Caching global memory accesses for an SM.
L2 Cache	Grid	On-Chip	Hardware	~200	Last-level on-chip cache to reduce DRAM traffic.
Global	Grid	Off-Chip (DRAM)	Programmer	400-800+	The main data storage for the GPU.
Local	Thread	Off-Chip (DRAM)	Compiler	400-800+	Spilling registers and large thread-private arrays.
Constant	Grid	Off-Chip (DRAM)	Programmer	Varies	Caching and broadcasting read-only data.
Texture	Grid	Off-Chip (DRAM)	Programmer	Varies	Caching data with 2D/3D spatial locality.

Preview of the Next Lesson:

Now that we understand the map of the memory hierarchy, our next task is to learn how to navigate it efficiently. Simply accessing Global Memory is not enough; we must access it in a way that maximizes the usable bandwidth. In the next lesson, we will explore memory coalescing, the single most important concept for optimizing global memory throughput.