Welcome back to your course on deep learning engineering. In our previous lesson, we established the fundamental difference between CPUs and GPUs, framing them as latency-optimized generalists versus throughput-optimized specialists. We introduced the GPU's Single Instruction, Multiple Thread (SIMT) execution model and the concept of arithmetic intensity, which helps determine if a workload is a good fit for a GPU.
Now it's time to move from that high-level philosophy and take a much closer look at the machine itself. Today's lesson addresses the physical architecture of a modern NVIDIA GPU. We will dissect the hardware, moving from the full chip down to the individual processing units. By the end of this lesson, you will be able to describe this hierarchical structure, identifying the roles of Streaming Multiprocessors (SMs), their internal warp schedulers, the general-purpose CUDA cores, and the all-important Tensor Cores that are foundational to deep learning performance.
From Chip to Streaming Multiprocessor
A modern GPU is not a monolithic block of compute. It's a highly organized, hierarchical system designed for managing and executing a staggering number of parallel threads. At the highest level, the GPU die is partitioned into several Graphics Processing Clusters (GPCs). These are large-scale blocks that group together the core processing machinery.
The real workhorse within each GPC is the Streaming Multiprocessor, or SM. You can think of the SM as the fundamental processing unit of the GPU, analogous to a core in a multi-core CPU, but vastly more parallel. When you launch a program (a "kernel") on the GPU, the work is distributed across the many SMs on the chip.
This video from Branch Education provides a concise visual overview of this top-down structure, using the NVIDIA GA102 chip (found in the RTX 3090) as an example.
How do Graphics Cards Work? Exploring GPU Architecture
This video offers a clear, animated breakdown of the GPU's hierarchical organization. As you watch, pay attention to the layered structure: the full chip, the GPCs, and the SMs inside them.
Please watch the section from the initial chip layout down to the core counts. This will help you visualize how the GPCs and SMs fit together. Note that the specific numbers are for a particular chip, but the hierarchical concept is universal across modern GPUs.
The diagram below illustrates this same concept. Part (a) shows a simplified view of a full GPU, composed of many SMs. Part (b) then zooms in to show the components of a single SM and how the CUDA programming model maps onto the hardware.
Inside the Streaming Multiprocessor (SM)
The SM is where the magic happens. It contains everything needed to fetch, schedule, and execute instructions for hundreds of threads concurrently. A key guarantee of the CUDA programming model is that a thread block—a group of collaborating threads—is always executed entirely on a single SM. This colocation is what enables fast on-chip communication and synchronization between threads in the same block.
To understand the SM in detail, we'll turn to a comprehensive article that breaks down the architecture of a modern Hopper-generation GPU (like the H100).
Warps, Memory Hierarchy, and Why Bandwidth Beats FLOPS : How GPUs Actually Work, Part 1
This article by Suchitra Malimbada provides an excellent, up-to-date description of the SM's internal structure and function. It precisely details the components you need to know.
Please read the section titled "The streaming multiprocessor on Hopper". As you read, focus on identifying these key components and their roles: Quadrants: The sub-divisions within a single SM. Warp Scheduler & Dispatch Unit: The "brain" of the quadrant that selects which group of 32 threads (a warp) to run next. CUDA Cores (FP32): The fundamental arithmetic units for floating-point math. Tensor Cores: Specialized units for high-performance matrix multiplication.
As the reading highlights, the SM is a sophisticated engine designed for one purpose: keeping its many execution units busy. Let's formalize the roles of its key components:
- Warp Schedulers: These are the traffic controllers of the SM. An SM can have many warps of threads "in flight" at once. If one warp gets stalled waiting for data from memory (a common occurrence), the scheduler can instantly switch to another "ready" warp, effectively hiding the memory latency. This is one of the most important mechanisms for achieving high throughput.
- CUDA Cores: These are the general-purpose workhorses. They execute basic arithmetic instructions like floating-point addition and multiplication (often combined in a Fused Multiply-Add, or FMA, operation). They are flexible and handle most of the non-matrix calculations in a kernel.
- Tensor Cores: These are the specialized accelerators that have revolutionized deep learning. Instead of operating on single numbers, a Tensor Core performs a matrix-multiply-and-accumulate operation, like
D = A * B + C, on small matrices (e.g., 4x4) in a single clock cycle. Since deep learning models are dominated by matrix multiplications in components like linear layers and attention mechanisms, having dedicated hardware for this provides a massive speedup over using standard CUDA cores.
Mapping the CUDA Programming Model to Hardware
Now we can connect the software abstractions of the CUDA programming model to the physical hardware we've just examined.
- Grid: When you launch a kernel, you define a grid of thread blocks. The GPU's global scheduler, called the GigaThread Engine, is responsible for distributing these blocks to the available SMs across the entire chip.
- Block: A thread block is a group of threads (up to 1024) that are assigned as a unit to a single SM. They run to completion on that SM and can cooperate using its fast, on-chip shared memory.
- Warp: Inside the SM, threads are bundled into warps of 32. The warp is the fundamental unit of scheduling. The SM's warp scheduler selects a ready warp and issues a single instruction to all 32 of its threads.
- Thread: A thread is the programmer's view of a single execution stream. It has its own program counter and registers to maintain its state, but it is always executed as part of a warp. The execution units—the CUDA cores and Tensor Cores—are what ultimately carry out the thread's instructions.
The article you just read provides a great textual description of this hierarchy.
Warps, Memory Hierarchy, and Why Bandwidth Beats FLOPS : How GPUs Actually Work, Part 1
Let's revisit the same article to solidify the connection between the programming model and the hardware.
Please read the introductory section on threads and warps, followed by the section on blocks and grids. This will formalize the mapping from the logical grid down to the hardware-managed warp.
A Deeper Look at Tensor Cores
Given your goal of optimizing deep learning models, Tensor Cores deserve special attention. They are the primary reason why GPUs are so effective for training and inference. To better understand what they do, let's watch a short clip that visualizes their core operation.
How do Graphics Cards Work? Exploring GPU Architecture
This segment from the Branch Education video clearly illustrates the matrix math performed by a Tensor Core.
Watch the section from discussing the design of tensor cores. Notice how it performs a matrix multiplication and addition concurrently. This single, powerful operation is what accelerates the forward and backward passes of your neural networks.
The ability of a Tensor Core to process entire matrices of numbers at once, often using lower-precision formats like FP16 or BF16, is the key to its efficiency. It achieves a much higher rate of FLOPs for these specific operations than would be possible using the more general-purpose FP32 CUDA cores.
Conclusion
In this lesson, we have peeled back the layers of a modern GPU to reveal its hierarchical architecture. We journeyed from the full chip down to the individual cores, building a mental map of the hardware you will be programming.
Key Takeaways:
- A GPU is organized hierarchically: the chip contains Graphics Processing Clusters (GPCs), which in turn contain multiple Streaming Multiprocessors (SMs).
- The SM is the fundamental processing engine. It executes a thread block and contains its own schedulers, execution units, registers, and L1/shared memory.
- Within an SM, warp schedulers manage the execution of warps (groups of 32 threads) to hide memory latency and keep the execution units fed.
- The primary execution units are CUDA Cores for general-purpose arithmetic and Tensor Cores, which are highly specialized hardware units that provide massive acceleration for the matrix multiplication operations at the heart of deep learning.
- There is a direct mapping from the CUDA programming model (grid, block, warp, thread) to this hardware hierarchy.
Now that we have a solid understanding of the GPU's physical structure, our next lesson will focus on the dynamics of how code actually runs on it. We will take a deeper dive into the Single Instruction, Multiple Thread (SIMT) execution model, exploring the critical performance implications of concepts like warp divergence and predication.