Decoding GPU Specs for Inference: Cores and Memory

Welcome to the second module of our course, where we shift our focus from the software that runs LLMs to the specialized hardware that powers them.

In our last lesson, you successfully ran an LLM locally using the Hugging Face transformers library. You saw how tools like accelerate and device_map="auto" can seamlessly offload the demanding computational work to a GPU. But what exactly is happening inside that GPU? As an LLM Ops engineer, understanding the hardware is just as crucial as understanding the software, as it's the foundation for all performance tuning and cost optimization.

This lesson will teach you how to decipher a GPU specification sheet to identify the three most critical components for LLM inference: CUDA cores, Tensor cores, and High Bandwidth Memory (HBM). By the end, you'll be able to look at a spec sheet and make an initial assessment of a GPU's suitability for a given LLM workload, a core skill for capacity planning and infrastructure design.

1. The GPU: A Parallel Processing Powerhouse

Before diving into specific components, let's establish a clear mental model of why GPUs are so well-suited for deep learning. Your experience with distributed systems has shown you the power of parallel processing, and GPUs take this concept to an extreme.

To build intuition, the following video provides an excellent analogy comparing the architectures of CPUs and GPUs.

In short:

• CPUs have a small number of highly capable cores, optimized for low-latency execution of sequential tasks. Think of them as a few jumbo jets, fast and flexible for a variety of jobs.
• GPUs have thousands of simpler cores designed for high-throughput, parallel execution of the same operation across vast amounts of data. They are the cargo ships, built for bulk transport and processing.

LLM inference is fundamentally a series of massive matrix multiplications. This is a task that can be broken down into millions of independent arithmetic operations, making it a perfect workload for the GPU's "cargo ship" architecture.

2. Anatomy of a Modern ML GPU

Now let's break down the key parts of that "cargo ship." Modern NVIDIA GPUs have a hierarchical structure. The entire chip is composed of several Graphics Processing Clusters (GPCs), which are in turn made up of multiple Streaming Multiprocessors (SMs). The SMs are the true heart of the GPU, containing the execution units that do the actual work.

Within each SM, we find our three key components of interest.

CUDA Cores

CUDA (Compute Unified Device Architecture) cores are the GPU's general-purpose parallel processors. Think of them as the fundamental Arithmetic Logic Units (ALUs) of the SM. They handle a wide range of tasks, including the pointwise vector operations common in neural networks, like applying activation functions (e.g., ReLU). While they are essential, they are not the primary source of an ML GPU's massive computational power for matrix operations.

Tensor Cores

Tensor Cores are the game-changer for AI workloads. They are highly specialized hardware units designed exclusively to accelerate one specific operation: the fused multiply-add (FMA) matrix operation (D = A * B + C), which is the mathematical foundation of transformer layers.

Because they are purpose-built, a single Tensor Core can perform the work of many CUDA cores in the same amount of time, but only for this specific type of matrix math. They are the reason modern GPUs can achieve TFLOPS (trillion floating-point operations per second) performance. The performance of a GPU for LLM inference is largely determined by the capability and number of its Tensor Cores. They support various numerical precisions, such as FP32, FP16, BF16, and on newer architectures, FP8/INT8, which are critical for optimization.

High Bandwidth Memory (HBM)

High Bandwidth Memory is a type of ultra-fast stacked RAM that is located on the same physical package as the GPU chip itself. This is the GPU's primary memory, where it stores the LLM's weights, the KV cache we discussed in Module 1, and other intermediate data (activations) needed during inference.

There are two critical specs for HBM:

1. Capacity (GB): This determines the maximum size of the model that can be loaded into memory. For example, a 70-billion parameter model at half-precision (FP16) requires 70B * 2 bytes/param = 140 GB of memory, which wouldn't fit on a single 80 GB H100 GPU.
2. Bandwidth (GB/s or TB/s): This measures how quickly data can be moved between the HBM and the SMs. The Tensor Cores are incredibly fast, and they need to be constantly fed with data. If the memory bandwidth is too low, the cores will sit idle waiting for data, creating a bottleneck. This is why HBM bandwidth is often more important than raw compute power for certain parts of LLM inference.

The Baseten article below clearly explains how to find these three key numbers on a spec sheet.

3. Reading a Real GPU Specification Sheet

Let's apply this knowledge to one of the most common datacenter GPUs for LLM inference today: the NVIDIA H100.

Using the image of the H100 spec sheet above, let's identify our key metrics for the H100 SXM variant:

1. Tensor Core Performance:
   Look at the rows labeled "BFLOAT16 (BF16) / FP16" and "FP8". These are common precisions for inference. The sheet lists 989 TFLOPS for BF16/FP16 and 1,979 TFLOPS for FP8. This massive number is the raw computational throughput provided by the Tensor Cores. The doubling of performance with FP8 is a key feature of the "Hopper" architecture and a major reason for its efficiency. The video below briefly explains the significance of different GPU architectures and their support for formats like FP8.

2. CUDA Core Performance:
   While Tensor Cores handle the heavy lifting, the spec sheet also lists "FP32" performance at 67 TFLOPS. This reflects the computational power of the more general-purpose CUDA cores. Notice how this is significantly lower than the Tensor Core TFLOPS, highlighting the specialization of the hardware.

3. High Bandwidth Memory (HBM):

   • Capacity: The "GPU Memory" row clearly states 80 GB. This tells you the maximum size of assets (model weights, KV cache) you can fit in the GPU's memory.
   • Bandwidth: The "GPU Memory Bandwidth" row shows 3.35 TB/s. This is the rate at which those 80 GB of memory can be accessed by the SMs to feed the Tensor and CUDA cores.

Summary of Key GPU Specs

To summarize, here's how these components from a GPU spec sheet map to their roles in LLM inference. This table, based on information from the resources you've reviewed, can serve as your cheat sheet.

Conclusion

In this lesson, we peeled back the first layer of the hardware that powers LLMs. You learned that while a GPU is a general-purpose parallel processor, its incredible performance for AI comes from specialized components.

Our key takeaways are:

• GPUs are "cargo ships" built for parallel data processing, making them ideal for the matrix math in LLMs.
• Tensor Cores are the specialized engines providing the massive TFLOPS performance for matrix multiplication.
• CUDA Cores are the general-purpose workers that handle other parallel tasks.
• High Bandwidth Memory (HBM) is the GPU's lifeblood, defined by its capacity (how much it can hold) and bandwidth (how fast it can deliver data).

You can now look at a GPU spec sheet and connect the numbers to the physical components and their roles in serving a model.

In our next lesson, we will go one level deeper. We will explore the full GPU memory hierarchy, including the on-chip L1/L2 caches, and discuss how the interplay between memory and compute leads to the distinct "compute-bound" and "memory-bound" phases of LLM inference.