Roofline Analysis of GPT-2 and ResNet-50 Layers

The Core Idea: Compute vs. Communication

At its heart, the time it takes to execute any operation on a GPU is limited by one of two fundamental bottlenecks:

1. Computation Time (T_math): The time required for the arithmetic units (like CUDA cores or Tensor Cores) to perform all the necessary floating-point operations.
2. Communication Time (T_comms): The time required to move the necessary data (model weights, activations) from high-bandwidth memory (HBM) to the on-chip processing units.

Since modern GPUs are highly pipelined and can often overlap these two tasks, the total time is determined by whichever one is longer. This gives us a simple but powerful lower bound on execution time.

The key insight is that an operation becomes compute-bound when $T_{\text{math}}>T_{\text{comms}}$ and memory-bound (or bandwidth-bound) when $T_{\text{comms}}>T_{\text{math}}$. To determine which regime an operation falls into, we need a way to characterize the operation itself, independent of the hardware. This brings us to the central concept of the roofline model: arithmetic intensity.

Arithmetic Intensity: The Kernel's Signature

Arithmetic intensity (often denoted as $I$) is the ratio of the total work an algorithm performs to the total data it moves.

This single number, with units of FLOPs/Byte, captures the data reuse characteristics of an algorithm. An algorithm with high arithmetic intensity performs many computations for each byte of data it loads, suggesting it's a good candidate for being compute-bound. Conversely, an algorithm with low arithmetic intensity is "data-hungry" and likely to be memory-bound.

To see how this works, let's watch a short video that formalizes the model.

As the video explains, we can compare an application's arithmetic intensity ($I$) to a critical value derived from the hardware itself, often called the machine balance or peak arithmetic intensity:

An operation is compute-bound if its arithmetic intensity is greater than the machine's peak intensity ($I>I_{\text{peak}}$), and memory-bound otherwise.

Let's calculate this for our target hardware, the NVIDIA H100 GPU (SXM variant). In a previous lesson, we found:

• Peak BF16/FP16 Performance: 989 TFLOP/s = $9.89\times 10^{14}$ FLOPs/s (non-sparse)
• Peak HBM Bandwidth: 3.35 TB/s = $3.35\times 10^{12}$ Bytes/s

Therefore, the peak arithmetic intensity for the H100 is:

Any operation with an arithmetic intensity greater than 295 FLOPs/Byte has the potential to be compute-bound on an H100.

The Roofline Plot

The relationship between performance, arithmetic intensity, and hardware limits is best visualized on a log-log plot.

This plot shows:

1. The Compute Ceiling (Horizontal Line): The top flat part of the roof represents the GPU's peak computational performance (989 TeraFLOPs for the H100). No matter how high the arithmetic intensity, performance cannot exceed this limit.
2. The Memory Ceiling (Diagonal Line): This slanted part of the roof represents the limit imposed by memory bandwidth. Along this line, performance is directly proportional to arithmetic intensity: Performance = I × Bandwidth.
3. The "Knee": The point where the two ceilings meet corresponds to the machine's peak arithmetic intensity (I_peak), which we calculated to be ~295 FLOPs/Byte for the H100. Kernels to the left of the knee are memory-bound; kernels to the right are compute-bound.

Now, let's use this framework to analyze our case studies.

Case Study 1: ResNet-50 Convolutional Layers

Let's analyze a typical 3x3 convolution, like the second layer in a conv3_x bottleneck block of ResNet-50, which we examined in the previous lesson.

• Input shape: $(B,C_{in},H,W)=(1,128,28,28)$
• Weight shape: $(C_{out},C_{in},K,K)=(128,128,3,3)$
• Output shape: $(B,C_{out},H^{\prime },W^{\prime })=(1,128,28,28)$
• Data type: FP16 (2 bytes per element)

1. Calculate FLOPs:
From our previous lesson, the formula is $2\cdot K^2\cdot C_{in}\cdot C_{out}\cdot H^{\prime }\cdot W^{\prime }$.

2. Calculate Bytes Moved:
We need to read the input tensor, read the weights, and write the output tensor.

• Bytes(Input) = $1\cdot 128\cdot 28\cdot 28\cdot 2\text{ bytes}\approx 0.2\text{ MB}$
• Bytes(Weights) = $128\cdot 128\cdot 3\cdot 3\cdot 2\text{ bytes}\approx 0.295\text{ MB}$
• Bytes(Output) = $1\cdot 128\cdot 28\cdot 28\cdot 2\text{ bytes}\approx 0.2\text{ MB}$
• Total Bytes $\approx 0.2+0.295+0.2=0.695\text{ MB}$

3. Calculate Arithmetic Intensity:

4. Plot and Predict:
Our calculated intensity $I_{\text{conv}}\approx 333$ is slightly greater than the H100's knee of $I_{\text{peak}}\approx 295$. This places the operation just inside the compute-bound region. This suggests that for this specific layer configuration, performance is limited by the GPU's arithmetic throughput. However, you can see it's very close to the knee. Other convolutions in the network, especially 1x1 convolutions or those with smaller channel counts, often have lower arithmetic intensity and fall squarely in the memory-bound region.

Case Study 2: GPT-2 Small Transformer Layers

The performance characteristics of Transformer layers are fascinatingly dual-natured, depending on whether we are processing an initial prompt (prefill) or generating subsequent tokens (decoding).

Let's summarize the key findings from the reading and apply them to GPT-2 small on our H100.

A. MLP and Attention Projections (Matrix Multiplies)

The core operations are matrix multiplications of the form $X[B,D]\cdot W[D,F]$, where $B$ is the batch of tokens being processed. As the article derives, the arithmetic intensity simplifies beautifully to:

Now we can analyze the two phases:

• Prefill: We process the entire input prompt at once. If the prompt length is 1024 tokens, then $B=1024$. Since $1024>I_{\text{peak, H100}}\approx 295$, the MLP operations during prefill are firmly compute-bound. This is represented by the "LLM Prefill" point in the roofline chart above.
• Decoding: We generate one token at a time. The "batch size" $B$ now refers to the number of separate sequences we are generating in parallel. If we are only processing a few requests (e.g., $B=16$), then $16<295$, and the operation is severely memory-bound. To become compute-bound, we would need to batch nearly 300 simultaneous generation requests, which presents significant systems challenges. This is the "LLM Decoding" point on the chart.

B. Self-Attention

The analysis for attention is more nuanced due to the reading and writing of the Key-Value (KV) cache. The derived arithmetic intensity is:

where $T$ is the number of query tokens (the ones we're processing now) and $S$ is the number of key/value tokens (the length of the KV cache).

• Prefill: Here, we attend to all tokens in the input prompt, so $S=T=\text{sequence length}$. The intensity becomes $I=T^2/(2T)=T/2$. For a 1024-token prompt, $I=512$. Since $512>295$, attention during prefill is also compute-bound.
• Decoding: We process one new query token, so $T=1$, against a cache of length $S$. The intensity becomes $I=S/(S+1)\approx 1$. This extremely low arithmetic intensity means that attention during the decoding phase is always profoundly memory-bound, as it involves reading the entire large KV cache from HBM just to perform a relatively small number of computations.

Conclusion

In this lesson, we bridged the gap between hardware specifications and model architecture using the Roofline Model. You now have a robust mental model for predicting performance bottlenecks before writing a single line of code.

Key Takeaways:

• The Roofline model provides an upper bound on performance, determined by the minimum of the hardware's compute ceiling and its memory ceiling.
• Arithmetic Intensity ($I=\text{FLOPs/Byte}$) is the crucial metric that characterizes a kernel. An operation is compute-bound if its intensity $I$ is greater than the machine's peak intensity $I_{\text{peak}}$, and memory-bound otherwise.
• For an H100 GPU, the critical intensity is approximately 295 FLOPs/Byte for FP16/BF16 operations.
• ResNet-50's convolutions often operate near the knee of the roofline, with some being compute-bound and others memory-bound depending on their specific shapes.
• GPT-2 layers exhibit a dual nature:
  • During prefill, both MLP and attention layers process long sequences, leading to high arithmetic intensity, making them compute-bound.
  • During decoding, both layers process one token at a time against large weight matrices or KV caches, leading to low arithmetic intensity, making them memory-bound.

This ability to predict whether a workload is limited by compute or memory is the first and most critical step in performance optimization. In our next lesson, we will begin Module 2 by learning how to measure these phenomena in practice. We will set up the CUDA toolkit and use a professional profiler, Nsight Systems, to visualize the execution of our PyTorch models and establish a concrete performance baseline.