GPU Utilization in LLM Inference: Prefill and Decode

Welcome back! In the previous lesson, we explored the GPU memory hierarchy and established that while GPUs have immense computational power, their performance on LLM workloads is often limited by memory bandwidth—the "memory wall." We introduced the idea that LLM inference is not a single, monolithic process, but rather a two-phase operation: prefill and decode, with distinct performance characteristics.

This lesson will connect those phases directly to their unique GPU resource utilization patterns. Your goal is to be able to explain precisely why the prefill phase is compute-bound, saturating the GPU's processing cores, while the decode phase is memory-bound, bottlenecked by the speed of HBM. Understanding this dichotomy is fundamental to diagnosing performance issues and designing efficient LLM serving systems.

1. The Two Phases of Inference

As a quick refresher from our previous discussions, an LLM generates text through an autoregressive process. This process is split into two distinct phases:

1. Prefill: The model processes your input prompt all at once.
2. Decode: The model generates the output one token at a time.

These two phases place very different demands on the GPU. Let's examine each one's resource profile.

2. The Prefill Phase: Parallel and Compute-Bound

When you send a prompt to an LLM, the model can process all the tokens in that prompt in parallel. This is because every input token is known upfront.

The key takeaway is that the prefill phase performs a large batch of computations simultaneously. Think of it as a large matrix-matrix multiplication. This type of operation has a high arithmetic intensity—it performs many floating-point operations (FLOPs) for every byte of data it reads from memory.

As a result, the GPU's Tensor Cores are kept extremely busy, often running at or near 100% utilization. The limiting factor is the raw computational power of the GPU, making the prefill phase compute-bound.

The following video provides a clear visual analogy for this.

3. The Decode Phase: Sequential and Memory-Bound

Once the prompt is processed, the model begins generating the response. This happens sequentially, one token at a time. To generate the next token, the model needs to consider the original prompt and all the tokens it has already generated.

This is where the process becomes memory-bound.

To generate a single token, the GPU must load the entire model's weights (which can be tens or hundreds of gigabytes) from the slow, off-chip HBM into the fast, on-chip SMs. It also needs to load the KV cache, which stores the intermediate attention states of all previous tokens.

This operation has a low arithmetic intensity. The GPU reads a massive amount of data from memory to perform a relatively small number of calculations (a matrix-vector operation) to produce just one token. The powerful Tensor Cores spend much of their time waiting for data to arrive. The performance bottleneck is no longer the speed of computation, but the speed of memory access. Therefore, the decode phase is memory-bound.

The animated diagram below perfectly illustrates the difference in GPU utilization.

This distinction between compute-bound prefill and memory-bound decode is one of the most important concepts in LLM inference performance.

4. Implications for Concurrent Serving

In a real-world serving environment, you aren't handling just one request; you're handling many concurrent requests. The different characteristics of the prefill and decode phases create a significant scheduling challenge, which is highly relevant to your background in SRE and distributed systems.

Imagine two users:

• User A submits a long document for summarization (a long prefill phase).
• User B is in the middle of an interactive chat session (many fast decode steps).

If the system naively processes User A's long prefill request, User B will experience a noticeable pause in their token generation. This is a form of Head-of-Line (HOL) blocking. The compute-intensive prefill task starves the memory-intensive decode tasks of GPU time, harming latency for interactive users.

The following video explains this challenge very well.

Because of this performance dichotomy, throughput and latency are affected differently depending on the workload.

As an LLM Ops engineer, you will constantly be balancing the trade-offs between these metrics. For interactive applications like chatbots, low TTFT and low time-per-output-token are critical. For offline batch processing jobs, maximizing overall token throughput is the main goal.

Conclusion

In this lesson, we dissected the two phases of LLM inference and connected them to their distinct GPU resource profiles. This understanding moves you beyond simply knowing that "LLMs are memory-hungry" to being able to articulate when and why they are limited by different parts of the GPU architecture.

Here are the key takeaways:

• The prefill phase processes the input prompt in parallel. It is a series of large matrix-matrix operations with high arithmetic intensity, making it compute-bound and leading to high GPU compute utilization.
• The decode phase generates output tokens sequentially. It is a series of matrix-vector operations with low arithmetic intensity, requiring repeated reads of the model weights and KV cache from HBM. This makes it memory-bound and leads to lower GPU compute utilization.
• In a concurrent environment, long prefill operations can block short decode operations, increasing latency for interactive users. This is a critical challenge for LLM serving systems.
• These phases directly map to key performance metrics: prefill latency determines Time To First Token, while decode latency determines the time per output token.

So far, we have discussed this behavior theoretically. In our next lesson, you will learn to use the nvidia-smi command-line utility to observe these patterns in practice. You will monitor a live GPU and see for yourself how utilization spikes during prefill and drops during decode, making these abstract concepts tangible.