GPU Batching and Padding: Optimizing Throughput

Hello! Welcome back to our course.

In our last lesson, we explored the KV cache, a critical optimization that prevents the re-computation of attention for past tokens. We established that while the KV cache makes inference feasible, it's a major memory consumer. Crucially, we saw that the token-by-token decode phase is often memory-bound, meaning performance is limited by the speed of GPU memory, not its raw computational power.

Today, we'll build directly on that foundation. A memory-bound process running on a single request leaves the powerful compute units of a GPU largely idle. This lesson tackles the primary strategy to overcome this inefficiency.

This lesson is designed to help you explain why batching is critical for GPU throughput and how padding impacts efficiency. Understanding this is fundamental to designing any high-performance LLM serving system, a key area for roles at companies like Anthropic.

1. The GPU Utilization Problem: Arithmetic Intensity

As we discussed, during the decode phase, the GPU loads the entire set of model weights—often over 100GB—from high-bandwidth memory (HBM) just to process a single token's worth of data. This ratio of computation to memory access is called arithmetic intensity (measured in FLOPs/byte). For a single request, the arithmetic intensity of the decode step is very low. The GPU's compute cores spend most of their time waiting for data, leading to poor utilization and low throughput.

To build a more concrete mental model of this relationship between compute, memory, and performance, let's watch a section of the following video. It introduces the "roofline model," a framework used to analyze whether a workload is limited by memory bandwidth or compute speed.

As the video explained, the core issue is that the decode phase for a single request falls on the left side of the roofline model—it's memory-bound. The overall performance is dictated by the memory bandwidth, not the GPU's peak computational power.

2. Batching: Amortizing the Cost of Memory Access

If loading the model weights is the expensive, fixed cost, how can we get more value from it? The answer is batching. Instead of processing one request at a time, the inference server groups multiple requests and processes them simultaneously in a single forward pass.

Here’s how it works:

1. The model weights are loaded once from HBM into the GPU's streaming multiprocessors (SMs).
2. Instead of being applied to a single token vector, they are applied to a batch of token vectors (one from each request in the batch).
3. This is typically done using efficient matrix-matrix multiplications instead of less efficient matrix-vector multiplications.

By doing this, we dramatically increase the amount of computation (the numerator in the arithmetic intensity formula) for the same amount of memory access for the model weights (the denominator). This amortizes the cost of loading the weights and increases the arithmetic intensity, pushing the operation up and to theright on the roofline model. The result is higher achieved FLOPs and, therefore, higher GPU throughput.

The next segment of the video you just watched explains exactly this.

Batching is the single most important technique for achieving high throughput in an LLM inference system. A system without it simply cannot serve requests economically at scale.

3. The Straggler Problem: How Naive Batching and Padding Hurt Efficiency

So, we group requests into a batch. But what happens when these requests are not uniform? In any real-world scenario, you will have:

• Varying input prompt lengths.
• Varying output generation lengths.

A simple approach, called static batching, is to:

1. Group a set of incoming requests.
2. Pad all sequences in the batch with a special padding token so they have the same length as the longest sequence.
3. Process the entire batch in lockstep until every request has finished.

This naive approach introduces significant inefficiency, a concept familiar from your work on schedulers like Rudi.

As the article explains, static batching suffers from two major issues:

1. Wasted Computation due to Padding: The GPU performs calculations for all tokens in the padded batch, including the padding tokens. These computations are useless and are immediately discarded, wasting valuable compute cycles.
2. GPU Idleness (The Straggler Problem): Shorter requests in the batch finish generating tokens early. However, because the batch is processed in lockstep, their "slots" remain occupied. The GPU is forced to either perform no-ops on these completed sequences or simply sit idle, waiting for the single longest-running sequence (the "straggler") to complete.

The video below gives a clear, step-by-step visual of this inefficiency.

Over the lifetime of a static batch, the effective batch size shrinks as requests complete, causing GPU utilization to plummet. The system reverts to the memory-bound, inefficient state we were trying to escape.

4. A Glimpse of the Solution: Dynamic, Continuous Batching

The inefficiencies of static batching—wasted compute on padding and idle resources from stragglers—are unacceptable in a production system. This led to the development of more advanced scheduling strategies.

The most common solution is known as continuous batching (or in-flight batching/iteration-level scheduling). The core idea is to make the batch dynamic. Instead of waiting for the entire batch to finish, the scheduler makes decisions after every single token generation step.

When a request finishes (generates an end-of-sequence token):

1. It is immediately evicted from the batch.
2. The resources it was using, especially its large KV cache in GPU memory, are freed.
3. A new request from the waiting queue is immediately admitted into the now-free slot.

This ensures the batch size stays consistently large and the GPU's compute units are kept busy with useful work.

This dynamic approach is far more efficient and is the standard for modern LLM serving frameworks. It maximizes GPU utilization, which directly translates to higher throughput and lower cost per token.

Conclusion

In this lesson, we established the critical role of batching and the severe performance penalty of naive padding strategies. Your background in designing high-throughput schedulers gives you a strong intuition for why these concepts are so important.

Key Takeaways:

• Batching is Critical for Throughput: The decode phase is memory-bound. Batching amortizes the high cost of loading model weights by performing more computation per memory access, thus increasing arithmetic intensity and overall GPU utilization.
• Static Batching is Inefficient: In a world of variable-length requests, static batching leads to significant waste.
  • Padding causes the GPU to perform useless computations.
  • The "straggler problem" forces shorter requests to wait for the longest one, causing GPU utilization to decay as the batch progresses.
• Continuous Batching is the Solution: By dynamically managing the batch at each step—evicting finished requests and adding new ones—continuous batching systems maintain high GPU utilization and maximize throughput.

Next Lesson Preview

We've now identified the problem (static batching) and the high-level solution (continuous batching). But how does this work in practice? How do you manage a batch of requests that are all at different points in their generation?

In our next lesson, we will dive into the mechanics. We will address the learning outcome: Design a dynamic batching strategy that balances latency and throughput by grouping requests. We'll explore the data structures and scheduling logic required to build a system that can efficiently orchestrate this complex process.