Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 65

LLM Production Infrastructure: Latency, Throughput, and Memory

Recommended resources

Welcome to the final lesson of our course on Large Language Models. Over the past eleven lessons, we have journeyed from the fundamental building blocks of tokenization and attention to the complexities of model training, scaling laws, and efficiency techniques like quantization and knowledge distillation. Now, we arrive at the critical final step: putting these models to work in the real world. Your background in DevOps and systems architecture makes you uniquely equipped to appreciate the immense challenges and ingenious solutions involved in this process.

This lesson addresses our final learning outcome: to identify key infrastructure considerations for serving LLMs in production, including latency, throughput, and memory management. We will explore why serving LLMs is a fundamentally different and more complex problem than serving traditional machine learning models, and we will dissect the state-of-the-art techniques that make large-scale deployment feasible.

The Fundamental Challenge of LLM Inference

When serving an LLM, we are primarily concerned with a few key metrics:

  • Throughput: How many requests or tokens can the system process per second? This is a measure of capacity and directly impacts the cost per request. We want to maximize it.
  • Latency: How quickly does the user receive a response? This is typically broken down into two parts:
    • Time to First Token (TTFT): The time from when a user sends a prompt to when the first output token is generated. This measures the initial responsiveness of the system.
    • Time Between Tokens (TBT) or Inter-Token Latency: The time to generate each subsequent token. This determines the perceived "fluidity" of the stream of text.
  • Cost: The hardware resources (and therefore money) required to serve requests. This is driven by GPU time and memory.

The core difficulty in optimizing these metrics stems from the two-phase nature of autoregressive generation, which we first touched upon in Lesson 9.

  1. Prefill Phase (Prompt Processing): The model takes the entire input prompt and processes all its tokens in parallel in a single, large forward pass. This phase computes the Key-Value (KV) pairs for the prompt tokens, which are then stored in the KV cache. This operation is typically compute-bound, meaning it can effectively saturate the GPU's processing cores.
  2. Decode Phase (Token Generation): The model generates output tokens one at a time. For each new token, the model performs a forward pass using the KV cache from all previous tokens (both prompt and previously generated). Since it processes only one token at a time, this operation has very low arithmetic intensity and is severely memory-bandwidth-bound. The speed is limited not by the GPU's computational power, but by how quickly it can load the model's weights and the massive KV cache from the GPU's high-bandwidth memory (HBM).

Think of this from a systems perspective: you are managing a service that must handle two starkly different types of tasks. The prefill is a large, computationally heavy batch job, while the decode is a sequence of tiny, I/O-bound tasks. Efficiently scheduling and multiplexing these tasks on the same hardware is the central challenge of LLM serving infrastructure.

The Bottleneck: Memory, Memory, and More Memory

As we've established, the decode phase is memory-bound. But what is consuming all that memory? While the model weights are large, the primary consumer of memory during inference is the KV cache.

Let's do a quick calculation. For a model like LLaMA-7B, the KV cache for a single token requires about 2 bytes/param * (2 for K/V) * num_layers * head_dim bytes per token. This can easily be hundreds of kilobytes per token. For a request with a sequence length of 4096 tokens, the KV cache can consume over a gigabyte of GPU RAM. If you want to serve a batch of 32 such requests, the KV cache alone would require over 32 GB, quickly overwhelming the memory of many GPUs even before accounting for the model weights themselves.

This memory pressure is the fundamental constraint. To understand how it dictates performance, let's explore the relationship between batch size, throughput, and the compute/memory bounds.

Exploring the Latency/Throughput & Cost Space for LLM Inference // Timothée Lacroix // CTO Mistral

The CTO of Mistral AI, Timothée Lacroix, provides an excellent, hardware-level explanation of the performance characteristics of LLM inference. This will ground our discussion in the physical realities of the GPU.

Please watch the first two segments of this talk: The first segment introduces the key metrics and then dives into the memory-bound vs. compute-bound regimes. Pay close attention to the graph he draws, showing how execution time is constant at small batch sizes (memory-bound) and then increases linearly (compute-bound). He defines the critical batch size, B*, where the system is optimally utilized. In the second part, he calculates the memory required for the model weights and the KV cache, demonstrating why the maximum practical batch size is often far below the optimal B*, forcing us to operate in the inefficient, memory-bound regime.

As the video explains, for small batch sizes, the time per generation step is dominated by memory access. Throughput increases linearly with batch size, but latency (TBT) remains constant. Once the batch size is large enough to saturate the compute units (at B*), we enter the compute-bound regime. Here, latency starts to increase with batch size, but throughput plateaus.

This graph illustrates the two primary performance regimes in LLM inference. At small batch sizes, performance is **Bandwidth Bound**; latency is constant, and throughput increases with batch size. At larger batch sizes, performance becomes **Compute Bound**; latency increases with batch size, and throughput hits a ceiling. The goal of optimization is to push the workload from the inefficient bandwidth-bound region toward the compute-bound region without exceeding memory capacity.

The takeaway is stark: throughput is maximized by increasing the batch size. However, the massive memory footprint of the KV cache makes it extremely difficult to fit large batches into GPU memory. This is the first major problem that modern serving systems must solve.

Solving the Memory Problem: PagedAttention

Traditional LLM serving systems allocated memory for the KV cache inefficiently. For each request, they would allocate one large, contiguous block of memory, sized for the maximum possible sequence length. This is analogous to an OS memory manager that only supports allocating the maximum possible stack size for every new thread, regardless of its actual needs. This leads to two major forms of waste:

  • Internal Fragmentation: Most requests don't use the full context length, leaving the pre-allocated space underutilized.
  • External Fragmentation: The memory allocator struggles to find contiguous blocks for new requests, leaving unusable gaps.

Combined, these issues could result in over 60-80% of the KV cache memory being wasted, severely limiting the batch size and thus throughput.

The breakthrough solution to this is PagedAttention, introduced by the vLLM project. The core idea is a direct application of a classic concept from operating systems that you will be very familiar with: virtual memory and paging.

Instead of storing the KV cache for a sequence in a single contiguous block, PagedAttention partitions it into fixed-size blocks. The system maintains a block table for each sequence, which maps the logical blocks of the sequence to non-contiguous physical blocks in GPU memory.

This diagram shows the powerful analogy between memory management in an Operating System and in vLLM. Just as an OS uses a page table to map a process's virtual memory pages to non-contiguous physical memory pages, vLLM uses a block table to map a request's logical KV cache blocks to non-contiguous physical blocks in the KV Cache. This eliminates fragmentation and dramatically improves memory utilization.

This paged memory management has profound benefits:

  • Near-zero Fragmentation: Memory is allocated block-by-block only as needed, leading to memory utilization of over 96%. This allows for much larger batch sizes and dramatically higher throughput.
  • Efficient Memory Sharing: For complex decoding strategies like parallel sampling or beam search, where multiple output sequences share a common prompt, PagedAttention can simply map the logical blocks of each sequence to the same physical blocks. It uses a copy-on-write mechanism, just like the fork() system call, to handle modifications. This avoids redundant memory usage and expensive memory copies.

To understand this mechanism in detail, let's go to the source: the vLLM paper.

Efficient Memory Management for Large Language Model ...

This paper from UC Berkeley introduced the PagedAttention algorithm and the vLLM system. Its insights are foundational to modern LLM serving.

Please read the following sections to grasp the mechanics: First, read Section 4.1 to understand how the attention algorithm itself is adapted to work with non-contiguous memory blocks. Next, read Sections 4.2 and 4.3. These explain the virtual memory analogy, detailing the block tables and walking through an example of how memory is dynamically allocated during decoding. Finally, skim Section 4.4. Focus on the descriptions of parallel sampling and beam search to see how copy-on-write enables efficient memory sharing.

PagedAttention effectively solves the memory fragmentation problem, allowing systems to pack far more requests into a batch. This leads to a huge leap in throughput. However, it also brings a new challenge to the forefront: scheduling.

Solving the Scheduling Problem: The Throughput-Latency Tradeoff

With PagedAttention enabling much larger and more dynamic batches, how should a serving system decide which requests to run at each step?

  • Request-Level Batching (e.g., FasterTransformer): A simple approach where a batch of requests is run to completion before any new requests are admitted. This avoids interference but leads to poor GPU utilization as the batch size shrinks when requests finish at different times.
  • Iteration-Level Batching (e.g., vLLM, Orca): A more advanced approach where the batch is dynamic. New requests can enter and finished requests can exit at every single generation step. This is also called continuous batching.

Continuous batching significantly improves GPU utilization. However, it creates a new, critical problem: prefill-decode interference. When a new request with a long prompt arrives, the system needs to perform a long, compute-intensive prefill operation. If the scheduler prioritizes this new prefill to increase the future batch size, it must pause the ongoing decode operations for all other requests in the batch. This pause, which can last hundreds ofmilliseconds, is called a generation stall and causes a massive spike in Time-Between-Tokens (TBT) for active users.

This creates a fundamental tradeoff:

  • Prioritize Prefills: Increases overall throughput by building larger decode batches, but kills latency (high TBT).
  • Prioritize Decodes: Maintains low TBT for active requests, but new requests must wait, reducing overall throughput.

State-of-the-art systems like Sarathi-Serve resolve this tradeoff with two clever techniques that leverage the distinct characteristics of the prefill and decode phases.

Taming Throughput-Latency Tradeoff in LLM Inference with ...

This paper from Microsoft Research and Georgia Tech directly tackles the generation stall problem created by continuous batching.

Please read the following sections: First, read Section 3.1. This reinforces the compute-bound nature of prefill and the memory-bound nature of decode, highlighting the "slack" in compute utilization during decode steps. Next, read the first two paragraphs of Section 3.2 and study Figure 7. This clearly explains and visualizes the "generation stall" problem in schedulers like vLLM. Finally, read Sections 4.1 and 4.2. This introduces Sarathi-Serve's two key solutions: chunked-prefills and stall-free batching.

The solution presented in Sarathi-Serve is elegant:

  1. Chunked Prefills: Instead of running a long prefill operation all at once, break it into smaller chunks. Each chunk is still large enough to be computationally efficient, but short enough to not cause a major stall.
  2. Stall-Free Batching (Coalescing): Remember that the decode phase is memory-bound, leaving GPU compute units underutilized. Sarathi-Serve uses this available compute "slack" to process a prefill chunk in the same batch as the ongoing decodes.

This creates a hybrid batch that contains both decode steps and a prefill chunk. By carefully budgeting the total number of tokens in the hybrid batch, the system can make progress on new requests without significantly increasing the latency for existing ones. It breaks the throughput-latency tradeoff, allowing for both high throughput and low, predictable tail latency.

Conclusion

In this final lesson, we have connected the theoretical models we've studied to the harsh realities of production deployment. The unique characteristics of autoregressive generation create a complex optimization landscape where memory management and scheduling are paramount.

Here are the key takeaways for serving LLMs in production:

  • The Two-Phase Problem: Inference is split into a compute-bound prefill phase and a memory-bound decode phase. Efficiently managing both is the core challenge.
  • The Memory Bottleneck: The KV Cache is the dominant consumer of GPU memory, and its size limits the achievable batch size, which in turn limits throughput.
  • PagedAttention as a Solution: By applying the principles of OS paging to the KV cache, systems like vLLM eliminate memory fragmentation, dramatically increasing memory utilization and enabling larger batch sizes.
  • The Throughput-Latency Tradeoff: Continuous batching improves GPU utilization but introduces generation stalls, where long prefill operations cause latency spikes for ongoing decodes.
  • Advanced Scheduling as a Solution: Systems like Sarathi-Serve resolve this tradeoff with chunked-prefills and stall-free batching, creating hybrid batches that process new prompts without stalling active users.

This concludes our course on Large Language Models. We have journeyed from the mathematics of a single attention head to the systems architecture of a global serving fleet. You have seen how LLMs are not just monolithic artifacts, but complex systems built upon layers of innovation in architecture, training algorithms, and infrastructure engineering. Your strong foundation in DevOps and systems is the perfect launchpad for diving deeper into this fascinating intersection of software and hardware, which is where the future of this technology will be forged.

End of Module,Exit lesson