Serving Stack Trade-offs: Latency, Throughput, and Cost

Hello! Welcome to the final lesson in our module on designing a high-performance LLM inference service.

In our journey so far, we've engineered the key components of a production-grade system: we've explored tokenization and the KV cache, designed dynamic batching and streaming responses, and finally, created a robust, token-aware autoscaling policy. We have a system that is fast, efficient, and scalable.

Now, we will take a step back and evaluate the system as a whole. Engineering is the art of making trade-offs, and nowhere is this more true than in LLM serving. This lesson is dedicated to that art.

Our learning outcome is to evaluate end-to-end trade-offs across the serving stack, focusing on time-to-first-token, throughput, and cost. We will analyze the fundamental tensions in the system, identify the levers we can pull to navigate them, and understand how to make informed decisions to meet specific performance and budget goals—a critical skill for any engineer building services at a company like Anthropic.

1. Defining the Battleground: Core Performance Metrics

To evaluate trade-offs, we must first speak a common language. In LLM serving, performance is not a single number; it's a multi-dimensional space defined by a few key metrics.

To summarize the essential metrics:

• Time To First Token (TTFT): The time from when a user sends a request to when they receive the first token of the response. This is dominated by the prefill step (processing the input prompt) and is a critical driver of perceived system responsiveness.
• Time Per Output Token (TPOT) / Time Between Tokens (TBT): The time it takes to generate each subsequent token after the first one. This is determined by the decode step and defines the "fluidity" or reading speed of the generated text.
• Throughput: The total rate of work the system can handle. This can be measured in requests per second (RPS), but as we learned in the last lesson, a more accurate measure is output tokens per second across all concurrent users.
• Cost: The total monetary cost to operate the hardware fleet required to meet the desired performance targets.

2. The Fundamental Conflict: Throughput vs. Latency

The central challenge in LLM serving is the inherent conflict between maximizing throughput and minimizing latency.

• To maximize throughput, you need to keep your expensive GPUs busy. This is achieved by processing requests in large batches, which amortizes the cost of loading the model weights and fully utilizes the GPU's parallel processing capabilities.
• To minimize latency for a single user, you want to process their request immediately with no waiting. This implies using a small batch size, ideally a batch size of one.

Processing in small batches leads to low GPU utilization, making the cost per-token very high. Conversely, large batches maximize GPU utilization but force individual requests to wait in a queue, which dramatically increases their TTFT.

These two images perfectly illustrate this tension:

This isn't just an abstract software problem; it's rooted in the physics of the hardware. To understand this, we need to distinguish between two types of bottlenecks.

• Compute-Bound: The operation is limited by the raw processing power (FLOPS) of the GPU. The prefill stage, which involves a large matrix multiplication across the entire prompt, is often compute-bound.
• Memory-Bound: The operation is limited by the speed at which data can be moved from the GPU's high-bandwidth memory (HBM) to its compute cores. The decode stage, where the entire model must be read from memory to generate just one token, is almost always memory-bound.

The trade-off exists because maximizing performance for a compute-bound task (prefill) requires different strategies than for a memory-bound task (decode).

3. The Engineer's Toolkit: Levers for Optimization

Understanding the trade-off is the first step. The next is knowing what levers you can pull to manage it. This involves making choices across the entire stack, from hardware to software.

The following video from the CTO of Mistral AI is an exceptional overview of these levers. It ties together many of the concepts we've discussed in this course into a single, coherent framework.

After watching the video, let's synthesize the key optimization levers:

1. Hardware Selection: Choosing a GPU is a cost-performance trade-off. An A100 has much higher memory bandwidth than an A10, making it faster for memory-bound decoding. An H100 adds massive compute power and new data formats like FP8. The "right" choice depends on your budget and latency targets.
2. Model Architecture & Quantization: Techniques like Grouped-Query Attention (GQA), Sliding-Window Attention (SWA), and quantization (e.g., FP16 -> INT8) all aim to reduce the memory footprint of the model and its KV cache. This allows you to fit larger batches into the same GPU memory, directly increasing potential throughput.
3. Inference Serving System: This is where techniques we've discussed, like PagedAttention and Continuous/Dynamic Batching, come into play. They are crucial for maximizing GPU utilization by eliminating wasted memory and idle compute cycles, effectively pushing the throughput-latency curve up and to the left.

4. Advanced Architectures: Disaggregating the Stack

Given your background in distributed systems at AWS, you can appreciate that when a single component has two conflicting workloads, a powerful architectural pattern is to split them. The prefill (compute-bound) and decode (memory-bound) phases are perfect candidates for this.

This evolution represents a sophisticated approach to trade-off management:

• Stage 1 (e.g., vLLM): Prefill and decode run on the same GPU. The system uses smart scheduling (continuous batching) to interleave them.
• Stage 2 (e.g., DistServe): Prefill and decode are split into separate GPU pools. This allows you to use compute-optimized GPUs (like H100) for prefill and memory-bandwidth-optimized GPUs for decode, fine-tuning the cost and performance of each.
• Stage 3 (e.g., Mooncake): The KV cache itself is treated as a separate, external resource. This tackles the memory bottleneck head-on, which becomes the dominant problem after compute is disaggregated.

This architectural approach allows for extremely fine-grained control over the end-to-end cost and performance of the serving stack.

Conclusion

We have dissected the core trade-offs at the heart of LLM inference. Building a production-grade service is not about finding a single "fastest" setup; it's about navigating a complex, multi-dimensional space to find the optimal operating point for your specific business needs.

Key Takeaways:

• LLM serving performance is a balancing act between latency (TTFT, TPOT), throughput (tokens/sec), and cost.
• The fundamental tension arises from the conflicting requirements of small batches for low latency and large batches for high throughput, a tension rooted in the compute-bound nature of prefill and the memory-bound nature of decode.
• There is no silver bullet. Improvement comes from a deep understanding of the bottlenecks and systematically applying optimizations across the stack: hardware selection, model quantization, attention mechanisms (GQA), and advanced serving techniques (continuous batching, disaggregation).
• You cannot optimize what you don't measure. A production system must have detailed monitoring for both user-visible metrics (TTFT/TPOT percentiles) and system-level metrics (GPU utilization, KV cache hit ratio) to guide tuning.

Next Module Preview

This lesson concludes our deep dive into the system design of an LLM inference service. We've covered the technical stack from the ground up. In our next module, LLM Token Economics and Billing Systems, we will move up a level to explore the business and economic layer built on top of this technology. Our first lesson will be: Describe typical input vs. output token pricing models and the rationale for their cost difference. This will connect directly to the performance characteristics and costs we've just analyzed.