Token-Aware Autoscaling for Inference Fleets

Recommended resources

The freely available resources this lesson will direct you to.

Mastering LLM Inference Optimization From Theory to Cost Effective Deployment: Mark Moyouwww.youtube.com
Go Big or Go OOM: The Art of Scaling vLLM - AI21www.ai21.com
Autoscaling LLM Services in 2025: Token‑Aware Metrics ... - DebuggAIdebugg.ai

Hello! Welcome back to our deep dive into building production-grade systems for LLM inference.

In our previous lessons, we've constructed the core of a high-performance serving stack. We've designed a dynamic batching scheduler, optimized memory with PagedAttention for the KV cache, and implemented a low-latency streaming system using Server-Sent Events (SSE). Our service is fast and efficient on a single node.

But in a real-world system like Bedrock, demand isn't constant. How do we elastically scale our expensive GPU fleet to match fluctuating user traffic, ensuring both performance and cost-effectiveness? This lesson directly addresses that challenge.

Our learning outcome is to design an autoscaling policy for the inference fleet based on token-aware queue depth, not just GPU utilization. We will dismantle why traditional metrics fail for LLMs and build, from first principles, a modern, robust scaling strategy fit for an Anthropic-level system.

1. The Failure of Traditional Autoscaling Metrics

In many distributed systems you've built, autoscaling might be triggered by simple rules like CPUUtilization > 70% or RequestCountPerTarget > 1000. For LLM inference, these metrics are not just suboptimal; they are fundamentally misleading.

Let's explore why.

Go Big or Go OOM: The Art of Scaling vLLM

The article 'Go Big or Go OOM: The Art of Scaling vLLM' from AI21 Labs provides a concise, real-world explanation of why common metrics are ineffective. It's a great starting point.

Please read the section 'Step 1: Defining an autoscaling metric', which starts under the heading 'The horizontal angle: scaling multi-node deployment'. Focus on the arguments against using GPU utilization, memory utilization, and raw throughput.

As the article points out:

GPU Utilization: A GPU at 100% utilization isn't necessarily overloaded. With effective batching, it means the system is working efficiently. Scaling on this metric would lead to constantly adding new, unnecessary instances.
Memory Utilization: The KV cache memory fluctuates constantly. High usage could indicate a healthy, large batch is being processed, not that the system is about to fail.
Requests Per Second (RPS): This is the most deceptive metric. Throughput (RPS or tokens/sec) will plateau once a server is saturated. The metric itself cannot distinguish between a server running at a healthy 90% capacity and one that is completely overwhelmed with a massive backlog of queued requests.

The core problem is that not all requests are created equal. The computational cost of a request is a function of its tokens, and this varies by orders of magnitude.

To solidify this concept, let's watch a segment on different LLM querying patterns.

Mastering LLM Inference Optimization From Theory to Cost Effective Deployment: Mark Moyou

In 'Mastering LLM Inference Optimization', Mark Moyou breaks down the four primary query patterns and their dramatically different impacts on the system.

Watch the segment from 00:17:29 to 00:19:20. As you watch, consider how each of these patterns—long input/short output (e.g., summarization) vs. short input/long output (e.g., content generation)—would stress different parts of the inference process (prefill vs. decode) and consume different amounts of KV cache.

This variability is precisely why we need a more nuanced, "token-aware" approach.

2. Designing a Token-Aware Autoscaling Policy

If requests/sec is the wrong unit of work, the right one must be tokens/sec. A robust autoscaling policy treats the system's capacity in terms of the number of prompt tokens it can process (prefill) and generation tokens it can produce (decode).

2.1. The Right Metrics for the Job

To build this policy, we first need to expose the right metrics from our inference servers.

Grafana Dashboard for LLM Inference Service Monitoring — A sample Grafana dashboard for an LLM serving system. Notice the emphasis on token-centric metrics like token throughput (prompt and generation), cache utilization, and latencies, rather than just CPU/GPU usage.

The article "Autoscaling LLM Services in 2025" provides an excellent blueprint for the metrics we need to collect.

Autoscaling LLM Services in 2025: Token‑Aware Metrics ...

Let's dive into 'Autoscaling LLM Services in 2025: Token‑Aware Metrics...' by DebuggAI. This article is a detailed guide to modern LLM autoscaling.

First, read the section 'Why RPS-based autoscaling fails for LLMs' to reinforce the concepts we just discussed. Then, carefully study 'Token-aware metrics: the only autoscaling signal that makes sense'. Pay close attention to the list of key metrics and the formulas for calculating the desired pod count.

To summarize the key metrics from the article:

Counters: tokens_in_total (prefill), tokens_out_total (decode).
Gauges: prefill_queue_len, decode_queue_len, kv_cache_bytes_used.
Rates (derived from counters): token_in_rate, token_out_rate.

2.2. The Core Autoscaling Logic

With these metrics, we can define a clear, token-based scaling logic. The process involves two steps:

Benchmark Capacity: First, you must empirically determine the sustainable token throughput of a single inference server (a "pod" in Kubernetes terms). This gives you two crucial numbers:
- $C_{in}$ : Sustainable prefill token capacity per pod (tokens/sec).
- $C_{o u t}$ : Sustainable decode token capacity per pod (tokens/sec).
Calculate Required Pods: Periodically, your autoscaler measures the current aggregate token rates across the fleet ( $R_{in}$ and $R_{o u t}$ ). It then calculates the number of pods needed for each phase, aiming for a target utilization $U_{t a r g e t}$ (e.g., 75%):

$N_{in} = ⌈ \frac{R _{in}}{C _{in} \times U _{t a r g e t}} ⌉$
$N_{o u t} = ⌈ \frac{R _{o u t}}{C _{o u t} \times U _{t a r g e t}} ⌉$

The final desired pod count is the maximum of the two, ensuring you have enough capacity for both prompt processing and token generation:

$N_{d es i r e d} = max (N_{in}, N_{o u t})$

This token-rate-based formula is the foundation of our policy.

2.3. The Critical Role of Queue Depth

Token rate tells you what the system is currently handling. But what about the work it isn't handling? Queue depth is the most direct measure of system backlog and, by extension, client-perceived latency. Your experience with priority-based queuing in Rudi makes this concept intuitive: a growing queue is the clearest sign that arrival rate is exceeding service rate.

LLM Service Request Monitoring for Autoscaling — This graph from a research paper illustrates how pending requests (the queue) remain at zero under a manageable load (6 RPS, graph B) but grow linearly once the system is saturated (7 RPS, graph A). This is the cliff-edge behavior that queue-depth monitoring detects.

An autoscaling policy that ignores queue depth is flying blind to impending overload. We can augment our token-based policy with a queue-based one.

A hybrid policy might look like this, combining metrics in a Kubernetes Horizontal Pod Autoscaler (HPA) or a KEDA ScaledObject:

Scale up if ANY of these are true:

Average token_out_rate per pod > 8000 (your C_out * U_target)
Average kv_utilization per pod > 0.8
Average num_requests_waiting per pod > 10

This hybrid approach is robust: it scales proactively based on token throughput but also reacts defensively to pressure on memory (KV cache) or a sudden backlog (queue depth).

3. Practical Implementation and Advanced Considerations

Knowing the logic is one thing; implementing it robustly in a production environment requires navigating real-world complexities.

3.1. Implementation in Kubernetes

The "Autoscaling LLM Services in 2025" article provides concrete examples of how to implement this.

Autoscaling LLM Services in 2025: Token‑Aware Metrics ...

Let's look at the practical implementation details from the same article.

Please read the section 'Kubernetes patterns with vLLM and TensorRT-LLM'. You don't need to understand every line of YAML, but focus on how the HPA and KEDA examples combine multiple metrics (llm_token_out_rate and llm_kv_utilization) to drive scaling decisions.

These examples show how to translate our design into declarative configuration that a system like Kubernetes can execute automatically. The use of a Prometheus query in the KEDA example is particularly powerful, allowing for very flexible and sophisticated scaling rules.

3.2. Real-World Operating Wisdom

Running such a system at scale reveals challenges that aren't apparent from the formulas alone. The AI21 article provides excellent operational advice.

Go Big or Go OOM: The Art of Scaling vLLM

Let's return to the 'Go Big or Go OOM' article for some hard-won practical tips.

Read 'Step 3: Finding the right threshold' and the 'Horizontal scaling' recommendations in the recap. Pay special attention to the points about slow instance startup and conservative scale-down.

The key operational principles are:

Account for Slow Startup: GPU instances are not like typical stateless web servers. Loading a multi-billion parameter model into VRAM can take several minutes. Your scaling thresholds must be "early" enough to trigger scale-up well before the system is overwhelmed, giving the new instance time to become ready.
Scale Down Conservatively: To prevent thrashing (rapidly scaling up and down), use a long stabilization window for scale-down decisions (e.g., 10-15 minutes). You don't want to pay the 3-minute startup cost again just because of a brief dip in traffic.
Separate Prefill and Decode: A more advanced design, mentioned in the DebuggAI article, is to treat prefill and decode as separate resource pools, potentially with their own queues and budgets. This is because their performance characteristics are so different—prefill is compute-bound, while decode is memory-bandwidth-bound. This aligns with the priority scheduling concepts you've implemented before, but applied to the distinct phases of LLM inference.

Conclusion

In this lesson, we have designed a sophisticated, production-ready autoscaling policy for an LLM inference fleet. We moved beyond simplistic, misleading metrics and embraced a token-aware philosophy that respects the unique workload characteristics of generative AI.

Key Takeaways:

Traditional metrics like CPU/GPU utilization and RPS are poor signals for LLM autoscaling because the cost of a request is highly variable and determined by token count.
The correct unit of capacity is tokens/sec, measured separately for the prefill (input) and decode (output) phases.
A robust scaling policy is a hybrid policy. It should be driven primarily by token rate metrics while using queue depth and KV cache utilization as critical leading indicators of saturation and backpressure.
Real-world implementation requires careful tuning of thresholds to account for slow GPU instance startup times and using long stabilization windows to prevent scale-down thrashing.

Next Lesson Preview

We've now engineered a complete, scalable, high-performance LLM serving system. But how does a business like Anthropic actually monetize this? The entire system's workload and cost are driven by tokens. In our next module, we will explore the economics of this. The first lesson will be: Describe typical input vs. output token pricing models and the rationale for their cost difference.