Static vs. Continuous Batching Performance

Welcome back! In our last lesson, you got hands-on experience with benchmarking, observing how increasing request concurrency (our simulation of static batching) boosts overall throughput at the cost of higher latency for each individual request. You now have a practical feel for this fundamental trade-off.

Today, we'll dig deeper into the "why." You'll learn why the static batching approach has inherent inefficiencies, especially for real-world traffic. We will then introduce its powerful successor, continuous batching, and you will measure the performance difference yourself. This is a critical concept for any LLM Ops engineer, as it's a cornerstone of modern, high-performance inference serving.

1. The Inefficiency of Static Batching

In the last lesson, we grouped requests into a fixed-size batch. This strategy, often called static or naive batching, has a significant weakness: the entire batch is shackled to the pace of its slowest request.

LLM requests are highly variable. Some prompts are short, some are long. Some generated responses are a single word, others are paragraphs. In static batching, if one request in a batch needs to generate 500 tokens, all other requests in that same batch—even those that finished after generating just 10 tokens—must wait. Their slots on the GPU sit idle until the entire batch is complete. This "head-of-line blocking" leads to wasted compute cycles and poor GPU utilization.

This diagram illustrates the wasted potential in static batching.

Given your background in Site Reliability Engineering, you can think of this as a form of resource contention where faster tasks are blocked by slower ones, leading to poor overall system efficiency—a classic performance anti-pattern.

2. The Solution: Continuous Batching

To solve this inefficiency, modern serving frameworks implement continuous batching (also known as dynamic batching or in-flight batching).

The core idea is simple but powerful: instead of fixing a batch at the start, the server re-evaluates and rebuilds the batch at every single token generation step.

Here's how it works:

1. The server maintains a running batch of requests.
2. In each forward pass, it generates one token for every request currently in the batch.
3. If any request finishes (by generating an end-of-sequence token), its slot is immediately freed.
4. The server can then instantly pull a new request from the waiting queue into that now-empty slot for the very next iteration.

This ensures the GPU is always working on a full batch of requests, dramatically increasing throughput and reducing idle time.

3. Hands-On: Measuring the Performance Gain

Now, let's move from theory to practice and measure this difference. We will write a Python script to compare the throughput of a basic transformers pipeline (which uses static batching) against vLLM (which uses continuous batching).

Your Experiment: Static vs. Continuous Throughput

We will run a simple benchmark. To best highlight the difference, we'll use a set of prompts where we request different output lengths.

1. Setup your Environment
If you haven't already from the previous lesson, set up your Python environment. You'll need transformers, torch, accelerate, and vllm.

pip install torch transformers accelerate vllm

2. Create and Run the Benchmark Script
Create a new Python file named benchmark_batching.py and paste the following code into it.

import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
from vllm import LLM, SamplingParams




# --- Configuration ---
MODEL_ID = "mistralai/Mistral-7B-Instruct-v0.2"
PROMPTS = [
    "Write a short story about a robot who discovers music.",  # Long output
    "What is the capital of France?",  # Short output
    "Translate the following to Spanish: 'Hello, how are you?'", # Short output
    "Explain the theory of relativity in simple terms.", # Medium output
    "List five benefits of exercise.", # Medium output
    "Who won the world cup in 2018?", # Short output
    "Tell me a joke about computers.", # Short output
    "Write a long and detailed review of the movie 'Inception'.", # Long output
]



# We want to stress the system, so we'll run a larger number of prompts
NUM_PROMPTS_TO_RUN = 160
prompts_to_run = PROMPTS * (NUM_PROMPTS_TO_RUN // len(PROMPTS))




# Specify max_new_tokens for each prompt to create variance
# (Simulates users asking for different length responses)
sampling_params_list = [
    SamplingParams(temperature=0.7, top_p=0.95, max_tokens=500), # Long
    SamplingParams(temperature=0.7, top_p=0.95, max_tokens=10), # Short
    SamplingParams(temperature=0.7, top_p=0.95, max_tokens=15), # Short
    SamplingParams(temperature=0.7, top_p=0.95, max_tokens=200), # Medium
    SamplingParams(temperature=0.7, top_p=0.95, max_tokens=150), # Medium
    SamplingParams(temperature=0.7, top_p=0.95, max_tokens=10), # Short
    SamplingParams(temperature=0.7, top_p=0.95, max_tokens=30), # Short
    SamplingParams(temperature=0.7, top_p=0.95, max_tokens=600), # Long
]
sampling_params_to_run = sampling_params_list * (NUM_PROMPTS_TO_RUN // len(PROMPTS))

print(f"--- Benchmarking with {MODEL_ID} on GPU ---")
print(f"Total prompts to run: {len(prompts_to_run)}")




# --- 1. Benchmark Static Batching (Hugging Face Transformers) ---
print("\n--- Running Static Batching (Transformers Pipeline) ---")
hf_pipe = pipeline(
    "text-generation",
    model=MODEL_ID,
    torch_dtype=torch.float16,
    device_map="auto"
)




# Extract max_tokens for the pipeline's generate method
max_tokens_hf = [params.max_tokens for params in sampling_params_to_run]

start_time_hf = time.time()



# The pipeline's batch_size argument implements static batching
hf_outputs = hf_pipe(prompts_to_run, max_new_tokens=256, batch_size=16, pad_token_id=hf_pipe.tokenizer.eos_token_id)
end_time_hf = time.time()
total_time_hf = end_time_hf - start_time_hf
throughput_hf = len(prompts_to_run) / total_time_hf

print(f"Static Batching finished in {total_time_hf:.2f} seconds.")
print(f"Throughput: {throughput_hf:.2f} requests/second.")




# --- 2. Benchmark Continuous Batching (vLLM) ---
print("\n--- Running Continuous Batching (vLLM) ---")



# vLLM internally uses continuous batching
llm = LLM(model=MODEL_ID, tensor_parallel_size=1)




# Add prompts with their specific sampling parameters
for i in range(len(prompts_to_run)):
    llm.add_request(prompts_to_run[i], sampling_params_to_run[i], request_id=str(i))

start_time_vllm = time.time()
vllm_outputs = llm.run()
end_time_vllm = time.time()
total_time_vllm = end_time_vllm - start_time_vllm
throughput_vllm = len(prompts_to_run) / total_time_vllm

print(f"Continuous Batching finished in {total_time_vllm:.2f} seconds.")
print(f"Throughput: {throughput_vllm:.2f} requests/second.")




# --- 3. Results ---
print("\n--- Comparison ---")
print(f"Static Batching Throughput:   {throughput_hf:.2f} req/s")
print(f"Continuous Batching Throughput: {throughput_vllm:.2f} req/s")
if total_time_hf > 0:
    improvement = (throughput_vllm - throughput_hf) / throughput_hf
    print(f"\nContinuous batching provided a {improvement:.2%} improvement in throughput.")

Now, run the script from your terminal:

python benchmark_batching.py

This will take a few minutes as it needs to download the model and run the inferences.

Analysis of Your Results
You should see a significant difference. The total time for vLLM will be much lower, and its throughput (requests/second) will be much higher.

This happens because the transformers pipeline, using batch_size=16, processes requests in rigid, static batches. It must wait for all 16 requests in a batch to complete before starting the next 16, even if many of them are short.

In contrast, vLLM processes all 160 requests in a continuous stream. As soon as the model finishes a short request (like "What is the capital of France?"), vLLM immediately uses that freed capacity to continue working on a longer request or even start a new one, maximizing GPU utilization. You have just empirically proven the value of this advanced batching strategy.

Conclusion

Today, you've taken a major step in understanding the mechanics of high-performance LLM serving. By moving beyond the simple model of batching from the last lesson, you can now articulate not just what batching does, but how different strategies impact performance.

Here are your key takeaways:

• Static batching is inefficient for workloads with variable input and output lengths due to head-of-line blocking, which wastes GPU resources.
• Continuous batching resolves this by dynamically managing the batch at each token generation step, immediately replacing finished requests with new ones.
• This leads to dramatically higher throughput and lower average latency, making it a non-negotiable feature for production-grade LLM serving systems.
• Frameworks like vLLM and Text Generation Inference (TGI) have this powerful optimization built-in.

In our next lesson, we will build on this knowledge. You've now used vLLM in a script and from a benchmark tool. Next, you will learn how to deploy an LLM using a high-performance framework like vLLM with GPU acceleration, setting up a persistent, production-style API server that you can configure and query.