Benchmarking Initial Model Performance

Hello! Welcome to the first lesson of our module on Inference Performance Optimization.

In the last module, we concluded by learning how to run load tests to compare the performance of different serving frameworks like vLLM and TGI. You saw that performance is highly dependent on the workload and that there's a fundamental trade-off between throughput and latency.

Now, we'll shift our focus from comparing frameworks to optimizing a single one. To do that effectively, we first need to know our starting point. This lesson is all about how to establish and benchmark baseline inference performance for an un-optimized model. This "before" snapshot is critical; it will be the reference against which we measure the success of every optimization technique we apply in the upcoming lessons.

1. The Four Horsemen of LLM Workloads

A meaningful baseline isn't just a single number from a random test. It's a performance profile that reflects how your system behaves under the kinds of load it will see in production. In the last lesson, we used a generic dataset. Today, we'll get more specific by categorizing workloads into four primary patterns. Understanding these is crucial because they stress different parts of the GPU and the inference process.

As you just saw, the four key patterns are:

1. Short Input, Short Output: Classic chatbot behavior. Low latency is key.
2. Short Input, Long Output: A user asks a simple question and expects a long, generated story or article. This is decode-heavy.
3. Long Input, Short Output: Common in RAG (Retrieval-Augmented Generation) where you provide a lot of context, or in classification tasks. This is prefill-heavy.
4. Long Input, Long Output: Complex tasks like summarizing a long document into a detailed report. This is the most demanding pattern, stressing both prefill and decode.

Your role as an LLM Ops Engineer is to identify which of these patterns (or which mix) represents your application's traffic and establish a baseline for each.

2. Hands-On: Establishing a Multi-Faceted Baseline

Let's get practical. We will run a series of benchmarks against an un-optimized model, with each benchmark targeting one of the four workload patterns. We will use the same benchmark_serving.py script from the vLLM repository that you used in the previous lesson.

Step 1: Start the vLLM Server

First, let's serve a standard, un-optimized model. We'll use meta-llama/Llama-2-7b-chat-hf as our guinea pig. Open a terminal and run:

# Ensure you are in an environment with vllm installed
# If you need to log in to Hugging Face, run: huggingface-cli login
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-7b-chat-hf

Wait for the model to load and the server to start. It will be listening on http://0.0.0.0:8000.

Step 2: Run the Benchmarks

Now, open a new terminal and navigate to your cloned vllm repository directory. We will run four tests. For each command, pay close attention to the --random-input-len and --random-output-len arguments, which we'll use to simulate our four patterns.

This is a good time to familiarize yourself with the script's arguments.

Now, execute the following four commands. Each one will run for a few minutes and save its results to a JSON file.

Test 1: Short Input, Short Output (Chat)

python benchmarks/benchmark_serving.py \
    --backend openai-chat \
    --endpoint "/v1/chat/completions" \
    --model meta-llama/Llama-2-7b-chat-hf \
    --tokenizer meta-llama/Llama-2-7b-chat-hf \
    --dataset-name random \
    --random-input-len 256 \
    --random-output-len 128 \
    --num-prompts 200 \
    --max-concurrency 50 \
    --save-result

Test 2: Short Input, Long Output (Generation)

python benchmarks/benchmark_serving.py \
    --backend openai-chat \
    --endpoint "/v1/chat/completions" \
    --model meta-llama/Llama-2-7b-chat-hf \
    --tokenizer meta-llama/Llama-2-7b-chat-hf \
    --dataset-name random \
    --random-input-len 256 \
    --random-output-len 1024 \
    --num-prompts 200 \
    --max-concurrency 50 \
    --save-result

Test 3: Long Input, Short Output (RAG/Classification)

python benchmarks/benchmark_serving.py \
    --backend openai-chat \
    --endpoint "/v1/chat/completions" \
    --model meta-llama/Llama-2-7b-chat-hf \
    --tokenizer meta-llama/Llama-2-7b-chat-hf \
    --dataset-name random \
    --random-input-len 2048 \
    --random-output-len 128 \
    --num-prompts 200 \
    --max-concurrency 50 \
    --save-result

Test 4: Long Input, Long Output (Summarization)

python benchmarks/benchmark_serving.py \
    --backend openai-chat \
    --endpoint "/v1/chat/completions" \
    --model meta-llama/Llama-2-7b-chat-hf \
    --tokenizer meta-llama/Llama-2-7b-chat-hf \
    --dataset-name random \
    --random-input-len 2048 \
    --random-output-len 1024 \
    --num-prompts 200 \
    --max-concurrency 50 \
    --save-result

3. Analyzing and Documenting Your Baseline

After the tests complete, you'll have four sets of results. The goal is to synthesize these into a clear, documented baseline.

Create a Summary Table

First, extract the key metrics from each run's output and organize them into a table. This is the kind of deliverable you would share with your team. Your numbers will vary, but the structure should be similar to this:

Workload	Input Len	Output Len	Mean TTFT (ms)	Mean TPOT (ms)	Output TPS (tok/s)
Short/Short (Chat)	256	128	Your Result	Your Result	Your Result
Short/Long (Generate)	256	1024	Your Result	Your Result	Your Result
Long/Short (RAG)	2048	128	Your Result	Your Result	Your Result
Long/Long (Summarize)	2048	1024	Your Result	Your Result	Your Result

Interpret the Data

Now, look at the patterns in your table:

• TTFT Analysis: Compare the TTFT for the "Short/Short" and "Long/Short" tests. You should see a significantly higher TTFT for the long-input workload. Why? Because the prefill stage, which processes the entire input prompt at once, is compute-bound and takes much longer for 2048 tokens than for 256.
• TPOT/Latency Analysis: Compare the TPOT or total request latency for the "Long/Short" and "Long/Long" tests. The total time to complete the request will be much higher for the long-output workload. This is because the decode stage, which generates tokens one by one, is memory-bandwidth bound. More tokens simply take more time.
• Throughput (TPS): How did the overall output tokens per second change? You might notice that the highest-demand workload (Long/Long) results in lower overall throughput as the system is heavily contended.

4. Beyond Averages: The Importance of Percentiles

Your SRE background tells you that averages can be misleading. A good average latency can hide a terrible P99 latency that affects a small but significant fraction of users. For defining and meeting SLOs, percentiles are what matter.

The benchmark_serving.py script reports mean values by default, but you can configure it to capture percentiles. You can also use other tools like NVIDIA's ai-perf which provide rich statistical outputs.

To capture these in your own tests with benchmark_serving.py, you can add arguments like --percentile-metrics ttft,tpot,itl,e2el and --metric-percentiles 90,95,99.

5. Under the Hood: How Metrics Are Calculated

As an engineer, it's valuable to understand how these metrics are derived from raw data. Off-the-shelf scripts are convenient, but knowing the underlying logic allows you to build custom tooling or debug unexpected results.

A benchmarking script essentially does the following:

1. Records a timestamp when a request is sent.
2. For streaming responses, it records a timestamp for every chunk of data received.
3. Calculates the metrics from this list of timestamps.

Conclusion

In this lesson, you've moved from running simple comparative tests to establishing a formal, multi-faceted performance baseline. This is a cornerstone of the MLOps lifecycle for LLMs.

Here are your key takeaways:

• Baselines must be workload-specific: A single performance number is not enough. You need to profile your model against representative query patterns (short/long input/output).
• Prefill vs. Decode: Long inputs stress the compute-bound prefill stage (increasing TTFT), while long outputs stress the memory-bound decode stage (increasing total generation time).
• Documenting is crucial: A clear summary table of your baseline results is the essential document you'll use to justify and measure future optimizations.
• Percentiles matter for SLOs: Always look beyond averages to understand the tail-end latency that impacts user experience.

You now have a robust baseline for your un-optimized Llama-2-7b model. In our next lesson, we will apply our first major optimization technique: weight quantization. You will then re-run this exact same benchmarking suite to measure, with hard data, the improvements in performance and memory usage.