Quantizing Models: Performance & Memory Impact

Hello! In our last lesson, you did the essential work of establishing a detailed performance baseline for an un-optimized Llama-2-7B model. You now have a "before" picture, with hard data on metrics like TTFT and throughput across four different workload patterns.

Today, we'll take that baseline and apply our first major optimization technique: weight quantization. This is one of the most impactful tools in the LLM Ops arsenal for making models smaller, faster, and cheaper to run. Our goal is to apply 8-bit (INT8) and 4-bit (INT4) quantization and, most importantly, to measure the exact impact on memory and performance against the baseline you've already created. By the end of this lesson, you'll have a clear, data-driven "after" picture demonstrating the trade-offs involved.

1. What is Quantization?

At its core, quantization in deep learning is the process of reducing the number of bits used to represent a model's weights. Models are typically trained using high-precision floating-point numbers (like FP32 or FP16), which require a lot of memory. Quantization converts these weights into lower-precision integers, such as 8-bit or 4-bit integers.

This has two main benefits:

1. Reduced Memory Footprint: Fewer bits per weight means the overall model size is smaller. This is crucial for fitting large models onto GPUs with limited VRAM.
2. Faster Inference: Operations on integers can be faster than on floating-point numbers, and transferring smaller data from memory to the GPU's compute units reduces memory bandwidth bottlenecks, often leading to higher throughput.

To get a more detailed conceptual overview, let's watch a video that breaks down the fundamentals.

The trade-off, of course, is a potential loss in accuracy. By using a less precise representation, we are performing a form of lossy compression. A key part of your job as an LLM Ops engineer is to determine if this accuracy trade-off is acceptable for your specific application.

2. Part 1: Hands-On Measurement of Memory Impact

Before we dive into performance benchmarking, let's get a direct, tangible feel for the memory savings. We will use the Hugging Face transformers library along with bitsandbytes, a popular library that enables on-the-fly quantization.

You'll write a simple Python script to load the same Llama-2-7b-chat-hf model three times—in its default FP16 precision, in INT8, and in INT4—and observe the VRAM usage for each.

Step 1: Prepare Your Environment
Ensure you have the necessary libraries installed. bitsandbytes is key here.

pip install transformers torch accelerate bitsandbytes

Step 2: Create the Measurement Script
Create a file named measure_memory.py and add the following code. This script loads a specified model and then pauses, waiting for you to manually stop it. This gives you time to check the VRAM usage.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import time
import os




# Get model name from environment variable, default to Llama-2-7B
MODEL_NAME = os.environ.get("MODEL_NAME", "meta-llama/Llama-2-7b-chat-hf")



# Get quantization mode from environment variable
QUANTIZE = os.environ.get("QUANTIZE", "none")

print(f"Loading model: {MODEL_NAME}")
print(f"Quantization mode: {QUANTIZE}")

quantization_config = {}
if QUANTIZE == "int8":
    quantization_config["load_in_8bit"] = True
elif QUANTIZE == "int4":
    quantization_config["load_in_4bit"] = True




# For all modes, use bfloat16 for computations where possible
# This is separate from the storage type defined by quantization
torch_dtype = torch.bfloat16

try:
    model = AutoModelForCausalLM.from_pretrained(
        MODEL_NAME,
        torch_dtype=torch_dtype,
        device_map="auto", # Automatically map to available GPU
        **quantization_config
    )
    
    print("\nModel loaded successfully!")



    # Get the memory footprint
    mem_footprint = model.get_memory_footprint()
    print(f"Model memory footprint: {mem_footprint / 1e9:.2f} GB")

    print("\nModel is loaded. Check nvidia-smi now.")
    print("Press Ctrl+C to exit.")
    



    # Keep the script running so the model stays in memory
    while True:
        time.sleep(1)

except KeyboardInterrupt:
    print("\nExiting.")
except Exception as e:
    print(f"An error occurred: {e}")

Step 3: Run the Measurements
Open two terminals. In the first one, you'll run the Python script. In the second, you'll monitor your GPU's VRAM using nvidia-smi.

Run this command in your second terminal to get a live view of your VRAM usage:

watch -n 1 nvidia-smi

Now, run the measure_memory.py script for each precision level in your first terminal. After each run starts and the "Model is loaded" message appears, note the "memory.used" value from nvidia-smi.

1. Baseline (FP16):

   
   
   
   # No QUANTIZE variable set, so it defaults to "none"
   python measure_memory.py
   

   Record the VRAM usage. It should be around 14 GB, as expected for a 7B parameter model at 2 bytes/parameter.

2. INT8 Quantization:

   QUANTIZE=int8 python measure_memory.py
   

   Record the VRAM. You should see it drop to roughly half, around 7-8 GB.

3. INT4 Quantization:

   QUANTIZE=int4 python measure_memory.py
   

   Record the VRAM again. It should drop by another half, to around 4-5 GB.

You have now directly observed and measured the primary benefit of quantization: a dramatic reduction in memory requirements.

3. Part 2: Measuring Performance Impact with vLLM

While bitsandbytes is excellent for its flexibility, high-performance serving frameworks like vLLM are optimized for specific, pre-quantized model formats to achieve maximum throughput. The most common formats you'll encounter are AWQ (Activation-aware Weight Quantization) and GPTQ (Generative Pre-trained Transformer Quantization).

For this part of the lesson, we will use a pre-quantized INT4 AWQ model, which is a common and practical workflow in production. This allows us to use our existing vllm benchmarking setup to measure the performance impact.

Step 1: Start the vLLM Server with an AWQ Model
We'll use a popular pre-quantized model from the Hugging Face Hub. Run the following command to start the vLLM server with the AWQ-quantized version of Llama-2-7B.

# Ensure you are in an environment with vllm installed
# vLLM automatically downloads the model
python -m vllm.entrypoints.openai.api_server \
    --model TheBloke/Llama-2-7B-Chat-AWQ \
    --quantization awq

Step 2: Re-run the Benchmark Suite
Now, you will re-run the exact same four benchmark commands from the previous lesson. The only change is to the --model and --tokenizer arguments to point to the new AWQ model.

Execute these commands from your vllm repository directory.

Test 1: Short Input, Short Output (Chat)

python benchmarks/benchmark_serving.py \
    --backend openai-chat \
    --endpoint "/v1/chat/completions" \
    --model TheBloke/Llama-2-7B-Chat-AWQ \
    --tokenizer TheBloke/Llama-2-7B-Chat-AWQ \
    --dataset-name random \
    --random-input-len 256 \
    --random-output-len 128 \
    --num-prompts 200 \
    --max-concurrency 50 \
    --save-result

Test 2: Short Input, Long Output (Generation)

python benchmarks/benchmark_serving.py \
    --backend openai-chat \
    --endpoint "/v1/chat/completions" \
    --model TheBloke/Llama-2-7B-Chat-AWQ \
    --tokenizer TheBloke/Llama-2-7B-Chat-AWQ \
    --dataset-name random \
    --random-input-len 256 \
    --random-output-len 1024 \
    --num-prompts 200 \
    --max-concurrency 50 \
    --save-result

Test 3: Long Input, Short Output (RAG/Classification)

python benchmarks/benchmark_serving.py \
    --backend openai-chat \
    --endpoint "/v1/chat/completions" \
    --model TheBloke/Llama-2-7B-Chat-AWQ \
    --tokenizer TheBloke/Llama-2-7B-Chat-AWQ \
    --dataset-name random \
    --random-input-len 2048 \
    --random-output-len 128 \
    --num-prompts 200 \
    --max-concurrency 50 \
    --save-result

Test 4: Long Input, Long Output (Summarization)

python benchmarks/benchmark_serving.py \
    --backend openai-chat \
    --endpoint "/v1/chat/completions" \
    --model TheBloke/Llama-2-7B-Chat-AWQ \
    --tokenizer TheBloke/Llama-2-7B-Chat-AWQ \
    --dataset-name random \
    --random-input-len 2048 \
    --random-output-len 1024 \
    --num-prompts 200 \
    --max-concurrency 50 \
    --save-result

4. Analysis: Comparing Before and After

This is the moment of truth. Let's aggregate your results into a single table to clearly see the impact of INT4 quantization. Combine the baseline table from the last lesson with the new results you just generated.

Your results should look something like this:

Model	Workload	Input Len	Output Len	Mean TTFT (ms)	Mean TPOT (ms)	Output TPS (tok/s)
FP16	Short/Short (Chat)	256	128	Your Result	Your Result	Your Result
INT4-AWQ	Short/Short (Chat)	256	128	New Result	New Result	New Result
FP16	Short/Long (Generate)	256	1024	Your Result	Your Result	Your Result
INT4-AWQ	Short/Long (Generate)	256	1024	New Result	New Result	New Result
FP16	Long/Short (RAG)	2048	128	Your Result	Your Result	Your Result
INT4-AWQ	Long/Short (RAG)	2048	128	New Result	New Result	New Result
FP16	Long/Long (Summarize)	2048	1024	Your Result	Your Result	Your Result
INT4-AWQ	Long/Long (Summarize)	2048	1024	New Result	New Result	New Result

Now, analyze the differences:

1. Throughput (Output TPS): You should see a significant increase in tokens per second for the quantized model across most, if not all, workloads. Why? The decode phase is memory-bound. By reducing the model size by ~75%, we drastically reduce the amount of data that needs to be moved from VRAM to the compute units for each token generation step. This allows the GPU to generate tokens faster.
2. Time to First Token (TTFT): The impact on TTFT may be less pronounced or even slightly higher in some cases. The prefill phase is compute-bound, and while the data transfer is faster, the de-quantization step adds a small computational overhead. However, the overall effect is often a wash or a slight improvement.
3. Accuracy (The Unseen Metric): We haven't measured accuracy directly, but it's the critical trade-off. The tables in the following article give you an idea of the typical quality degradation.

Conclusion

In this lesson, you've taken a crucial step from simply benchmarking a model to actively optimizing it. You've used quantization, a core technique in LLM Ops, to make a model smaller and faster.

Here are the key takeaways:

• Quantization drastically reduces memory: You observed this directly, seeing VRAM usage for a 7B model drop from ~14 GB (FP16) to ~7 GB (INT8) and ~4 GB (INT4).
• Quantization boosts throughput: By alleviating the memory bandwidth bottleneck, INT4 quantization significantly increased the tokens-per-second output in your benchmarks, especially for long-generation tasks.
• There is no free lunch: The benefits of quantization come at the cost of a small, measurable drop in model accuracy. Evaluating this trade-off against product requirements is a critical engineering decision.
• The right tool for the job: You used bitsandbytes for a quick, flexible way to check memory savings and a specialized, pre-quantized AWQ model for high-performance serving with vLLM, reflecting real-world workflows.

You've now benchmarked an un-optimized model and a 4-bit quantized model. In the next lesson, we'll dive deeper by comparing the accuracy-performance trade-offs of different quantization methods, specifically looking at how AWQ and GPTQ stack up against each other for the same model.