Serving Quantized LLMs

Welcome back! In our last lesson, you gained control over the performance of LLM serving by configuring core parameters like concurrency and token limits. We saw how these settings are fundamentally constrained by GPU memory, especially when running models in their default 16-bit precision (FP16).

This lesson introduces quantization, a powerful optimization technique that directly addresses this memory bottleneck. You will learn how to load and serve models whose weights have been converted to a lower precision, such as 4-bit integers. For your role as an LLM Ops engineer, this is a critical skill. Mastering quantization allows you to significantly reduce operational costs by using smaller GPUs or increase throughput by serving more users on the same hardware.

We will explore what quantization is, examine popular methods like GPTQ and AWQ, and then get hands-on with loading these compact models into the serving frameworks you're now familiar with, vLLM and TGI.

1. What is Quantization?

At its core, quantization is the process of reducing the numerical precision of a model's weights. Most LLMs are trained using 32-bit or 16-bit floating-point numbers (FP32 or FP16/BF16), which offer high precision but consume a lot of memory. For a 7-billion parameter model, the weights alone in FP16 require about 14 GB of GPU RAM.

Quantization converts these weights to lower-precision formats, most commonly 8-bit or 4-bit integers (INT8 or INT4). This has two major benefits:

1. Reduced Memory Footprint: A 4-bit model uses approximately 4 times less memory for its weights than a 16-bit model. That 7B model now only needs about 3.5 GB.
2. Faster Computation (Potentially): Modern GPUs have specialized hardware that can perform integer math much faster than floating-point math, which can lead to higher inference speeds.

Of course, there is a trade-off: reducing precision can lead to a loss in model accuracy. The goal of sophisticated quantization techniques is to minimize this quality degradation.

2. An Overview of Popular Quantization Methods: GPTQ and AWQ

While simple quantization can harm model quality, several advanced algorithms have been developed to perform this conversion more intelligently. For an Ops role, you don't need to implement these algorithms, but you do need to know what they are and which ones your serving frameworks support. GPTQ and AWQ are two of the most widely used.

• GPTQ (Post-Training Quantization): This method quantizes a model after it has been trained. It uses a small calibration dataset to analyze the weight distributions and iteratively quantizes the model layer by layer, making adjustments to minimize the impact on accuracy.
• AWQ (Activation-aware Weight Quantization): AWQ improves upon GPTQ by recognizing that not all weights are equally important. It measures the model's activations (the outputs of neurons) to identify the 1% or so of "salient" weights that are most critical for performance and protects them from quantization. This often results in better accuracy for the same level of compression.

The performance differences between these methods can be seen in benchmarks. Different techniques excel at different things.

3. Hands-On: Serving Quantized Models

Now, let's put this into practice. We'll use pre-quantized models from the Hugging Face Hub. A popular contributor, "TheBloke," provides high-quality quantized versions of most open-source LLMs.

Serving a GPTQ Model with Text Generation Inference (TGI)

TGI has built-in support for various quantization schemes, which you can enable with a simple command-line flag.

Now, let's launch a TGI container with a GPTQ model.

1. Stop any running containers.

2. Run the TGI Docker command. We are pointing to a GPTQ version of Mistral-7B-Instruct-v0.2 and telling TGI to use its GPTQ engine via --quantize gptq.

   docker run --rm --gpus all --shm-size 1g -p 8080:80 \
   -v $HOME/.cache/huggingface:/data \
   ghcr.io/huggingface/text-generation-inference:2.0 \
   --model-id TheBloke/Mistral-7B-Instruct-v0.2-GPTQ \
   --quantize gptq
   

3. Wait for the server to be ready. Once you see the "Connected" message, you can send a request from another terminal, just as you did in previous lessons.

   curl http://localhost:8080/generate \
       -X POST \
       -H "Content-Type: application/json" \
       -d '{"inputs":"What is model quantization?","parameters":{"max_new_tokens":100}}'
   

You've just served a 4-bit model! Notice the startup command was nearly identical to before; the only changes were the model ID and the --quantize flag.

Serving Quantized Models with vLLM

vLLM makes serving quantized models even simpler, as it often automatically detects the quantization format from the model's configuration files. It supports GPTQ, AWQ, and others.

Let's try serving both an AWQ and a GPTQ model with vLLM's OpenAI-compatible server.

1. Stop any running TGI container.

2. Launch the vLLM server with an AWQ model. We'll use an AWQ version of zephyr-7B-beta.

   
   
   
   # Ensure you have vllm installed: pip install vllm
   python -m vllm.entrypoints.openai.api_server \
       --model TheBloke/zephyr-7B-beta-AWQ
   

   vLLM will download the model and start a server on localhost:8000.

3. From a new terminal, send a request.

   curl http://localhost:8000/v1/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "TheBloke/zephyr-7B-beta-AWQ",
       "prompt": "Explain the benefit of AWQ quantization in one sentence.",
       "max_tokens": 50
     }'
   

4. Now, stop the server (Ctrl+C) and repeat for GPTQ. The command is identical, just with a different model ID.

   python -m vllm.entrypoints.openai.api_server \
       --model TheBloke/Mistral-7B-Instruct-v0.2-GPTQ
   

   You can test it with a similar curl request, updating the "model" field.

The key takeaway is that modern serving frameworks make using these optimized models incredibly simple from an operational standpoint. The main task is selecting the right pre-quantized model from the Hub.

4. The Real-World Impact: Cost and Throughput

Why is this so important? Serving a 4-bit 7B model requires less than 5GB of VRAM. This means you can:

• Run it on consumer-grade GPUs (like an RTX 3060 with 12GB VRAM), which are far cheaper than datacenter GPUs.
• On a large datacenter GPU (like an A100 with 40GB VRAM), you could fit multiple quantized models, or, more likely, dramatically increase the concurrency (--max-num-seqs) for a single model, boosting your service's throughput.

This analysis directly ties into your goal of mastering cost modeling and capacity planning. By using a quantized model, you change the fundamental performance and cost characteristics of your service.

Conclusion

In this lesson, we demystified model quantization and demonstrated its practical application. You've learned how converting models to lower-precision formats is a key strategy for optimizing LLM inference.

Here are the key takeaways:

• Quantization reduces model memory and can increase inference speed by using lower-precision data types like INT4 or INT8 for model weights.
• GPTQ and AWQ are advanced methods that quantize models post-training while seeking to minimize accuracy loss. AWQ is generally considered to have a slight edge in quality by protecting the most important weights.
• Serving frameworks make it easy to deploy quantized models. Both TGI and vLLM can load pre-quantized models with minimal configuration changes—often just a flag or by simple auto-detection.
• The operational benefit is significant: Quantization enables running models on cheaper hardware or achieving higher throughput on existing hardware, directly improving the cost-efficiency of your service.

In our next lesson, we will explore another powerful optimization technique: prompt caching. You will learn how to configure serving frameworks to reuse computations for common prompt prefixes, further reducing latency and improving server efficiency.