vLLM quantization can make a model fit in less GPU memory, but it is not a universal speed switch. A smaller checkpoint may leave more room for the KV cache and concurrent requests. Actual throughput depends on the model, quantization format, GPU architecture, kernel, prompt length, output length, and traffic pattern. The sensible decision is to choose a supported format, then measure a representative workload against an unquantized baseline.
This vLLM quantization guide gives you a practical selection and test process. It does not claim a benchmark result for a GPU or model we have not tested.
What vLLM quantization changes
Quantization stores some model values in fewer bits than a typical FP16 or BF16 checkpoint. The clearest first-order effect is lower weight memory. If weight memory is the reason a model cannot fit, a compatible quantized checkpoint may let it run on a smaller GPU or with more memory left for requests. A GPU still needs memory for activations, the KV cache, runtime overhead, and any parallel workers. Dividing parameter count by bits per weight is therefore only a rough lower bound, not a deployable memory estimate.
There are two common paths for vLLM quantization. You can load a checkpoint that was quantized ahead of time, or use a supported online quantization path at load time. In both cases, inspect the model card and the current vLLM compatibility matrix before choosing a format. The vLLM quantization documentation is the source of truth for methods and hardware support; it explicitly warns that the matrix changes as support evolves.
Pick the format from the checkpoint and GPU
| Option | Practical starting point | Verify before deployment |
|---|---|---|
| AWQ | A model distributed as an AWQ checkpoint and a compatible GPU | Model quality, kernel selection, prompt lengths, and actual latency. |
| GPTQ | A model distributed as a GPTQ checkpoint | Exact checkpoint configuration, vLLM version, GPU support, and quality regression. |
| FP8 | A compatible checkpoint and GPU with supported FP8 execution | Whether the chosen vLLM implementation supports your architecture and model layers. |
| GGUF | A model already distributed in GGUF format | The specific architecture and vLLM feature support; do not assume Ollama behavior maps one-to-one. |
| BitsAndBytes or compressed tensors | A checkpoint and deployment path that explicitly requires them | Runtime dependencies, loading behavior, supported kernels, and throughput. |
The current vLLM matrix lists AWQ and GPTQ for Ampere, Ada, and
Hopper. It lists FP8 through llm-compressor for Ada,
Hopper, and some AMD GPU paths, while GGUF has a different support
pattern. A100 is an Ampere GPU and H100 is a Hopper GPU, but support is
method-specific. Do not infer that every FP8, FP4, or Marlin variant
works on every card in a family. Check the exact row in the hardware
compatibility table and the checkpoint’s own metadata.
If you have not selected a model yet, compare two candidate checkpoints that use the same base model and evaluation set. Comparing an AWQ model from one family with an unquantized model from another will confound vLLM quantization with model quality.
A deployment workflow that avoids false wins
- Define the service target. Record the model, context length, expected input and output lengths, concurrent requests, p95 latency target, and quality tests. A chatbot, batch classifier, and long-context agent have different bottlenecks.
- Record the baseline. Run the unquantized checkpoint on the same GPU and vLLM version. Capture peak GPU memory, time to first token, output-token throughput, end-to-end p50/p95 latency, errors, and accepted-result rate.
- Choose a compatible quantized checkpoint. Read its model card and vLLM’s current method matrix. Pin checkpoint revision, vLLM version, CUDA/driver stack, and hardware SKU in the test record.
- Start with a small canary. Confirm the model loads, serves your required endpoint and generation settings, and produces plausible outputs before applying traffic.
- Benchmark at several concurrency levels. Measure at your real prompt and response lengths. A single request can favor a different configuration than sustained serving.
- Run the same quality suite. Quantization can change outputs. Evaluate domain-specific correctness, format adherence, refusals, and safety-sensitive cases instead of trusting perplexity alone.
- Compute cost per accepted result. GPU hourly price divided by raw tokens per second is not enough if the quantized model fails more tasks or needs retries.
For a serving benchmark, vLLM documents vllm bench serve
and its datasets and endpoint options in the official
benchmark CLI guide. Use a traffic dataset that matches your
application; synthetic random prompts are useful for controlled
comparisons but should not be presented as production performance.
Memory planning: weights are only one line item
A simple planning worksheet separates four buckets:
total GPU memory needed ≈ model weights + KV cache + runtime/activation overhead + safety headroom
The quantization format mostly changes the model-weight term. KV cache pressure grows with context length and active sequences. A model that barely fits at one request may still fail or slow sharply under concurrent long-context traffic. On multi-GPU setups, tensor parallelism changes both per-card memory and communication overhead; the smallest possible checkpoint is not automatically the cheapest deployment.
Before renting a GPU, ask whether you need to fit one larger model, raise concurrency for a smaller model, or reduce cost while preserving quality. That answer determines which memory and throughput tradeoff matters. Flatkey’s Cloud GPU Compute lists GPU rental options and asks customers to confirm configuration, region, capacity, and rental terms. Treat live availability and price as quote-time inputs, not fixed numbers in a benchmark article.
Benchmarking vLLM quantization fairly
Keep the base model, generation parameters, prompt set, output limit, cache state, and GPU fixed. Measure at least three loads: a single request for responsiveness, a representative production concurrency, and a stress point where latency begins to degrade. Report both total throughput and p95 latency. If you only report tokens per second at saturation, users cannot tell whether interactive requests remain acceptable.
Include model quality in the scorecard:
| Measure | Why it matters |
|---|---|
| Peak GPU memory | Shows whether quantization creates usable headroom. |
| Time to first token | Matters for streaming interfaces. |
| Output tokens per second | Shows generation throughput, but not task quality. |
| p95 end-to-end latency | Reveals tail behavior under load. |
| Error and OOM rate | Detects a configuration that looks fast only when requests succeed. |
| Accepted-result rate | Converts raw speed into useful work. |
The key comparison for vLLM quantization is not “quantized versus unquantized” in the abstract. It is “this checkpoint and kernel on this GPU, at this workload, versus our baseline.” Publish the conditions with any result. If your test lacks a quality check, call it a performance-only experiment.
Common failure modes
The model will not load. Check checkpoint metadata, quantization method, vLLM version, GPU capability, and whether the method supports every layer used by the architecture. The vLLM method matrix should be checked before you change command flags blindly.
Memory savings did not improve latency. A smaller weight footprint does not guarantee a faster kernel. Test at your target concurrency and inspect time to first token, decode speed, and GPU utilization. Larger KV-cache headroom may be the real benefit.
A low-bit checkpoint looks fast but fails quality tests. Keep a production acceptance set and measure the percentage of outputs that meet your application criteria. A rollback threshold should be set before the canary.
The benchmark cannot be reproduced. Record model and checkpoint revisions, vLLM and driver versions, GPU SKU, dataset, concurrency, prompt lengths, output lengths, and command arguments. Without these, a throughput number is marketing rather than engineering evidence.
When to use Flatkey Cloud GPU
If you need to run your own vLLM server and compare quantized checkpoints, choose GPU capacity based on a measured memory and concurrency requirement. Flatkey provides NVIDIA GPU rental options through Cloud GPU Compute. Confirm the available SKU, deployment region, capacity, pricing, and service terms before committing. A vLLM quantization test belongs on the configuration you actually intend to rent. If you need a managed model API instead of operating vLLM, evaluate Flatkey’s API route and model directory separately; that is a different operating model.
For adjacent deployment decisions, see our vLLM deployment guide, vLLM multi-GPU inference guide, and vLLM versus Ollama comparison.
FAQ
Does vLLM quantization always make inference faster?
No. It can reduce weight memory, but latency and throughput depend on the checkpoint, kernel, GPU, sequence lengths, and request load. Benchmark a real workload against a baseline.
Can I use AWQ on an A100 or H100?
The current vLLM compatibility matrix lists AWQ support for Ampere and Hopper, which include A100 and H100. Check the exact checkpoint, current vLLM version, and model architecture before deployment.
Is FP8 supported on every NVIDIA GPU?
No. Support varies by quantization implementation and architecture.
The current matrix distinguishes methods such as
llm-compressor FP8 and Marlin; verify the exact row for
your hardware.
Should I quantize before or after selecting a GPU?
Size the workload and quality target first, then test candidate checkpoints on compatible GPUs. The format and hardware choice should be evaluated together.
