A vLLM benchmark is useful only when it resembles the traffic your application will actually send. A headline tokens-per-second number can hide a slow first token, an overloaded queue, or a prompt distribution that never appears in production. This guide gives you a repeatable serving benchmark: choose a workload, sweep offered load, record the latency distribution and throughput together, and compare GPU configurations at the same quality target.
Quick answer: Use vllm bench serve
against a pinned vLLM version and model on a dedicated GPU server. Keep
the model, precision, prompt and output lengths, sampling settings,
cache state, and client location fixed. Run several request rates,
capture TTFT, TPOT, end-to-end latency, successful request rate, and
output tokens per second, then choose the least costly configuration
that meets your service objective at peak load. A single
maximum-throughput run is not a capacity plan.
What to measure in a vLLM benchmark
| Measure | What it tells you | Common mistake |
|---|---|---|
| Time to first token (TTFT) | Delay before the first streamed output reaches the benchmark client | Reporting only average latency while p95 or p99 users wait in a queue |
| Time per output token (TPOT) | Per-request decode time after the first token, averaged across requests | Treating it as identical to every inter-token gap under speculative decoding |
| End-to-end latency | Time from request submission to final output | Comparing runs with different output lengths |
| Successful requests per second | Completed work at an offered load | Counting requests that timed out or failed as capacity |
| Output tokens per second | Aggregate generated-token throughput | Dividing by GPU price without checking the latency target |
| GPU memory and utilization | Whether the chosen batch, KV cache, and model fit with headroom | Calling high utilization good when the queue and tail latency are already failing |
The vLLM benchmark documentation defines TTFT, inter-token latency, and TPOT by their measurement points. Use those definitions when comparing tools. In particular, client TTFT includes transport and frontend work; optional server-side metrics can separate scheduler waiting from model execution, but enabling them adds overhead. Keep diagnostic runs separate from headline results.
Pin a fair comparison before you run
Write down the model revision and tokenizer, vLLM version, GPU type and count, tensor-parallel size, serving flags, quantization format, context length, and software environment. Use the same prompts and output targets for every candidate. If you change the model, precision, and GPU at once, the result cannot tell you which change helped.
Choose at least two workloads. A synthetic fixed-length workload is useful for controlled sweeps; a sanitized trace of real request lengths is better for a production decision. Preserve the distribution of long prompts, short answers, and bursts. For a chat product, measure the chat endpoint with the corresponding benchmark backend. For a completions workload, use the completions backend. Do not compare their numbers as though request formatting were identical.
Decide the pass criteria before the run. For example, state a required successful request rate and maximum p95 TTFT and p95 end-to-end latency for your own workload. We do not publish a universal threshold: it depends on the user experience and output size. An acceptance table with the thresholds, measured results, and pass/fail reason is more useful than a single performance score.
Run a controlled serving test
The current vLLM CLI
guide lists vllm bench serve for online-serving
measurements and recommends the bench extra for its dependencies. The
following outline uses vLLM’s documented completions example. Replace
the model and dataset with assets you are licensed to run and that fit
your hardware. These commands are a reproducible template, not measured
Flatkey results.
In a second shell, after the server is ready:
vllm bench serve \
--backend vllm \
--model YOUR_MODEL_ID \
--endpoint /v1/completions \
--dataset-name sharegpt \
--dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \
--num-prompts 1000 \
--request-rate 1 \
--save-result \
--save-detailedThen repeat with a short series of higher --request-rate
values while keeping the other inputs fixed. Use enough prompts to avoid
interpreting a handful of requests as a tail-latency distribution. Save
raw results and the exact command for each run. The official online benchmark
example shows the output fields and explains how
--save-result and --save-detailed preserve
evidence. For chat requests, use the documented openai-chat
backend and /v1/chat/completions endpoint with a dataset
and model that support chat formatting.
Do not run this benchmark against a shared production endpoint without a traffic plan. It intentionally creates load. Use an isolated test deployment or a controlled maintenance window, and record any background work that may affect the result.
Sweep until you find the latency knee
Plot offered request rate on the horizontal axis. Plot completed requests per second, p95 TTFT, p95 end-to-end latency, and output tokens per second as separate series. At first, throughput may rise roughly with offered load. Beyond the sustainable region, the queue grows, tail latency rises, and successful throughput may flatten. That turning point matters more than the largest number printed in one run.
Keep a warm-up run separate. vLLM warns that repeating a benchmark against the same server can reuse cached prefixes and inflate throughput. If cache reuse is not part of the product workload, vary the seed or reset the server or cache according to the official benchmark guidance. If cache reuse is realistic, report the cache-hit rate and include a cold-cache comparison. A benchmark without a stated cache policy is difficult to reproduce.
When a run misses its target, inspect the sequence. A TTFT increase with a growing queue may point to saturation or admission control. Poor TPOT with stable queue time may point to decode, memory bandwidth, parallelism, or an inefficient precision choice. If failures rise, look at timeouts, OOMs, and server logs before adjusting model settings. Changing multiple variables at once hides the cause.
Compare GPU options by accepted work, not raw speed
A larger GPU can produce more output tokens per second yet cost more per accepted request. Calculate an approximate cost for each configuration:
GPU cost per 1,000 successful requests = hourly GPU cost ÷ successful requests per hour × 1,000
Use a current quote for the exact region and commitment, and include non-GPU infrastructure or minimum billing increments if they matter. Compare only runs that meet your latency and error targets. A cheaper configuration that fails p95 TTFT is not a cheaper way to deliver the same service.
For an initial cloud comparison, keep the model and serving configuration fixed while changing only the GPU shape. Then test a small number of tuning changes, such as batch and memory settings, one at a time. Capture utilization, VRAM headroom, queue depth, request success rate, and the benchmark artifact. Flatkey Compute can be one capacity option to evaluate; verify the offered GPU, region, and current quote before using it in a cost comparison. This guide makes no claim that a particular GPU is always faster or cheaper.
A vLLM benchmark result you can hand to a reviewer
For each run, record: model and revision; vLLM and driver versions; GPU and topology; command and dataset checksum; input/output length distribution; client location; cache policy; request rate; completed and failed requests; p50/p95/p99 TTFT; p50/p95/p99 end-to-end latency; output token throughput; GPU utilization and VRAM; hourly quote; cost per accepted request; and pass/fail against the predetermined target.
This is also the point to keep security and quality visible. Benchmark prompts should be scrubbed of customer data, and output quality should be checked when comparing quantized models or different model revisions. An apparent speed gain that changes task quality is not an apples-to-apples win.
Common questions
Is
vllm bench throughput the same as
vllm bench serve?
No. The first targets offline inference throughput; the second sends
online requests to a running server and measures client-facing serving
behavior. Use bench serve for latency and capacity planning
of an API deployment. The vLLM CLI reference lists
both commands.
Which number should I optimize first?
Start with the service objective: success rate and tail latency at expected peak traffic. Then minimize cost among configurations that pass. Aggregate token throughput is useful for diagnosing capacity, but it cannot replace a user-facing latency target.
Can I compare published benchmark numbers from different providers?
Only if the model, precision, workload, output length, endpoint, cache policy, request-rate regime, and measurement definitions align. In most cases, rerunning the same workload on both environments gives a more defensible comparison.
Next step: take one representative trace, fix the pass criteria, run a short rate sweep on two GPU configurations, and keep the full results with the decision. If you are also changing precision, use the separate vLLM quantization guide to evaluate memory and quality before attributing any speed difference to the GPU.
