Continuous batching lets a vLLM server admit a new request while other requests are still generating tokens. A fixed batch waits for all members to finish before it can form the next batch; continuous batching revisits the active set at each scheduling step. That distinction matters when prompts and outputs have different lengths. Short requests can join the GPU work without waiting for the longest response in a previous group.
For an inference team, the useful question is not whether continuous batching is enabled in a feature list. It is whether your actual request mix gains completed responses per second without pushing time to first token, queue time, or error rate beyond your service target. This guide explains the mechanism, the limits that matter, and a benchmark you can run before choosing GPU capacity.
How vLLM continuous batching works
In vLLM's current V1 architecture, the scheduler fills a token budget at each step from running and waiting requests. A request tracks how many tokens have been computed; the scheduler assigns work to close that gap. The same loop handles chunked prefill, prefix caching, and speculative decoding. This is a more accurate mental model than a diagram of one static batch that runs to completion. See the vLLM architecture overview for the scheduler description.
Imagine three arrivals: a short chat response, a long document prompt, and another short chat. With a fixed batch, the second short chat might wait for the long request's batch boundary. With continuous batching, it can be considered at a later scheduling step while the first requests remain active. Admission is still constrained by the token budget, sequence limits, and available KV-cache space. Continuous batching does not guarantee zero queueing or a faster result for every request.
PagedAttention addresses a different part of the problem: storing and managing attention KV cache in blocks. It helps the engine use memory more flexibly, while the scheduler decides which requests receive compute. Keeping those roles separate helps you diagnose whether low throughput is caused by scheduling, KV-cache pressure, long prompts, or an upstream bottleneck. The vLLM PagedAttention design describes the memory mechanism.
The controls that shape a batch
| Control | What it bounds | What to watch when you change it |
|---|---|---|
max_num_batched_tokens |
Tokens processed in one scheduler iteration | Throughput, time to first token, inter-token latency, and long-prompt progress |
max_num_seqs |
Sequences processed in one iteration | Running requests, memory headroom, and tail latency |
| Chunked prefill | Splits a long prefill across scheduling steps | Long-prompt latency and interference with decoding requests |
| Queue or admission limit | Requests allowed to wait or run | Queue time, rejection rate, and whether overload reaches callers |
| KV-cache capacity | Active context that fits on the GPU | Preemption, recomputation, and out-of-memory failures |
These settings interact. Raising a token budget can improve throughput for one workload and worsen interactive latency for another. Increasing the number of admitted sequences can raise concurrency but also consume more KV cache. Version, model architecture, context lengths, and hardware all affect the result. Read the scheduler configuration reference for the behavior of your installed vLLM release, then change one control at a time.
When KV cache cannot accommodate new work, vLLM may preempt a running request and later recompute its progress. A rising preemption count is a warning that simply adding more arrivals to the batch is not free. The architecture overview explains the scheduling and preemption rules.
A benchmark that matches production traffic
Do not compare continuous batching settings with a single short prompt fired as fast as the client can send it. Build three request groups from your own logs or a privacy-safe trace:
- Short interactive chats with brief prompts and responses.
- Long-context requests with large prefills and moderate output lengths.
- Agent or RAG requests with repeated prefixes, varied output lengths, cancellations, and bursts.
Keep the model revision, tokenizer, GPU type, vLLM version, and
request trace fixed between runs. Sweep offered request rate from below
expected load through the point where queue time or errors become
unacceptable. The official vllm bench serve
guide documents request-rate and dataset controls. Record the exact
command, configuration, and trace version so the comparison can be
repeated.
For each run, record completed requests per second, output tokens per second, p50 and p95 time to first token, p95 inter-token latency, p95 end-to-end latency, queue time, failures, and GPU memory high-water mark. vLLM's production metrics reference includes gauges for running and waiting requests and histograms for queue time, time to first token, and preemptions. A useful scorecard is:
| Run | Offered requests/s | Completed requests/s | p95 first token | p95 queue | p95 end-to-end | Error rate | Preemptions |
|---|---|---|---|---|---|---|---|
| Baseline | measured | measured | measured | measured | measured | measured | measured |
| Token-budget change | measured | measured | measured | measured | measured | measured | measured |
| Sequence-limit change | measured | measured | measured | measured | measured | measured | measured |
Fill this table with measurements; there is no universal percentage gain to copy from another model or provider. Choose the configuration that meets latency and reliability targets at the lowest cost per accepted response, not the one with the highest token throughput in isolation.
Diagnose a disappointing result
If continuous batching appears ineffective, look for the constraint before changing another flag.
Waiting requests rise while GPU memory remains comfortable. Check admission limits, a small sequence cap, CPU tokenization, and client-side concurrency. A scheduler cannot fill a GPU with requests that never reach it.
Time to first token rises on long prompts. Separate prefill-heavy traffic from decode-heavy traffic in the benchmark. Review chunked prefill and token-budget settings, and compare the long-prompt distribution rather than only the average.
Throughput improves but p95 latency worsens. The server may be accepting more concurrent work than your interactive service target allows. Limit admission, use a queue timeout or a separate serving pool, and retest at the same offered load.
Preemptions or out-of-memory events rise. The active contexts may not fit the GPU's KV cache. Reduce concurrency or context limits, select a GPU with sufficient memory, or revisit model and precision choices. Avoid treating a preempted request as a free scheduling event.
GPU utilization is low even when requests wait. Inspect CPU, network, storage, model loading, and client pacing before attributing the bottleneck to continuous batching. Our vLLM deployment guide covers the broader serving runbook; this article focuses on scheduler behavior and measurement.
Where GPU choice enters the decision
Continuous batching can improve how you use a GPU, but it cannot make a model fit memory it does not have. Estimate weights, KV cache under expected concurrency and context length, runtime overhead, and a safety margin. Then benchmark on the GPU configuration you intend to rent. An 80 GB GPU and a larger-memory GPU can produce different admission and preemption behavior for the same model and request trace.
Flatkey GPU Compute lists rental options including H100 SXM, H200, B200, B300, A100 80GB, and RTX 5090. Configuration, region, availability, and price must be confirmed for the selected rental. Bring a benchmark trace, target latency, model revision, and memory estimate when you compare options. That makes the capacity discussion about completed work rather than a headline GPU specification.
Practical decision checklist
Before committing a vLLM continuous batching configuration, answer these questions:
- Does the top traffic scenario meet p95 first-token and end-to-end latency targets?
- What happens at a burst above normal request rate: bounded queue, rejection, or a large latency spike?
- Are preemptions and KV-cache pressure stable at the intended context length?
- Is the result repeatable across the exact model revision and GPU shape you plan to run?
- Does cost per accepted response beat the alternative after retries and idle capacity are counted?
Frequently asked questions
Does vLLM continuous batching mean every request runs immediately?
No. A new arrival can be considered at a scheduling step, but it may still wait for token budget, a sequence slot, or KV-cache capacity. Watch the running and waiting gauges together with queue-time histograms before promising an interactive latency target.
Is vLLM continuous batching the same as chunked prefill?
No. Continuous batching describes how requests join and leave scheduled work over time. Chunked prefill splits a long prompt's prefill work so it can share the token budget across steps. They work together in the current scheduler, but turning one tuning knob does not substitute for measuring the other.
Should I maximize the batch size?
Not automatically. A larger batch may improve throughput while increasing queue time, time to first token, or preemption pressure. Run the same trace at the same offered load and compare the entire service target, including failures and cost per accepted response.
vLLM continuous batching is a scheduling method, not a benchmark result. The strongest setup is the one whose throughput, latency, memory behavior, and failure mode you can explain with your own measurements.
