Long-context serving eventually turns GPU memory into a scheduling problem. A request may finish its first turn but leave behind KV state that is expensive to recompute and not needed on the GPU right now. KV cache offload is the technique of moving reusable cache blocks from the fast GPU tier to a larger, slower tier—usually host RAM first—then promoting them back only when a later request can reuse them.
The goal is not to make slow storage look like HBM. The goal is to reserve expensive GPU KV capacity for the active working set while keeping high-value prefixes available elsewhere. That makes KV cache offload most useful when your traffic has long contexts and meaningful reuse: recurring system prompts, RAG scaffolds, multi-turn sessions, or repeated document prefixes.
This guide shows how to decide whether tiering is appropriate, how the current vLLM controls fit together, what TensorRT-LLM exposes for host-memory offload, and how to test the change without trading a GPU out-of-memory failure for a tail-latency incident.
The quick answer
Use KV cache offload as a controlled extension of prefix reuse—not as a default memory fix.
| Traffic pattern | Start with | Why |
|---|---|---|
| Repeated long system prompts or RAG prefixes | GPU prefix caching, then CPU offload pilot | A later request can reuse a cached prefix instead of recomputing it. |
| Long-lived interactive sessions with inactive periods | GPU + CPU warm tier | Inactive context can leave GPU memory while remaining recoverable. |
| Mostly one-shot, unique prompts | GPU capacity and admission control first | A cache block with no future reuse only adds transfer work when offloaded. |
| Tight GPU memory, but host RAM is available | Small CPU offload experiment | Offload can create room for active KV blocks, but promotion cost must stay inside the SLO. |
| Need a durable lower tier | CPU primary tier plus a tested secondary tier | A secondary tier can extend retention, but it is not a substitute for a fast active tier. |
The important constraint is locality. When an offloaded block is reused, it must return to GPU memory before attention uses it. That transfer path is part of the request’s latency budget.
What a KV cache tier is actually storing
During autoregressive generation, attention layers produce key and value tensors for processed tokens. Retaining those tensors lets the server avoid recomputing the already-seen prefix on a later decoding step or on a compatible prefix reuse. In TensorRT-LLM, the KV cache is represented as pools of blocks that hold state for a fixed token count; its documentation also describes reuse across requests, prioritized retention, and host-memory offload. TensorRT-LLM KV Cache System
A practical serving stack usually has three possible storage temperatures:
| Tier | Typical role | Strength | Cost |
|---|---|---|---|
| GPU HBM | Hot, active prefill and decode state | Lowest access latency | Smallest and most expensive capacity |
| CPU DRAM | Warm reusable blocks | Much larger capacity; can act as the first offload target | Promotion to GPU adds transfer time |
| Filesystem or object tier | Cold retained blocks | Durable or shared capacity | Highest retrieval cost and more operational failure modes |
The vLLM offloading guide is explicit about the ordering: its multi-tier design uses CPU as the primary tier, and transfers between GPU and a secondary tier are staged through that CPU tier. In other words, a cold-tier hit is not a direct GPU read. vLLM KV Offloading Usage Guide
promote on a usable cache hit
Request ──> GPU HBM <──────────────────── CPU DRAM
│ ▲
│ evict completed, reusable │ stage
▼ │
active GPU cache ─────────────> secondary storage
hot warm / cold
This vllm kv cache offload diagram is deliberately simple: only the GPU tier should sit on the critical compute path all the time. CPU and storage tiers should serve a measurable reuse policy.
KV cache offload is not model-weight offload
This distinction prevents a surprisingly common misconfiguration:
- KV cache offload moves attention state created by requests. It primarily changes the memory available for active context and the reuse path.
- Model-weight offload moves model parameters. In vLLM, the
--cpu-offload-gbcontrol is documented under model weight offloading and can add CPU–GPU transfer work to every model forward pass. vLLM serve arguments
Do not assume that a setting which lets a model load also creates more usable KV capacity. First separate the model-weight budget, runtime workspace, and KV-cache budget; then decide whether inactive KV blocks are worth retaining outside GPU memory.
vLLM KV cache offload: start with the native CPU path
For the current vLLM CLI, the relevant vllm kv cache offload flag is --kv-offloading-size. The official CLI reference says it is the KV offload buffer size in GiB; when tensor parallelism is greater than one, the size is the total across tensor-parallel ranks. Offloading is disabled when it is unset. The companion --kv-offloading-backend supports the native CPU backend and LMCache. vLLM serve arguments
Use a bounded pilot rather than sizing the host tier to all available RAM:
vllm serve <your-model> \
--gpu-memory-utilization 0.85 \
--kv-offloading-size 16 \
--kv-offloading-backend native
The example is an experiment shape, not a universal setting. The right value depends on the model’s cache layout, context length, tensor-parallel configuration, host-RAM headroom, request mix, and latency objective. Pin the vLLM image version and run vllm serve --help in that image before taking flags into production; serving configuration evolves quickly.
The native path suits a first vllm kv cache offload cpu trial because it keeps the tiering question narrow: can a CPU warm tier preserve enough useful blocks to improve admission or avoid GPU cache churn without worsening p95 and p99 behavior?
Choose cache admission before cache capacity
More retained blocks do not necessarily mean more useful blocks. Before increasing the offload buffer, answer four questions:
- Which prefixes are shared across requests?
- How long after creation are those prefixes reused?
- How large is a useful block relative to your host transfer budget?
- Which blocks should be evicted outright because their chance of reuse is too low?
If the workload cannot answer the first two questions from traces, start by measuring prefix reuse on GPU. A CPU tier can preserve warm cache state; it cannot manufacture reuse for entirely unique prompts.
Build a vLLM tiering path only after the CPU pilot
vLLM’s OffloadingConnector is the path to a CPU primary tier plus optional secondary tiers. The official guide describes CPUOffloadingSpec for a single CPU tier and TieringOffloadingSpec for a CPU primary tier with one or more secondary tiers. It also documents LRU and ARC as built-in eviction-policy options. vLLM KV Offloading Usage Guide
The following is a faithful configuration shape from the documented multi-tier example. Replace the model, paths, block size, bytes, and thread counts only after testing them on your own host:
vllm serve <your-model> \
--kv-transfer-config '{
"kv_connector": "OffloadingConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"spec_name": "TieringOffloadingSpec",
"cpu_bytes_to_use": 10737418240,
"block_size": 16,
"eviction_policy": "lru",
"secondary_tiers": [
{
"type": "fs",
"root_dir": "/mnt/kv_cache",
"n_read_threads": 32,
"n_write_threads": 16
}
]
}
}'
Three operational details matter more than copying the snippet:
cpu_bytes_to_useis the total CPU-tier reservation across workers, not a vague suggestion to the operating system. Budget for the process, the OS, storage page cache, and any colocated services.- The configured block or chunk size sets the I/O unit. Very small chunks can raise metadata and scheduling overhead; very large chunks can promote bytes that a request does not need. Test both with real prefix lengths.
- A secondary tier is downstream of CPU. If the CPU tier is too small or saturated, adding filesystem capacity may retain more data without improving the latency of a later hit.
The goal of vllm kv cache offload is therefore not “retain everything.” It is “retain the blocks with a plausible, timely reuse path.”
TensorRT-LLM KV cache offload to host memory
TensorRT-LLM exposes the same basic idea through its KV-cache configuration. Its documentation says host_cache_size allocates host memory for offloading, defaults to zero, and lets a block remain reusable until it is later evicted from host memory. On reuse, the block is copied back to GPU memory. TensorRT-LLM KV Cache System
For a tensorrt-llm kv cache offload host memory rollout, pay attention to retention policy as well as capacity:
| TensorRT-LLM control | What to verify |
|---|---|
host_cache_size |
The byte budget is intentional and leaves host-memory headroom. |
| Block priority / retention policy | Frequently reused prompt regions are retained ahead of disposable data. |
secondary_offload_min_priority |
Low-priority blocks can bypass host offload and be evicted directly, reducing avoidable host traffic. |
| Cross-request block reuse | Reuse remains enabled and is observable separately from scheduler estimates. |
TensorRT-LLM documents a default secondary_offload_min_priority of 35 at the time of writing. Treat that as a versioned default, not a policy recommendation: a threshold should reflect observed reuse value and transfer cost for your service.
A rollout method that protects latency
Do not turn on a cold tier during a capacity incident and declare success when the OOM stops. Instead, run a controlled sequence:
- Capture a GPU-only baseline. Record context-length distribution, concurrency, time to first token, inter-token latency, end-to-end p50/p95/p99, prefix-reuse rate, and GPU KV eviction behavior.
- Reserve real host headroom. Include runtime overhead and the operating system; do not size a CPU cache from free-memory snapshots alone.
- Enable a small CPU tier. Keep model, batch policy, traffic shape, and SLO alerts stable while you change one cache dimension.
- Compare hit quality, not just hit count. Separate GPU hits, CPU promotions, cold-tier promotions, and misses that trigger recomputation.
- Examine tail latency under contention. Run mixed long/short requests and a concurrency ramp. A mean-only result hides transfer queues.
- Add a secondary tier only if retention value justifies it. Test cache loss, storage pressure, restart behavior, and promotion bursts before rolling it out broadly.
- Set a rollback rule. For example: disable the newest tier when p99 time to first token or inter-token latency breaches the agreed budget for a sustained window.
The vLLM guide documents configuration and offload metrics, including configuration information and CPU-cache usage signals. Use them with application traces; a cache metric without request-level latency makes it hard to distinguish a valuable promotion from transfer thrash. vLLM KV Offloading Usage Guide
Failure modes to design out
| Failure mode | What it looks like | Better response |
|---|---|---|
| Treating all cache blocks as valuable | CPU tier fills, promotions rise, latency worsens | Admit and retain by reuse evidence, not creation order alone. |
| Confusing weight offload with KV cache offload | Model loads, but long-context concurrency remains poor | Separate weight, KV, workspace, and safety-headroom budgets. |
| Overcommitting host RAM | The host starts reclaiming memory or colocated services destabilize | Reserve a fixed budget and load-test under realistic contention. |
| Adding a secondary tier too early | More retained bytes, no meaningful SLO improvement | Prove CPU-tier reuse first; then test the cold tier independently. |
| Ignoring transfer topology | Good lab result, poor shared-node p99 | Benchmark on the exact GPU, CPU, PCIe/NVLink topology, and storage path used in production. |
| Hiding cache behavior from operators | An incident looks like random latency variance | Expose tier-specific hit, promotion, eviction, capacity, and error signals. |
When KV cache offload is the wrong optimization
Skip KV cache offload for now when:
- GPU memory is not the limiter.
- Prompts are nearly all unique, so retained prefixes do not return.
- Your primary SLO is very low tail latency and the transfer path cannot be shown to fit inside it.
- Host DRAM or storage is already a contested production resource.
- Prefix caching, batching, context limits, model selection, or cache quantization have not yet been evaluated.
In those cases, the cleaner first step may be a GPU-memory budget review, a different context policy, or a serving architecture change. For topology and parallelism choices, see vLLM Multi-GPU Inference, vLLM vs SGLang, and TensorRT-LLM vs vLLM.
FAQ
Does KV cache offload reduce GPU memory use?
It can reduce pressure from reusable inactive KV blocks by moving them out of the GPU tier. It does not remove the need for GPU-resident cache for active attention work, and a reused offloaded block must be promoted back before use.
What is the vLLM KV cache offload flag?
In current vLLM CLI documentation, --kv-offloading-size enables a KV offload buffer when set, while --kv-offloading-backend selects the backend. Verify the exact flags in your pinned vLLM release. vLLM serve arguments
Is a filesystem tier a direct GPU cache?
No. vLLM documents CPU as the primary tier for its multi-tier implementation; transfers between GPU and a secondary tier are staged through CPU memory. That is why secondary-tier tests must include the full promotion path.
Is KV cache offload better than buying a larger GPU?
They solve different constraints. A larger GPU can increase the hot working set without promotion overhead. Offload can retain useful warm or cold state with existing GPU capacity. Compare both against the workload’s measured reuse, latency SLO, operational burden, and total serving cost.
Conclusion: treat KV cache offload as a cache policy, not a capacity trick
The strongest KV cache offload design keeps active state on GPU, reserves CPU memory for evidence-backed warm reuse, and adds a cold tier only when retention remains valuable after transfer cost and failure handling are measured. Start with a bounded CPU pilot, instrument promotions and latency by tier, and promote only the policy that improves a production SLO—not merely the amount of cache retained.
If the operational cost of a self-hosted serving stack outweighs the control your workload needs, compare a GPU cloud deployment with a model API or explore Flatkey Compute before committing to a larger fleet.
