Speculative Decoding for LLM Inference: Throughput Gains and Tradeoffs
Speculative decoding LLM serving can lower the time spent waiting for the next token—without asking the target model to become a different model. Instead of letting the large target model produce only one token per decode step, a cheaper proposer drafts a short continuation. The target model verifies that continuation in parallel, commits the accepted prefix, and continues from the first disagreement.
That sounds like a universal accelerator. It is not. The outcome depends on draft quality, request mix, batching, sampling settings, GPU memory headroom, and whether your workload is truly decode-bound. This guide explains what is happening under the hood, how EAGLE and Medusa change the design, and how to decide whether speculative decoding belongs in your serving stack.
The practical rule: treat speculative decoding as a measured serving optimization—not a paper-speedup setting. Keep it only when it improves the latency or output-token throughput target that matters for your actual traffic without reducing concurrency or reliability.
What is speculative decoding LLM inference?
Autoregressive generation is sequential: token t + 1 depends on the tokens before it. A normal decoder therefore waits for a large target model to run once per generated token. Speculative decoding changes the schedule, not the target distribution: a smaller or specialized draft mechanism predicts several candidate tokens; the target model scores or verifies the candidate positions together; the system accepts the matching prefix and falls back at the first rejected position.
The original speculative-decoding paper describes this as a way to obtain exact samples from the target model while evaluating several candidate positions in parallel. Its reported 2–3× result was a benchmark result for its T5 setup, not a promise for every model or service. Read the original paper.
A speculative decoding LLM inference diagram
request + current context
|
v
draft mechanism proposes: [A] [B] [C] [D]
|
v
target model verifies all proposed positions in one pass
|
+-- accepts [A] [B] [C] -> commit that prefix (and possibly one extra target token)
|
+-- rejects at [C] -> commit [A] [B], sample/correct at [C], then repeat
The work saved is not the draft computation itself. The potential win comes from replacing several serial target-model decode iterations with one target-model verification pass. If the proposal is often wrong, the target still has to recover frequently and the gain shrinks.
The metric that matters: accepted tokens per target verification
The most useful health signal is the number of proposed tokens that survive verification. Teams often call this acceptance rate or accepted length. Track both:
| Metric | What it tells you | Why it can mislead alone |
|---|---|---|
| Acceptance rate | What share of proposed tokens the target accepts | It does not include draft cost or target verification cost. |
| Mean accepted length | How many committed draft tokens you get per round | Averages can hide long-tail requests and different sampling modes. |
| Inter-token latency (ITL) | Time between streamed tokens | This is usually the user-visible decode metric. |
| Aggregate output tokens/second | Useful serving capacity across concurrent requests | It can improve while a low-QPS interactive user sees little benefit, or the reverse. |
| GPU memory and active-request count | Whether the proposer crowds out KV cache or concurrency | A faster single request can be a worse fleet-level configuration. |
For a fair A/B test, hold model revision, tokenizer, quantization, maximum output length, prompt mix, sampling parameters, hardware, and concurrency policy steady. Compare end-to-end latency, time to first token, p50/p95 ITL, output tokens/second, and error rate. A single “tokens/sec” number without its load shape is not enough to choose a configuration.
Where speculative decoding helps—and where it does not
Speculative decoding LLM serving is usually most attractive when decoding is memory-bound and each request has enough generation length for repeated verification rounds to matter. Current vLLM guidance specifically frames it as a way to reduce inter-token latency for medium-to-low-QPS, memory-bound workloads and notes that real gains depend on model family, traffic, hardware, and sampling settings. See the vLLM speculative-decoding guide.
It is less compelling in these cases:
- Prefill dominates the request. Long prompts with short answers spend most of their time ingesting context. Drafting changes decode work, not the prompt-prefill work that already dominates end-to-end latency.
- Acceptance is poor. Creative decoding, a mismatched draft model, or a configuration that produces weak candidate continuations can cause repeated verification failures.
- Your server is already batch-saturated. At high concurrency, the target GPU may be efficiently occupied by the normal scheduler. Adding a drafter can consume memory or compute that would otherwise serve more requests.
- KV cache headroom is tight. An extra draft model or proposer state may reduce the number of live sequences before eviction or queueing begins. Review LLM GPU memory requirements before interpreting a single-request speedup as a fleet win.
- The target model is not compatible with your chosen proposer. Model-based methods require compatible artifacts, tokenization, and serving support; never assume a small model is a valid draft model just because it is cheaper.
The right question is therefore not “Can speculative decoding make this model faster?” It is “Which service-level constraint does it improve under our traffic distribution?”
Draft models, EAGLE, and Medusa: three ways to propose tokens
Separate draft model
The classic design pairs a fast draft model with a larger target model. The draft proposes k tokens; the target verifies them. This can be simple conceptually, but it adds another model artifact, memory footprint, compatibility work, and an operational surface to version and monitor.
Use it when you have a demonstrably compatible, low-latency proposer and enough GPU capacity to preserve the target service’s concurrency. Start with a small set of speculative lengths—rather than assuming that a deeper proposal is always better.
EAGLE speculative decoding LLM
EAGLE is a model-based approach that drafts at a feature level rather than using only a conventional small-token-model proposal path. The paper describes conditioning its feature prediction with a one-step-ahead token sequence to address feature-level uncertainty. In the paper’s tested settings, it reports improved latency and throughput while preserving the generated-text distribution; that is useful evidence of the method’s potential, not a capacity forecast for another model or GPU fleet. Read the EAGLE paper.
For operators, the implication is straightforward: EAGLE can be attractive when supported artifacts are available for the target family, but it is not a generic switch for every checkpoint. Validate the exact target/proposer pairing and serving-engine version before promoting it.
Medusa speculative decoding LLM
Medusa puts multiple decoding heads on a backbone model and uses tree-based attention to create and verify several candidate continuations. That avoids maintaining a separate fully independent draft model, but the heads must be trained or fine-tuned for the backbone. The Medusa paper distinguishes a frozen-backbone path from joint fine-tuning and reports benchmark speedups for those experimental setups. Read the Medusa paper.
The operational tradeoff is different from a draft-model pair: potentially less separate-model management, but a model-specific artifact and training or integration requirement. Treat it as a model-serving choice, not a runtime trick.
A practical rollout plan for GPU inference
1. Establish a representative baseline
Capture at least one production-like slice of requests: prompt-length bands, output-length bands, language or task type, temperature/top-p settings, streaming behavior, and concurrency levels. Record the baseline separately for time to first token and decode. If the workload is prompt-heavy, do not let an output-token benchmark obscure a prefill problem.
2. Pick one proposer strategy and a conservative depth
Start with one supported method—draft model, EAGLE, Medusa, or another engine-supported proposer—and a small candidate length. vLLM, for example, documents --speculative-config settings such as a method, a proposer model, and num_speculative_tokens; consult the version-matched documentation before copying any configuration into production. For broader operational setup, see the vLLM deployment guide. Do not tune speculative depth and batching at the same time on the first pass.
3. Run a load matrix, not one benchmark
Test the same traffic slice at low, medium, and peak-like concurrency. For each point, compare:
- Baseline vs. speculative end-to-end latency.
- p50 and p95 inter-token latency for streaming requests.
- Aggregate output tokens/second and completed requests/second.
- Accepted length, rejection frequency, and proposer latency.
- GPU memory, KV-cache pressure, queue delay, and error/timeout rate.
If possible, split results by prompt band and output band. A configuration that helps long code completions may not help short chat answers, and a good offline result can degrade after queueing is included.
4. Set guardrails before the canary
Define rollback thresholds before traffic sees the change: for example, a regression in p95 ITL, a drop in completed requests/second at peak load, an increase in out-of-memory events, or a persistent acceptance-rate collapse. Route only an eligible slice at first and preserve a baseline path for fast comparison.
5. Keep measuring after launch
Acceptance shifts with model updates, prompt distributions, tool-use formats, and sampling defaults. Re-run the comparison after any material target-model, tokenizer, quantization, GPU, serving-engine, or traffic-profile change.
A minimal vLLM starting point
For a serving engine that supports it, the configuration should state the proposer method rather than hiding it in an opaque default. The current vLLM documentation uses a structure like this for a separate draft model:
vllm serve <target-model> \
--speculative-config '{
"method": "draft_model",
"model": "<compatible-draft-model>",
"num_speculative_tokens": 5
}'
The placeholders are deliberate. The compatible model, number of speculative tokens, parallelism settings, and memory budget must be chosen from the artifacts and engine version you have tested. Do not copy this snippet as evidence that a particular target/draft pair is supported.
Throughput gains and tradeoffs at a glance
| Decision | Likely upside | Main cost or failure mode | What to measure first |
|---|---|---|---|
| Enable a separate draft model | Fewer serial target decode steps when acceptance is high | Extra model memory, compatibility, and draft latency | Accepted length, KV-cache headroom, p95 ITL |
| Increase speculative length | More potential tokens per verification | More wasted draft work when later tokens are rejected | Net ITL and aggregate output TPS, not acceptance alone |
| Use EAGLE | Strong model-based proposal path where artifacts are supported | Model-family and engine compatibility requirements | Target/proposer compatibility and load-shaped benchmark results |
| Use Medusa | Multi-head proposal without a separate full draft model | Model-specific heads and integration/training work | Output quality checks, serving support, and rollback path |
| Turn it on at peak QPS | Possible throughput benefit in the right engine/workload | Scheduler, memory, or compute contention can erase the gain | Completed requests/sec, queue time, and error rate |
FAQ
Does speculative decoding change the answer from the target model?
Proper speculative sampling uses target-model verification and correction so that sampling can preserve the target distribution. The guarantee depends on using a correct algorithm and compatible implementation. A serving change can still affect operational behavior—such as latency, queueing, or errors—so evaluate quality and reliability as well as speed.
Does speculative decoding improve time to first token?
Usually, it targets the decode phase after generation begins. If prompt processing dominates your requests, time to first token may change little. Measure prefill and decode separately.
How many speculative tokens should I use?
There is no universal best value. Test a small range against your model, proposer, sampling settings, GPU type, and concurrency. A longer proposal only wins when enough of its tokens are accepted and its extra work does not reduce service capacity.
Should I use EAGLE or Medusa?
Choose the method your target model and serving engine support well, then benchmark it on your workload. EAGLE relies on supported feature-level drafting artifacts; Medusa relies on multiple heads associated with a backbone. Neither is automatically interchangeable with a generic draft model.
Put the decision in context
Speculative decoding is one layer of LLM-serving optimization. It works best alongside a clear model-serving choice, realistic capacity limits, and explicit monitoring. For a broader serving-engine comparison, see vLLM vs SGLang: Throughput, Latency, Model Support, and Operations. For the capacity side of production APIs, see RPM vs TPM: LLM Rate Limits and 429 Fixes.
If you are planning GPU experiments or a new inference environment, explore Flatkey Compute and start with a benchmark plan that reports both user-visible latency and fleet-level throughput. The best speculative decoding LLM configuration is the one that improves your constrained metric under real load—and can be rolled back safely when the workload changes. This speculative decoding LLM guide is a starting point for that measurement discipline, not a substitute for it.
Sources
- Leviathan, Kalman, and Matias — Fast Inference from Transformers via Speculative Decoding
- Cai et al. — Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- Li et al. — EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- vLLM documentation — Speculative Decoding
