SGLang Prefill Decode Disaggregation: Architecture and Deployment Guide

SGLang Prefill Decode Disaggregation: Architecture and Deployment Guide

Last verified: October 16, 2026. SGLang flags and supported feature combinations change quickly; pin a tested SGLang image or commit before applying this guide in production.

SGLang prefill decode disaggregation separates the two very different jobs inside LLM serving: prompt processing runs on a prefill pool, while token generation runs on a decode pool. The pools exchange the request's KV cache, and a router presents one inference endpoint to the client.

That separation can protect inter-token latency when long prompts arrive, and it lets you scale compute-heavy prefill capacity differently from memory-bound decode capacity. It also adds a transfer path, another scheduler boundary, and a larger compatibility surface. Treat it as a measured serving architecture—not a switch that automatically makes every workload faster.

This guide explains the SGLang prefill decode disaggregation architecture, a minimal deployment shape, what to measure, and the conditions that should stop a rollout.

The quick answer: when PD disaggregation earns its complexity

SGLang's upstream documentation describes prefill as computation-intensive and decode as KV-cache- and memory-intensive. In a unified scheduler, incoming prefill work can interrupt decode work; with data-parallel attention, different workers can also become imbalanced between the two phases. PD disaggregation separates those phases so each can be scheduled and provisioned independently. SGLang's PD documentation is the source of truth for the current supported combinations.

Workload signal	Is SGLang prefill decode disaggregation a good candidate?	Why
Long or highly variable prompts, with users sensitive to streaming smoothness	Often	Long prefills are less likely to interrupt an active decode pool.
Decode memory pressure differs materially from prompt-compute pressure	Often	You can size pools and concurrency limits by their actual bottleneck.
A multi-node GPU deployment already has a validated low-latency KV-transfer fabric	Often	The additional transfer boundary is operationally realistic.
One small GPU server, short uniform prompts, or low traffic	Usually not first	A unified server is simpler and may avoid transfer overhead.
A workload that depends on process-local response history or built-in server tools	No, until validated	Current upstream PD documentation lists stateful Response API workflows and built-in tools among unsupported workflows.

The decision is not “prefill fast, decode slow.” It is whether isolating their queues and hardware limits improves the user-visible objective you care about—usually time to first token (TTFT), inter-token latency (ITL), tail latency, or completed requests under a fixed capacity budget.

How prefill decode disaggregation works in SGLang

The SGLang PD topology has three logical roles:

1. Router: receives the client request, selects a prefill and decode path, and streams the merged response back to the client.
2. Prefill workers: tokenize and process the full prompt, then produce the KV-cache state needed to continue generation.
3. Decode workers: receive that KV state and generate output tokens until the request completes.

The handoff is the important part. The prefill worker does not send text to a fresh decoder and make it repeat the prompt. It transfers the KV-cache state so decode can continue from the same attention context. SGLang currently documents Mooncake and NIXL as PD transfer engines; choose the transport and network path that your pinned version and hardware actually support.

client
  │
  ▼
PD router ──► prefill pool ──► KV-cache transfer ──► decode pool ──► streamed tokens
                 compute-heavy          fabric            memory-sensitive

In a multi-node cluster, each role can have its own replica count, GPU shape, queue limit, autoscaling policy, and failure budget. That is useful only if the router and KV-transfer fabric are treated as production dependencies rather than invisible glue.

Before you deploy: the five checks that prevent misleading results

1. Pin the exact SGLang release and model configuration

Run the same tested SGLang image or commit across the router, prefill workers, and decode workers. Keep model revision, tokenizer, quantization, attention backend, parallelism strategy, and model-specific flags explicit in your manifests. A command that worked on one version can be invalid or semantically different on another.

2. Prove the KV-transfer path before scaling traffic

PD is a data-plane architecture. Validate the NIC or interconnect mapping, container device exposure, host limits, and transfer backend before interpreting model latency. Mooncake-specific deployments can require InfiniBand/RDMA configuration; the SGLang docs also document NIXL backend selection for compatible infrastructure. Do not infer that a fast collective-communication path automatically proves a healthy KV-transfer path.

3. Establish a unified-server baseline

Use the same model, prompts, generation lengths, concurrency mix, sampling settings, and hardware budget for the baseline and PD test. Measure at least p50/p95 TTFT, p50/p95 ITL, request throughput, cancellation behavior, and error rate. A higher GPU utilization number alone is not a serving outcome.

4. Define the role boundary in your scheduler

Keep prefill and decode pods, services, logs, and alerts separately addressable. If you cannot tell which role is queueing, out of KV capacity, or waiting for transfer, the architecture will be harder to operate than a unified fleet.

5. Read the current compatibility constraints—not just the quick-start block

The SGLang prefill decode disaggregation docs include restrictions for specific policies and stateful workflows. For example, the documented prefill_complete decode-allocation policy has a narrower support envelope than the default policy. Validate every advanced feature combination in a staging run before making it part of an availability promise.

A minimal SGLang prefill decode disaggregation deployment

Start with one prefill worker, one decode worker, and one router on a two-GPU host. This is a functional acceptance test, not a claim that two GPUs are a production shape. The following NIXL example follows the current upstream single-node pattern; use the exact command syntax from the documentation for your pinned SGLang version.

# GPU 0: prefill role
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode prefill \
  --disaggregation-transfer-backend nixl \
  --port 30000

# GPU 1: decode role
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend nixl \
  --base-gpu-id 1 \
  --port 30001

# Client-facing PD router
python -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill http://127.0.0.1:30000 \
  --decode http://127.0.0.1:30001 \
  --host 0.0.0.0 \
  --port 8000

Send a short request through the router rather than directly to either worker:

curl http://127.0.0.1:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "prompt": "Explain why KV-cache handoff matters in one sentence.",
    "max_tokens": 64
  }'

Success means more than receiving text. Confirm that the router selected both roles, the KV transfer completed, streaming behavior is intact, and each process reports the expected model and compatibility settings. Keep this small SGLang prefill decode disaggregation deployment in CI or a pre-production environment so an upgrade does not quietly break the PD path.

For a Mooncake deployment, use the current SGLang PD guide for the required transfer-engine installation and IB-device configuration. Do not copy Mooncake environment variables from a different transport, cloud, or SGLang release.

Moving from one pair to a production topology

When the basic pair is healthy, scale each role based on queue and resource data rather than an assumed 1:1 ratio.

Layer	Scale from this observation	Practical control
Prefill pool	Prompt queue grows, prefill compute saturates, or TTFT tails rise	Add prefill replicas or change the prefill GPU shape after a controlled test.
Decode pool	Active sequences exhaust KV capacity, ITL tails rise, or decode admission waits	Add decode replicas, lower concurrency, or increase per-worker memory only after validating the model footprint.
KV-transfer fabric	Transfer time or transfer errors dominate the request timeline	Validate device mapping, backend, fabric health, and topology locality before adding model replicas.
Router	Unhealthy-worker churn, uneven queues, or routing delay appears	Use a production router topology and policy supported by your SGLang release.

For multi-node patterns, SGLang publishes separate prefill and decode launch examples for DeepSeek-class deployments, including role-specific distributed initialization and parallelism arguments. Those examples are a useful template, but the model, hardware, and SGLang version still need their own acceptance test. See the upstream multi-node PD examples before adapting a cluster manifest.

If Kubernetes owns the lifecycle, give prefill and decode separate labels, services, disruption budgets, dashboards, and readiness probes. The broader cluster prerequisites—GPU exposure, queueing, storage, networking, and observability—are covered in our Kubernetes GPU cluster guide.

Metrics for SGLang prefill decode disaggregation

SGLang prefill decode disaggregation should improve a chosen workload outcome, not merely produce a different resource chart. Instrument the request timeline with distinct prefill, transfer, decode-admission, and generation intervals.

Metric	What it reveals	Read it with
TTFT, p50 and p95	Prompt queueing plus prefill execution and handoff	Prompt-length buckets and prefill queue depth
ITL, p50 and p95	Token cadence after generation starts	Active decode sequences and decode KV occupancy
KV-transfer duration and errors	Whether the handoff is a bottleneck or failure source	Fabric throughput, retries, NIC health, and backend logs
Prefill/decode queue age	Whether the pool ratio is wrong	Role-specific replicas and concurrency limits
Completed requests per second	Useful capacity under the actual mix	Error rate, cancellation rate, and latency SLOs
GPU compute, HBM, and host CPU utilization	Supporting diagnosis, not the outcome	The four request-stage timings above

Run two test sets. One should use the traffic mix you expect today; the other should deliberately contain long prompts, variable generation lengths, cancellations, and bursts. PD can look excellent on an average synthetic prompt while failing its real purpose under a skewed production mix.

Decode allocation policy: a targeted control, not a default tuning trick

By default, SGLang documents an early decode allocation behavior: destination KV slots are reserved before prefill finishes. Long prefills can therefore hold decode capacity while the decode side waits for KV data.

The documented alternative is:

--disaggregation-decode-allocation-policy prefill_complete

With prefill_complete, SGLang delays destination allocation until prefill is ready. That can help when long prompts compete with active decode traffic for destination KV capacity, but it also changes the overlap and support envelope. Upstream documentation says to set the policy on both prefill and decode and lists several incompatible combinations. Treat it as a controlled experiment with rollback criteria, not a universal optimization.

Common failure modes and the fastest next check

Symptom	Likely boundary to inspect	Fast next check
The router accepts requests but no tokens return	Worker reachability or a failed role handoff	Check router logs, worker health, and one trace spanning prefill through decode.
Decode wait time grows while prefill is healthy	Decode admission or destination KV pressure	Inspect active sequences, KV occupancy, and the allocation policy before adding arbitrary prefill capacity.
TTFT improves but ITL gets worse	Decode pool or KV-transfer bottleneck	Compare decode queue age, transfer duration, and active sequence count against the unified baseline.
Multi-node traffic intermittently hangs or retries	Network fabric, device mapping, or inconsistent worker config	Verify the exact SGLang build, transfer backend, NIC mapping, and role manifests on every node.
An advanced feature works in unified mode but fails in PD	Unsupported feature combination	Read the current PD compatibility section and reproduce with the smallest supported topology.
GPU utilization rises but user latency does not improve	The optimization targeted the wrong stage	Return to TTFT, ITL, queue age, errors, and completed-work throughput.

The SGLang prefill decode disaggregation rollout should be reversible. Keep a unified serving route or a known-good deployment revision until the PD configuration meets its defined latency and reliability thresholds across the workload mix that matters.

A safe rollout sequence

1. Baseline: Capture a unified-SGLang benchmark with production-like prompts, outputs, concurrency, and failure cases.
2. One pair: Launch one prefill worker, one decode worker, and a router; pass functional, streaming, cancellation, and restart checks.
3. Role visibility: Split logs, dashboards, alerts, and capacity limits by router, prefill, transfer, and decode.
4. Controlled load: Compare PD against the baseline by prompt-length and output-length bucket, not only by an average.
5. Scale one constraint at a time: Adjust pool size, GPU shape, concurrency, or transfer configuration in separate experiments.
6. Canary and rollback: Route a bounded slice of traffic, define a latency/error rollback threshold, and preserve the last healthy configuration.

This sequence turns SGLang PD disaggregation prefill decode from a promising architecture diagram into an operationally testable change.

Frequently asked questions

Does PD disaggregation always increase throughput?

No. It may improve a workload where prefill and decode contention is the constraint, but it also introduces KV transfer, router scheduling, and operational overhead. Test it against a unified baseline using the same traffic mix and capacity budget.

Does prefill run on a different model than decode?

No. The roles continue one logical request. Keep the tested model, tokenizer, revision, quantization, and serving configuration aligned across the pair unless upstream documentation explicitly supports a different arrangement.

Which transfer backend should I use?

Use a backend supported by your pinned SGLang release and validated for your actual fabric. Current SGLang documentation describes Mooncake and NIXL, but the correct choice depends on the platform, container image, devices, and operational support you have verified.

Can I combine SGLang prefill decode disaggregation with every advanced feature?

No. PD support is feature- and version-sensitive. Check the current upstream compatibility notes before combining it with stateful responses, tool execution, cache features, speculation modes, or nonstandard role behavior.

The practical next step

Start with a one-pair acceptance environment, write down the TTFT/ITL/error thresholds that justify the architecture, and promote only after the transfer and failure paths are observable. For adjacent serving decisions, compare vLLM vs. SGLang, review speculative decoding for LLM inference, and plan the underlying Kubernetes GPU cluster separately from the PD rollout.