Next-Level Inference: Why Your Single-Node vLLM Setup Needs Prefill-Decode Disaggregation

Standing: vendor-run measurement (AMD hardware, AMD-affiliated connector), but the comparison is same-hardware / same-model / same-request-stream, with the arms differing only in whether prefill and decode share GPUs. It also reports its own regression (TTFT), which is what separates an engineering post from a benchmark ad.

What is being measured, and why goodput is the right unit

Standard vLLM runs prefill and decode collocated on the same GPUs. The two phases have opposite hardware appetites: prefill is compute-bound (large matrix ops over the whole prompt), decode is memory-bandwidth-bound (one token at a time). Collocating them means an arriving prompt's prefill work preempts in-flight decode steps — head-of-line blocking that shows up to the user as inter-token latency spikes.

Disaggregation gives each phase dedicated GPUs, and the KV cache produced by prefill must then cross the wire to the decode instance.

The post reports goodput — requests that met BOTH latency SLOs — rather than raw throughput. That is the load-bearing methodological choice here: a server can raise tokens/second while serving a larger share of requests that are commercially useless because they missed their latency budget.

Mechanism: the MORI-IO KV connector

MORI-IO is an RDMA-based connector moving KV cache from prefill to decode instances, in two modes:

• Read mode (VLLM_MORIIO_CONNECTOR_READ_MODE=1) — the proxy dispatches serially: it waits for prefill to finish, extracts the remote block IDs, then hands them to decode, which pulls the KV over RDMA.
• Write mode (default) — prefill pushes the KV layer-by-layer into the decode instance's memory as computation proceeds. The proxy dispatches both phases concurrently, so the serialisation overhead disappears and the RDMA transfer overlaps with prefill compute.

Write mode is the better of the two precisely because the transfer hides behind work already being done.

Test bed

Results (at 8 req/s)

Across the sweep: both disaggregated modes hold 100% SLO attainment up to roughly 5–5.5 req/s, then decline as TTFT starts breaching 1 s. Collocated serving never reaches that plateau — its ITL violations begin much earlier.

The bimodal ITL signature of the 2×TP4 arm (~30 ms when undisturbed, ~150 ms when a prefill lands mid-decode) is the clearest direct evidence of head-of-line blocking in the post.

The stated trade-off

Disaggregation does not make serving uniformly better — it moves the failure from ITL to TTFT:

• ITL improves and stabilises. Dedicated decode GPUs are never interrupted by arriving prompts, so token generation is predictable regardless of new arrivals.
• TTFT worsens. Read mode adds proxy serialisation plus RDMA transfer time; write mode removes the serialisation but still pays the transfer, overlapped though it is.

The post's own deployment rule, which is the part worth keeping:

• Disaggregate when p99 ITL is what breaches your SLO under concurrency, and when prompts are long and concurrency is high.
• Stay collocated when TTFT is the binding constraint — interactive chat UX, where the user feels the first token.

Limitations

• Vendor-run on vendor hardware with a vendor connector; not an independent bench.
• Measured on an MoE model (Qwen3-235B-A22B-FP8). The authors argue prefill/decode interference is fundamental to transformer inference and so applies to dense models too — argued, not shown.
• Single node, 1P+1D topology. Nothing here says how the goodput multiple scales at rack level or with multiple prefill workers per decode worker.
• Benchmarks run 2026-03-12 on release-candidate vLLM (0.16.0rc1); the authors note results vary with configuration, software version and tuning.
• No cost figure. Goodput is per-GPU-hour only implicitly — the GPU count is held constant, so the 2.5× is a genuine efficiency gain, but the post never converts it to $/request.

How it bears on MenFem

• The rung's scope names prefill/decode disaggregation as in-scope and held no measured figure for it. This is that figure, and it arrives in the right unit.
• The two cost-decline calls recorded against serving both assume the run gets cheaper without new silicon. 26/100 → 73/100 on identical hardware is exactly that claim, measured: a 2.5× effective capacity gain from scheduling, not from a chip.
• Non-NVIDIA hardware is a feature here — it separates the mechanism from the vendor, which matters when the question is whether serving-side wins are structural or CUDA-specific.
• Teaching moment: "the same eight GPUs served nearly three times as many useful requests once we stopped making prefill and decode share" — plain language, a real mechanism, and a named price (first-token latency).