Prefill/Decode Disaggregation

LLM inference is not one workload. It splits into two phases with opposite hardware appetites, and almost every confused argument about "which accelerator is faster" dissolves once the phases are measured separately:

This is the hinge concept for this topic. It is why memory architecture — not FLOPs — sets the economics of serving (HBM4, KV Cache Management); it is the explicit scope condition on the $/PB cost unit in Memory Scarcity & Inference Economics, which is defined only for saturated, bandwidth-bound decode; and it is the architectural idea behind NVIDIA's now-cancelled disaggregated long-context SKU.

The independent benchmark

Usami, Vishwanath & Bethel, "Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators", arXiv:2606.17104, 14 June 2026 — accepted at HPAI4S'26, co-located with IEEE IPDPS 2026 (8 pages, 5 figures; cs.AR / cs.AI / cs.DC). The source registry attributes Argonne National Laboratory per author affiliations; affiliations are not listed on the arXiv abstract page.

This is the KB's first non-vendor, phase-separated head-to-head between GPUs and a non-GPU inference accelerator, on a common model (Llama2-7B). Findings, as stated in the abstract:

• GPUs consistently excel in the compute-intensive prefill phase.
• GroqRack achieves significantly lower TPOT during decode — with the explicit caveat that batching is not currently supported on that platform.
• GPUs regain the advantage in decode throughput as batch size increases.

The conclusion the authors draw is the useful one: each platform exhibits distinct phase-dependent strengths. There is no single winner — the answer depends on which phase, which metric (latency vs throughput), and which batch regime you are optimizing.

This matters commercially because it converts the standard "GPUs vs specialized inference silicon" argument into a workload-segmentation question. Low-batch, latency-sensitive decode (interactive agents, single-user long-generation) is exactly the regime where a bandwidth-first non-GPU architecture shows an advantage; high-batch throughput serving — where hyperscaler economics actually live — is where GPUs recover.

Provenance note. Abstract verified against arXiv; full text not ingested and no raw/*.md file exists. The abstract states directions, not magnitudes — no numeric TTFT/TPOT values are recorded here because none were read. Do not quote a speedup for this paper.

Why the phase split keeps showing up

• Decode re-reads all prior KV state per token, so attention becomes a bandwidth- and capacity-heavy memory task rather than a compute task. This is the mechanism TokenStack attacks by putting PIM logic next to hot KV (KV Cache Management).
• The $/PB inference-cost unit is defined only for bandwidth-bound decode. Any cost model built on it silently assumes the decode regime; prefill-heavy workloads (RAG with large contexts, batch document processing) are outside its scope (Memory Scarcity & Inference Economics).
• NVIDIA built and then cancelled a disaggregated prefill SKU. Rubin CPX (30 PFLOPS NVFP4, 128 GB GDDR7 — cheaper, lower-bandwidth memory chosen deliberately for the compute-bound prefill/ long-context phase) was roadmapped Sept 2025 and pulled at GTC 2026. The architectural idea — spend cheap memory on prefill, expensive HBM on decode — outlived the product (Rack-Scale AI Compute).
• "Disaggregated-pipeline" is one of the five KV-cache archetypes in the 2026 survey — i.e. splitting prefill and decode across separate engines is now a recognized standard serving pattern, not an experiment.

Key Claims

• Prefill is compute-bound (TTFT), decode is memory-bandwidth-bound (TPOT) — distinct computational characteristics requiring separate measurement. Evidence: strong — the paper's framing is standard across the serving literature and independently reflected in the KV-cache survey's archetypes (arXiv:2606.17104, arXiv:2607.02574)
• GPUs consistently win prefill; GroqRack wins decode TPOT at small batch; GPUs regain decode throughput as batch grows. Evidence: moderate — independent (non-vendor) peer-reviewed workshop benchmark on Llama2-7B, but abstract-level only and directional (no magnitudes read); GroqRack measured without batching support (arXiv:2606.17104)
• Accelerator advantage is phase- and metric-dependent — there is no single winner. Evidence: moderate (arXiv:2606.17104)
• $/PB inference-cost modeling is scoped to bandwidth-bound decode only. Evidence: moderate — explicit scope condition stated in the source model (close read)
• Disaggregated prefill/decode is a standard serving archetype, not an experiment. Evidence: moderate (arXiv:2607.02574)
• NVIDIA's cheap-memory prefill SKU (Rubin CPX, GDDR7) was cancelled at GTC 2026 while the disaggregation idea persisted. Evidence: moderate (multi-source) (NVIDIA full production)

Open Questions

• What are the actual TTFT/TPOT numbers in the Argonne study, and at what batch sizes does the GPU decode crossover happen? (Requires full-text ingest.)
• Does the GroqRack decode advantage survive once batching support lands — is it an architectural property or an artifact of a single-stream comparison?
• Does the result generalize past Llama2-7B to MoE models, where decode is sparse and routing-bound?
• Which side of the phase split do hyperscaler ASICs (Ironwood, Trainium3, Maia 200, MTIA) actually optimize for, and does anyone publish phase-separated numbers?
• Does a disaggregated prefill SKU return after the Rubin CPX cancellation, or does rack-level heterogeneity handle it instead?
• If a desk models cost-per-phase rather than cost-per-token, does the ASIC-vs-GPU verdict flip for any real workload mix?

Related Concepts

• Rack-Scale AI Compute — the rack is where phase heterogeneity gets resolved in practice
• KV Cache Management — decode's cost is KV traffic
• Memory Scarcity & Inference Economics — the cost unit that assumes this phase
• HBM4 Memory Architecture — bandwidth is the decode-phase currency
• Custom Silicon vs GPU — phase-aware benchmarking is how that argument should be settled

Backlinks

Pages that reference this concept:

• Rack-Scale AI Compute
• KV Cache Management
• Memory Scarcity & Inference Economics
• Groq

Changelog

• 2026-07-22 — Initial compilation from the Argonne prefill/decode evaluation (arXiv:2606.17104), cross-linked to the KV-cache survey's disaggregated-pipeline archetype, the $/PB decode scope condition, and the Rubin CPX cancellation. Abstract-level only — directions recorded, no magnitudes, since full text was not ingested.