# Prefill/Decode Disaggregation

Canonical URL: https://menfem.com/kb/hardware/concepts/prefill-decode-disaggregation
Knowledge base topic: [Hardware & Computing](https://menfem.com/kb/hardware)
Frontier status: active
Tags: prefill-decode, inference-hardware, benchmarking, memory-bandwidth, groqrack, ai-memory-wall

---

LLM inference is not one workload. It splits into two phases with opposite hardware appetites, and
almost every confused argument about "which accelerator is faster" dissolves once the phases are
measured separately:

| Phase | What it does | Bound by | Latency metric |
|---|---|---|---|
| **Prefill** | Processes the whole input prompt in one parallel pass | **Compute** (dense matmul, high arithmetic intensity) | **TTFT** — time to first token |
| **Decode** | Emits one token at a time, re-reading all prior KV state each step | **Memory bandwidth and capacity** | **TPOT** — time per output token |

This is the hinge concept for this topic. It is why memory architecture — not FLOPs — sets the
economics of serving ([HBM4](./hbm4-memory-architecture.md),
[KV Cache Management](./kv-cache-management.md)); it is the explicit scope condition on the $/PB cost
unit in [Memory Scarcity & Inference Economics](./memory-scarcity-inference-economics.md), which is
defined only for *saturated, bandwidth-bound decode*; and it is the architectural idea behind
NVIDIA's now-cancelled disaggregated long-context SKU.

## The independent benchmark

*Usami, Vishwanath & Bethel, "Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI
Accelerators", [arXiv:2606.17104](https://arxiv.org/abs/2606.17104), 14 June 2026 — accepted at
HPAI4S'26, co-located with IEEE IPDPS 2026 (8 pages, 5 figures; cs.AR / cs.AI / cs.DC). The source
registry attributes Argonne National Laboratory per author affiliations; affiliations are not listed
on the arXiv abstract page.*

This is the KB's first **non-vendor, phase-separated** head-to-head between GPUs and a non-GPU
inference accelerator, on a common model (Llama2-7B). Findings, as stated in the abstract:

- **GPUs consistently excel in the compute-intensive prefill phase.**
- **GroqRack achieves significantly lower TPOT during decode** — with the explicit caveat that
  **batching is not currently supported** on that platform.
- **GPUs regain the advantage in decode throughput as batch size increases.**

The conclusion the authors draw is the useful one: *each platform exhibits distinct phase-dependent
strengths.* There is no single winner — the answer depends on which phase, which metric (latency vs
throughput), and which batch regime you are optimizing.

This matters commercially because it converts the standard "GPUs vs specialized inference silicon"
argument into a **workload-segmentation** question. Low-batch, latency-sensitive decode (interactive
agents, single-user long-generation) is exactly the regime where a bandwidth-first non-GPU
architecture shows an advantage; high-batch throughput serving — where hyperscaler economics
actually live — is where GPUs recover.

> **Provenance note.** Abstract verified against arXiv; **full text not ingested** and no `raw/*.md`
> file exists. The abstract states directions, not magnitudes — no numeric TTFT/TPOT values are
> recorded here because none were read. Do not quote a speedup for this paper.

## Why the phase split keeps showing up

- **Decode re-reads all prior KV state per token**, so attention becomes a bandwidth- and
  capacity-heavy memory task rather than a compute task. This is the mechanism TokenStack attacks by
  putting PIM logic next to hot KV ([KV Cache Management](./kv-cache-management.md)).
- **The $/PB inference-cost unit is defined only for bandwidth-bound decode.** Any cost model built
  on it silently assumes the decode regime; prefill-heavy workloads (RAG with large contexts, batch
  document processing) are outside its scope
  ([Memory Scarcity & Inference Economics](./memory-scarcity-inference-economics.md)).
- **NVIDIA built and then cancelled a disaggregated prefill SKU.** Rubin CPX (30 PFLOPS NVFP4, 128 GB
  **GDDR7** — cheaper, lower-bandwidth memory chosen deliberately for the compute-bound prefill/
  long-context phase) was roadmapped Sept 2025 and pulled at GTC 2026. The *architectural idea* —
  spend cheap memory on prefill, expensive HBM on decode — outlived the product
  ([Rack-Scale AI Compute](./rack-scale-ai-compute.md)).
- **"Disaggregated-pipeline" is one of the five KV-cache archetypes** in the 2026 survey — i.e.
  splitting prefill and decode across separate engines is now a recognized standard serving pattern,
  not an experiment.

## Key Claims

- **Prefill is compute-bound (TTFT), decode is memory-bandwidth-bound (TPOT)** — distinct
  computational characteristics requiring separate measurement. *Evidence: strong — the paper's
  framing is standard across the serving literature and independently reflected in the KV-cache
  survey's archetypes* ([arXiv:2606.17104](https://arxiv.org/abs/2606.17104),
  [arXiv:2607.02574](https://arxiv.org/abs/2607.02574))
- **GPUs consistently win prefill; GroqRack wins decode TPOT at small batch; GPUs regain decode
  throughput as batch grows.** *Evidence: moderate — independent (non-vendor) peer-reviewed workshop
  benchmark on Llama2-7B, but abstract-level only and directional (no magnitudes read); GroqRack
  measured without batching support* ([arXiv:2606.17104](https://arxiv.org/abs/2606.17104))
- **Accelerator advantage is phase- and metric-dependent — there is no single winner.** *Evidence:
  moderate* ([arXiv:2606.17104](https://arxiv.org/abs/2606.17104))
- **$/PB inference-cost modeling is scoped to bandwidth-bound decode only.** *Evidence: moderate —
  explicit scope condition stated in the source model*
  ([close read](../../raw/memory-scarcity-2607.07207-closeread.md))
- **Disaggregated prefill/decode is a standard serving archetype, not an experiment.** *Evidence:
  moderate* ([arXiv:2607.02574](https://arxiv.org/abs/2607.02574))
- **NVIDIA's cheap-memory prefill SKU (Rubin CPX, GDDR7) was cancelled at GTC 2026 while the
  disaggregation idea persisted.** *Evidence: moderate (multi-source)*
  ([NVIDIA full production](../../raw/nvidia-vera-rubin-full-production-2026.md))

## Open Questions

- What are the actual TTFT/TPOT numbers in the Argonne study, and at what batch sizes does the GPU
  decode crossover happen? (Requires full-text ingest.)
- Does the GroqRack decode advantage survive once batching support lands — is it an architectural
  property or an artifact of a single-stream comparison?
- Does the result generalize past Llama2-7B to MoE models, where decode is sparse and routing-bound?
- Which side of the phase split do hyperscaler ASICs (Ironwood, Trainium3, Maia 200, MTIA) actually
  optimize for, and does anyone publish phase-separated numbers?
- Does a disaggregated prefill SKU return after the Rubin CPX cancellation, or does rack-level
  heterogeneity handle it instead?
- If a desk models cost-per-phase rather than cost-per-token, does the ASIC-vs-GPU verdict flip for
  any real workload mix?

## Related Concepts

- [Rack-Scale AI Compute](./rack-scale-ai-compute.md) — the rack is where phase heterogeneity gets resolved in practice
- [KV Cache Management](./kv-cache-management.md) — decode's cost *is* KV traffic
- [Memory Scarcity & Inference Economics](./memory-scarcity-inference-economics.md) — the cost unit that assumes this phase
- [HBM4 Memory Architecture](./hbm4-memory-architecture.md) — bandwidth is the decode-phase currency
- [Custom Silicon vs GPU](./custom-silicon-vs-gpu.md) — phase-aware benchmarking is how that argument should be settled

## Backlinks

*Pages that reference this concept:*
- [Rack-Scale AI Compute](./rack-scale-ai-compute.md)
- [KV Cache Management](./kv-cache-management.md)
- [Memory Scarcity & Inference Economics](./memory-scarcity-inference-economics.md)
- [Groq](../entities/groq.md)

## Changelog

- **2026-07-22** — Initial compilation from the Argonne prefill/decode evaluation (arXiv:2606.17104), cross-linked to the KV-cache survey's disaggregated-pipeline archetype, the $/PB decode scope condition, and the Rubin CPX cancellation. Abstract-level only — directions recorded, no magnitudes, since full text was not ingested.

## Sources

- prefill-decode-aware-evaluation-emerging-accelerators
- kv-cache-management-survey-tensor-buffer-distributed-memory
- memory-scarcity-2607.07207-closeread
- nvidia-vera-rubin-full-production-2026

---

Cite as: MenFem Knowledge Base — https://menfem.com/kb/hardware/concepts/prefill-decode-disaggregation