Prefill/Decode-Aware Evaluation on Emerging Accelerators

The finding

Accelerator advantage is phase-dependent, and a single aggregate number hides it. Measuring prefill and decode separately on a common model (Llama2-7B):

• GPUs consistently win the prefill phase — it is compute-bound, which is what GPUs are for.
• GroqRack achieves significantly lower TPOT during decode — decode is memory-bound, a different problem. (Batching not currently supported on that platform.)
• GPUs regain the decode advantage as batch size increases — so the crossover is a function of batch size, not a fixed ranking.

Why this is a serving source and not a hardware one

The question it answers is how do you run this cheaply, not what is the silicon. It also demonstrates the rung's central methodological point: "which accelerator is faster" is an unanswerable question until you say which phase and which batch size. TTFT and TPOT are different metrics measuring different bottlenecks, and a benchmark reporting one number for "inference" is reporting an average over two unlike regimes.

The paper goes on to analyse heterogeneous prefill/decode disaggregation across platforms — running each phase on the hardware that suits it — and identifies the workload and network conditions under which the gains are real.

Limits

The abstract carries no specific numbers — the directional claims are stated without the magnitudes, which have to come from the paper body. Single model (Llama2-7B), and GroqRack is measured without batching support, which is precisely the condition under which GPUs later win. A workshop paper at IPDPS.

Provenance

conference, arXiv 2606.17104, HPAI4S'26. Ingested 2026-08-06 — listed without a close read since before the rung split.