In scope: batching and scheduling, KV-cache management and eviction, quantization at serving time, speculative decoding, prefill/decode disaggregation, paged attention, routing, throughput and latency engineering. Out: what the model can do (models), what a finished task costs (inference-economics), the silicon it runs on (hardware).
| Type | Source | Published |
|---|---|---|
| REPORT | Is a Compute Crunch Coming? (inference serving-capacity model, calibrated to SemiAnalysis InferenceX) Luke Emberson, Jaime Sevilla · Epoch AI First-principles prefill(compute-bound)/decode(bandwidth-bound) serving-throughput model CALIBRATED to 111 measured SemiAnalysis InferenceX Kimi K2.5 runs (fitted: compute eff 65%, bandwidth eff 30%, 5ms/step). GB200 NVL72 ~400k tok/s; global capacity 500M-20B tok/s; capacity growth ~3.4x/yr vs demand ~10x/yr => crunch. Denominated in tokens/sec, NO $/token. As-of 2026-05-25. | 2026-05-25 |
| REPORT | Next-Level Inference: Why Your Single-Node vLLM Setup Needs Prefill-Decode Disaggregation AMD and Embedded LLM (Hongxia Yang, Gilbert Lei, Mingzhi Liu, et al.) · AMD / Embedded LLM / vLLM project Same 8x MI300X, Qwen3-235B-A22B-FP8, 2000-in/1000-out, SLOs TTFT<1s AND ITL<50ms: requests meeting BOTH SLOs at 8 req/s go 26/100 (1xTP8) and 30/100 (2xTP4) collocated to 70/100 (MORI-IO read) and 73/100 (write) disaggregated — 2.5x goodput. Trade-off stated: ITL stabilises, TTFT worsens (RDMA transfer + proxy serialisation); 100% SLO attainment holds to ~5-5.5 req/s. Benchmarks run 2026-03-12. | 2026-04-07 |
| REPORT | Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators Shun Usami, Venkatram Vishwanath, E. Wes Bethel · Argonne National Laboratory (per author affiliations); accepted HPAI4S'26 / IEEE IPDPS 2026 Empirical head-to-head of GPUs vs. GroqRack across the prefill (compute-bound) / decode (memory-bound) split using Llama2-7B: GPUs consistently win prefill, GroqRack wins decode TPOT at small batch, GPU decode advantage returns at large batch. Grounds 'which architecture wins which inference phase' in independent (Argonne) benchmarks rather than vendor claims — relevant to both this topic's rack-scale-compute frontier and inference-economics cost-per-phase modeling. | 2026-06-14 |