Serving & Runtime — Frontier
Serving & Runtime — Frontier
The open questions on the rung that turns a model into throughput. Compiled 2026-08-28 from the 15 close-reads held here since the rung was created on 2026-08-06. This rung had never been compiled before that date: the reading was done, the wiki was not.
The binding constraint, named
The KV cache at decode. The rung's survey states it plainly — "the key-value (KV) cache has become a first-order memory object in LLM serving rather than a temporary per-request tensor" (KV Cache Management Survey) — and TokenStack gives the mechanism: "during decode, each new token rereads prior KV state, so attention becomes a bandwidth- and capacity-heavy memory task" (TokenStack).
That is why nine of the fifteen sources here are about the KV cache or the memory path, and why the rung could not stay filed under models or hardware.
The number this rung owns
A 44x cost spread on a single SKU, from traffic shape alone. One H200 at $3.44/GPU-hr running Llama-3.3-70B FP8 on vLLM, swept across batch/concurrency 1 to 256, moves from $20.32 to $0.45 per million output tokens (DigitalOcean measured benchmark, as-of 2026-07-08).
Same model, same chip, same framework. Only the traffic shape changes. (The source states 44x; the quoted endpoints divide to 45.2x. Cited as published.)
This is the rung's signature claim, and it is the reason serving is not a subheading of inference-economics: the economics rung prices a finished task, but this is the number that says how you serve it moves the cost by an order of magnitude and a half before anyone negotiates a rate.
Active questions
| Question | State of evidence | What would settle it |
|---|---|---|
| Do quantization and speculative decoding compound, or fight? | The rung contains a genuine contradiction. SpecMQuant finds EAGLE-2 tree drafts cancel 4-bit quantization's memory benefit — verifying a tree draft costs more than a single-token forward pass on 4-bit weights — and restores it only by converting tree drafts to sequence drafts: 2.78x on 4-bit Llama-3-70B, beating EAGLE-2 by 1.31x. Nota's competition entry quantizes both target and drafter and reports 6.978x over BF16, concluding they compound. The likely reconciliation is that draft shape, not the pairing, is the variable — but no source tests that directly. | A matched-hardware comparison holding draft shape fixed while varying bit-width. |
| Who should own the KV cache? | Open, and it is the survey's load-bearing finding: across 30+ systems on four axes (locality, lifetime, ownership, substrate), once workload and hardware are fixed, ownership accounts for much of the remaining design variance — not locality, not substrate. The same survey names seven missing KV-specific measurements. | Any evaluation that reports those seven measurements against the five archetypes. |
| How far can KV compression go before it stops being free? | Two incompatible answers to one pressure. LaProx holds performance at 5% KV retention across 19 datasets and halves accuracy loss under extreme compression. VeriCache refuses the trade entirely — it uses the lossy KV as a drafter and verifies against full KV, yielding 2.7–4.3x at KL < 0.01 nats, sustaining 25–40 accepted tokens per round against 2–3 for small-model drafters. | Whether the verification path's PCIe/network cost survives at production concurrency. Both are single-lab, self-measured. |
| Does the memory-centric turn survive production? | Unresolved and consequential. AMMA reports 15.5x lower attention latency and 6.9x lower energy vs H100 on 1M-token decode attention; StreamDQ (SK Hynix, on the HBM base die) reports 7.08x GEMM speedup, 90.23% lower energy, 54.68% end-to-end latency reduction. All of it is simulated or vendor-measured. None is deployed at scale. | A shipping part. C-HBM4E is the one with a roadmap behind it. |
| Is phase-aggregated benchmarking simply wrong? | Partly answered, and the answer is yes. Measured separately on Llama2-7B, GPUs consistently win prefill (compute-bound) while GroqRack achieves significantly lower TPOT during decode (memory-bound) — a single aggregate number hides an inversion. (Caveat: batching unsupported on that platform, so the comparison is not like-for-like at production concurrency.) | Re-running the comparison once batching exists on the non-GPU platform. |
| Does serving capacity keep up with demand? | Modelled, not measured, but calibrated. Epoch's prefill/decode model — fitted to 111 real SemiAnalysis InferenceX runs (compute efficiency 65%, bandwidth 30%, 5ms/step, 400k tok/s on GB200 NVL72) — puts capacity growth at ~3.4x/yr against demand at ~10x/yr. | Whether the gap shows up as price, queueing, or silent quality degradation. Nobody is measuring the third. |
What this rung does not yet hold
- Only one dollar figure in fifteen sources. Everything except the DigitalOcean benchmark is a speedup ratio against a baseline — and a ratio cannot be added to a bill. This is the same hole
harnessesnames, arriving from the other direction. - No independent reproduction of anything. DSpark is DeepSeek measuring DeepSeek's own production stack (60–85% on V4-Flash, 57–78% on V4-Pro vs the MTP-1 baseline, distribution-preserving guarantee intact); VeriCache, OSAQ, Spec-AUF and StreamDQ are all self-measured by their authors. The one externally adjudicated result is Nota's, by competition placement (3rd of 40+) rather than by replication.
- Nothing at all on batching and scheduling — despite both sitting explicitly in this rung's scope in
_meta.json. Zero sources. The 44x number above is produced by batching, and the rung holds no source about how to choose it. - Nothing on routing or multi-tenant serving, also in scope, also zero.
- Nothing on MoE serving, which the survey names as an open problem.
Standing note
Connor's seat on this rung is read it. He has run none of these systems, and the two that would be cheapest to run — a vLLM batch sweep reproducing the 44x, and a speculative-decoding acceptance-rate measurement — have not been run.
Per the teaching doctrine that caps this rung at coverage, not teaching: a piece built on this rung today may report and explain, but a how to do it claim here would fail the standing check. Deep-dive menu #2 (the frozen-harness benchmark) is the item that converts this seat, and the DigitalOcean sweep is the cheapest single experiment that would do it.