Serving & Runtime

Serving is what happens between a trained model and a served token: how requests are batched, how the key-value cache is stored, evicted and reused, how weights are compressed and unpacked, and which of the two phases of generation — the compute-bound prefill that reads the prompt, and the bandwidth-bound decode that emits one token at a time — is the thing actually holding the box up. The clearest statement of why this deserves its own rung comes from the field's own survey: "the key-value (KV) cache has become a first-order memory object in LLM serving rather than a temporary per-request tensor" (From Tensor Buffer to Distributed Memory Hierarchy).

What this rung prices

A model price is quoted per token, but the tokens have to be produced by real silicon that is either full or idle, and this rung is where that gap is set. The sharpest measurement on it is also the simplest: one H200, one model, one serving framework, nothing changed but the traffic shape, and the cost to produce a million output tokens moves from $20.32 to $0.45 (Token Economics Across Traffic Profiles on Dedicated GPUs). That is a ~44× swing with no new model, no new chip and no new price list — so a quoted cost per token without a stated utilisation is not a number, and a buyer paying per finished job is paying for a batch-occupancy outcome he cannot see. Everything else here works the same lever from a different direction: shrink what has to be moved (KV compression, eviction, quantization), overlap what has to be waited on (speculative decoding), or put the arithmetic closer to the memory. Each one changes how many finished jobs a fixed box yields per hour, which is the only place a token price and a task price are joined.

The load-bearing findings

• Traffic shape alone moves cost per million output tokens by ~44× on identical hardware. A single H200 at $3.44/GPU-hour running Llama-3.3-70B in FP8 on vLLM 0.24.0 measured 47 tok/s at batch 1 and 2,140 tok/s at batch 128 — $20.32/M against $0.45/M, with the own-a-GPU versus buy-serverless crossover for that configuration sitting near 73% sustained utilisation (Token Economics Across Traffic Profiles on Dedicated GPUs). Throughput was measured; the dollar figures are derived arithmetically from it at the list rate, and the author states plainly that they are a reference point for one configuration, not a guarantee.
• The two phases of generation are two different machines, and averaging them hides the answer. Prefill is compute-bound and decode is bandwidth-bound, so accelerator advantage is phase-dependent: GPUs consistently win prefill, GroqRack posts significantly lower per-output-token latency in decode at small batch, and GPUs take decode back as batch size rises (Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators). That close read is abstract-level and the abstract carries no magnitudes, so the directions are what stands, not the sizes. The same split is what a serving-capacity model has to fit: Epoch's first-principles model, calibrated against 111 measured SemiAnalysis InferenceX runs of Kimi K2.5, fits compute efficiency at 65% and bandwidth efficiency at 30%, and lands ~400,000 tokens/sec on a GB200 NVL72 (Is a Compute Crunch Coming?) — modelled rather than measured, and denominated in tokens/sec with no dollar figure attached. See prefill/decode disaggregation.
• Most of the KV cache can go, and compressing it need not cost exact output. LaProx scores token importance on a single global scale across the whole model rather than per attention head, and holds performance at 5% KV-cache retention across 19 long-context datasets, cutting accuracy loss by up to 2× against prior methods under extreme compression (Reformulating KV Cache Eviction Problem, read at abstract and reported-metrics grade). Separately, VeriCache uses the compressed cache in GPU memory as a drafter and checks it against the full cache held on CPU or over the network, giving output identical to full-KV decoding (KL below 0.01 nats) at 2.73× throughput on Llama-70B, and ~25–40 accepted tokens per verification round against 2–3 for a small-model drafter (VeriCache; authors' own benchmarks, no independent reproduction). See KV cache and KV cache management.
• The levers compound, but not automatically — one pairing had to be measured before it worked. Applying tree-style EAGLE-2 speculative decoding to a 4-bit weight-quantized model erodes the quantization win, because verifying a tree draft costs more than a single-token forward pass on that target; inserting a small intermediate model that converts tree drafts into sequence drafts restores it, at 2.78× on 4-bit Llama-3-70B on an A100 and +1.31× over EAGLE-2 on the same setup (Speculative Decoding Meets Quantization, abstract and reported-metrics grade). Where the design is built for the pairing from the start it does stack: quantizing target and drafter together with sliding-window attention gave a 6.978× average speedup while clearing fixed quality thresholds — though that average runs 11.1× on short prompts down to 2.85× at 8,192-token inputs, and the reported latencies were reproduced on an RTX 5000 Ada because the competition's A10G was unavailable (Quantize the Target, Quantize the Drafter). See speculative decoding and low-bit quantization.
• Speculative decoding is in production and the production numbers come from the vendor. DeepSeek reports deploying DSpark in the DeepSeek-V4 serving system and measuring, on its own live traffic, 60–85% faster per-user generation on V4-Flash and 57–78% on V4-Pro against its prior production baseline, at matched throughput and with the exact acceptance rule left intact (DSpark). This is first-party self-measurement of the authors' own fleet with no independent reproduction, and the paper's own much larger figures under strict interactivity constraints are a different operating regime, not the headline. The academic end of the same lever is incremental by comparison: truncating the drafter's training loss at its first predicted failure lifts average emitted length from 2.40 to 2.61 tokens on Qwen3-8B, and the authors state that 8B and a block size of 16 are the only configuration tested (Spec-AUF).
• Three separate 2026 designs argue the memory stack should do arithmetic, and they disagree about how far to go. SK Hynix engineers put dequantization logic on the HBM base die, reporting up to 7.08× GEMM speedup, 90.23% lower energy and 54.68% end-to-end latency reduction, at 0.127 mm² and 0.355 W per block in 12 nm (StreamDQ); TokenStack splits a stack into capacity layers and compute layers for 1.62× geomean throughput against a prior PIM baseline (TokenStack); AMMA removes the GPU compute die from the attention path entirely — 16 memory-with-compute cubes at 44 TB/s and 240 W package power, reporting 15.5× lower attention latency and 6.9× lower energy than an H100 on 1M-token decode attention, but only 1.1× against NVIDIA's Rubin at that length (AMMA). All three are simulated or synthesised by their own authors, and all three concentrate their wins in the memory-bound small-batch regime — StreamDQ's authors say so explicitly, that the advantage shrinks as batch size grows.
• Compression and reuse can be made to co-exist, and the software layer is where it is cheapest. C²KV learns a position-agnostic cache representation so non-prefix blocks can be concatenated and reused without the accuracy collapse naive compression-plus-reuse causes: at 4× compression it scores 0.3587 on MuSiQue against 0.3198 for full recompute, with up to 17× long-context speedup and no change to the frozen base model (C²KV; authors' own benchmarks). The survey's structural finding sits underneath it — across 30-plus KV-management systems on four axes, once workload and hardware are fixed it is ownership, not locality or substrate, that accounts for much of the remaining design variance (From Tensor Buffer to Distributed Memory Hierarchy).

What we do not know yet

• Nothing on this rung has been run here. Every number above is read, not measured. The one measured cost series in the KB is DigitalOcean's own H200 sweep, and even there the throughput is theirs and the dollars are arithmetic over a list rate (Token Economics Across Traffic Profiles on Dedicated GPUs).
• The vendor and the academy are not measuring the same thing, and neither checks the other. DSpark's 60–85% is DeepSeek measuring DeepSeek (DSpark), StreamDQ's 7.08× is SK Hynix simulating SK Hynix (StreamDQ), and no source here reproduces another source's result. The near-memory papers are the clearest case: TokenStack's 1.62× is measured against one prior PIM design rather than against a GPU stack anyone deploys (TokenStack).
• The field is missing measurements it knows it is missing. The survey audits current evaluations and names seven KV-specific measurements that are absent, tied to fault tolerance, tiered eviction, speculative decoding and MoE serving (From Tensor Buffer to Distributed Memory Hierarchy). A benchmark blind to tiered eviction cannot rank tiered-eviction designs, which is most of what this rung now contains.
• Where the levers stop compounding is visible but not mapped. The one systematic pairing study covers weight quantization only, on one model family and one accelerator, and says activation and KV quantization interact differently (Speculative Decoding Meets Quantization); the drop from 11.1× to 2.85× as prompts lengthen (Quantize the Target, Quantize the Drafter) says the decay is real without saying what governs it.
• Two of the phase-level claims reach us thin. The prefill/decode accelerator comparison is abstract-level with no magnitudes at all and measures GroqRack without batching support, which is exactly the condition under which GPUs later win (Prefill/Decode-Aware Evaluation), and the capacity model behind the compute-crunch claim is calibrated on one model family and states no dollars (Is a Compute Crunch Coming?). The demand-growth side of that gap — roughly 10× a year against roughly 3.4× a year of capacity — is drawn from disclosed industry metrics of varying reliability, by the authors' own admission.
• A quantization result that reads as capability-neutral may not be. OSAQ's null-space transform is stated to suppress outliers "without affecting the task loss", and the KB's own close read flags that this is a perplexity and task-loss statement rather than a capability one, since outlier weights are implicated in rare behaviours (OSAQ). Nothing here tests that.

Read next

• measured-h200-token-serving-cost-digitalocean — the only dollar-denominated cost series here, and the 44× utilisation swing the whole rung turns on.
• kv-cache-management-survey-tensor-buffer-distributed-memory — the map of the serving-software layer, and the honest list of what nobody is measuring.
• vericache-lossy-kv-lossless-inference — the cleanest worked example of two levers becoming one mechanism, with exact output preserved.