Serving & Runtime
How a model is actually run, per token — the mechanics that turn capability into throughput. In scope: batching and scheduling, KV-cache management and eviction, quantization at serving time, speculative decoding, prefill/decode disaggregation, paged attention, routing, throughput and latency engineering. Out: what the model can do (models), what a finished task costs (inference-economics), the silicon it runs on (hardware).
DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
First-party DeepSeek disclosure of a production speculative-decoding system combining parallel semi-autoregressive drafting with confidence-scheduled adaptive verification; claims 60-85% real-world deployment speedups. DeepSeek is already a KB-tracked and MenFem-studied entity (deepseek-v4-2026 source, entities/deepseek.md) — this is a first-party look at their inference-efficiency stack, directly bearing on the 'who captures inference margin' thesis the inference-economics topic tracks.
Spec-AUF: Accept-Until-Fail Training under Train-Inference Misalignment for Masked Block Drafters
Single-mechanism fix for a known train/inference mismatch in block (DLM-style) speculative drafters: truncates the cross-entropy training support to the drafter's own first predicted failure point, concentrating supervision on the accepted prefix rather than the full block. No auxiliary objective, no verifier rollouts, no inference-pipeline change. Raises average emitted length τ from 2.40→2.61 on Qwen3-8B (DFlash) and transfers to a second drafter family (Domino, 2.56→2.68) — representative of the current 'free efficiency' phase of speculative-decoding research.
Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B
Combines quantization and speculative decoding in one system for constrained hardware (NVIDIA A10G): quantization-aware distillation for the target model + a two-stage-trained block-diffusion drafter, plus sliding-window attention. Achieves 6.978x average speedup over baseline while meeting quality thresholds (3rd place, Efficient Qwen Competition). Concrete evidence the two inference-efficiency techniques this lane tracks (quantization, speculative decoding) compound rather than substitute.
OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization
Training-free, closed-form (no iterative optimization) post-training weight-only quantization: exploits a stable low-rank null space in the Hessian to build an additive weight transformation that suppresses outliers without affecting the task loss, absorbed into the weights offline with zero inference overhead. At 2-bit, OSAQ *integrated with* GPTQ achieves >40% lower perplexity than vanilla GPTQ (a complement to GPTQ, not a replacement).
VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
Makes lossy KV-cache compression lossless by using compressed KV (GPU HBM) as a speculative drafter and verifying against full KV (CPU/remote) in parallel — drafting is HBM-bound, verification PCIe/network-bound, so the paths overlap. Up to ~2.7-4.3x throughput over full-KV with output identical to full-KV (KL<0.01 nats), ~25-40 accepted tokens/round vs 2-3 for small-model drafters; composes with EAGLE/MTP to 4.35x. Opens KV-cache coverage in the KB.
Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design
Resolves the KB's flagged speculative-decoding × quantization CONFLICT with measurement: applying tree-style EAGLE-2 to a 4-bit weight-quantized model diminishes the quantization memory benefit because tree-draft verification costs significantly more than a single-token forward pass; a hierarchical framework using a small intermediate model to convert tree drafts into sequence drafts restores it — 2.78× speedup on 4-bit-weight Llama-3-70B (A100), +1.31× over EAGLE-2 on the same setup. Resolves the KB's highest-priority flagged tension (compound-vs-conflict) with measurement. Weight-quantization finding only.
Reformulating KV Cache Eviction Problem for Long-Context LLM Inference (LaProx)
Recasts KV-cache eviction from head-wise attention-weight averaging into an output-aware, layer-wise matrix-multiplication approximation (LaProx), modeling the multiplicative interaction between attention maps and projected value states to yield the first GLOBALLY-comparable token importance score for model-wide (not per-head) selection; maintains model performance at just 5% KV-cache retention across 19 datasets (LongBench + Needle-In-A-Haystack) and cuts accuracy loss up to 2× vs prior SOTA under extreme compression, with minimal overhead. Broadens the KB's KV-cache lane from compression into eviction.
From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
First systematic taxonomy of 30+ KV-cache-management systems across four design dimensions (locality, lifetime, ownership, substrate) and five architectural patterns (local-paged, disaggregated-pipeline, shared-store, memory-pool, hybrid-tier); identifies seven evaluation-methodology gaps tied to fault tolerance, tiered eviction, speculative decoding, and MoE serving. Best current map of how the serving-software layer answers the memory-wall problem this topic already tracks at the hardware layer.
TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inference
Concrete HBM4-logic-die PIM architecture: separates memory stacks into capacity-focused vs. compute-enabled layers, using the HBM4 base die as a local controller; claims 1.62x geomean token throughput, 1.70x latency-sensitive serving capacity, 30-47% per-token energy reduction on production-derived workloads. Directly answers two open Knowledge Gaps already flagged in this topic's frontier.md ('Commercial PIM product specs' and 'C-HBM4E NMC workload adoption'). Published 2026-05-07 — just outside the strict 60-day discovery window but flagged given the direct gap match.
StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration
SK Hynix engineers put dequantization logic on the HBM base die (DeQuantization Blocks) so quantized weights are unpacked memory-side during standard loads, eliminating GPU CUDA-core dequantization. Up to 7.08x GEMM speedup, 90.23% lower energy, per-DQB 0.127mm2/0.355W in 12nm, 54.68% end-to-end LLM latency reduction and 2.20x decode throughput. First primary memory-maker technical source in this KB (architecture side).
AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving
Removes the GPU from the center of long-context attention serving: replaces GPU compute dies with 16 HBM-PNM cubes (4x4 UCIe 3.0 mesh) in a multi-chiplet package -- 44 TB/s aggregate bandwidth / 240W package (vs H100 3.35 TB/s / 700W), 15.5x lower attention latency and 6.9x lower energy vs H100 on 1M-token decode attention (also 1.1-2.5x vs Rubin, 1.4-3.4x vs NeuPIMs). DSE finding: per-cube compute saturates ~96 TFLOPS then bandwidth-bound; comms-reordering speedup scales 2.7x->65.4x from 8K->1M sequence length. Names NVIDIA Rubin's GPU-LPU platform as the GPU-centric baseline it argues against. Boundary case (Apr 2026, outside 7-day window) ingested for direct chiplet + memory-centric gap match.
C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
Software framework making the KV cache position-agnostic, compressible AND reusable at once via a lightweight sidecar Extractor learning a composable/compressed KV manifold, so non-prefix KV blocks concatenate and reuse without accuracy collapse. Compression ratio 4x primary (8x/16x also evaluated); up to 17x long-context inference speedup preserving quality; C2KV-4x accuracy beats even Full Recompute on MuSiQue (0.3587 vs 0.3198) and 2WikiMQA (0.4477 vs 0.4018); decode-time latency scaling flattened even at 128k tokens. Algorithm-layer complement to the near-memory KV hardware work.
Is a Compute Crunch Coming? (inference serving-capacity model, calibrated to SemiAnalysis InferenceX)
First-principles prefill(compute-bound)/decode(bandwidth-bound) serving-throughput model CALIBRATED to 111 measured SemiAnalysis InferenceX Kimi K2.5 runs (fitted: compute eff 65%, bandwidth eff 30%, 5ms/step). GB200 NVL72 ~400k tok/s; global capacity 500M-20B tok/s; capacity growth ~3.4x/yr vs demand ~10x/yr => crunch. Denominated in tokens/sec, NO $/token. As-of 2026-05-25.
Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators
Empirical head-to-head of GPUs vs. GroqRack across the prefill (compute-bound) / decode (memory-bound) split using Llama2-7B: GPUs consistently win prefill, GroqRack wins decode TPOT at small batch, GPU decode advantage returns at large batch. Grounds 'which architecture wins which inference phase' in independent (Argonne) benchmarks rather than vendor claims — relevant to both this topic's rack-scale-compute frontier and inference-economics cost-per-phase modeling.
Token Economics Across Traffic Profiles on Dedicated GPUs (measured H200 serving-cost benchmark)
First MEASURED cost-per-token series in this KB: single H200 ($3.44/GPU-hr, Llama-3.3-70B FP8, vLLM 0.24.0), swept batch 1->256, yields $20.32 -> $0.45 per M output tokens (~44x spread on ONE SKU from traffic shape alone), independently reproducing Patil's utilization thesis with dollar-anchored levels. As-of 2026-07-08; MEASURED throughput, cost derived at list rate.