KV Cache Management

The KV cache has stopped being a per-request scratch tensor and become a first-order memory object in its own right — one with locality, a lifetime, an owner, and a storage substrate. That reclassification is the single most useful frame this KB has acquired for the serving-software layer, because it is the exact point where the hardware memory wall this topic tracks (HBM bandwidth, processing-in-memory, $/PB) meets the software layer that decides which bytes move.

The mechanism is simple and brutal: during decode, every new token re-reads all prior KV state. Attention therefore becomes a bandwidth- and capacity-heavy memory task rather than a compute task — which is why decode is the phase where memory architecture decides the outcome (see Prefill/Decode Disaggregation) and why bytes-per-token is the efficiency term in the inference-cost models on Memory Scarcity & Inference Economics.

Two 2026 preprints cover the two halves of the response — a taxonomy of what the software layer has built, and a concrete hardware architecture for putting compute where the hot KV lives.

Provenance note. Both sources are registered in raw/_sources.json from a discovery pass with abstracts verified against arXiv but full text not ingested. There is no raw/*.md file for either; links below go to arXiv. Claims are therefore capped at what the abstracts state — none of the internal methodology, baselines, or workload definitions has been read.

The survey: four axes, five archetypes

Li, Wang & Chen, "From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving", arXiv:2607.02574, 30 June 2026 (cs.DC / cs.AI / cs.LG).

The survey classifies more than thirty KV-management systems and frameworks along four design axes, and finds those axes collapse into five recurring architectural archetypes.

The survey's sharpest finding: once workload and hardware are fixed, ownership accounts for much of the remaining design variance among distributed systems. That is a non-obvious claim — it says the interesting differences between serving systems are governance/authority differences, not placement differences.

It also audits current evaluations and identifies seven missing KV-specific measurements, tying them to open problems in fault tolerance, isolation, tiered eviction, speculative decoding, MoE serving, and shared-cache semantics. For a KB that keeps hitting vendor-claimed inference numbers, an explicit evaluation-methodology gap list is worth more than another benchmark.

The hardware answer: TokenStack (heterogeneous HBM-PIM)

Li, Bian, Huang, Sun, Liang & Zhuo, "TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inference", arXiv:2605.05639, 7 May 2026 (cs.AR). Author affiliations not listed on the arXiv abstract page; the source registry attributes Peking University per author affiliations.

TokenStack's premise is that uniform HBM-PIM stacks waste resources: only hot KV blocks benefit from near-memory compute, while weights, activations and cold KV mainly need dense storage and GPU-visible bandwidth. A uniform PIM stack makes every layer pay for PIM logic; a dedicated-PIM design (their named baseline is AttAcc) recovers capacity but shrinks the HBM bandwidth left for GPU-side work.

The proposal is vertical heterogeneity inside the stack, and it lands directly on the HBM4 architectural shift this KB already tracks:

• Separate each stack into dense capacity layers and PIM-enabled compute layers.
• Use HBM4's logic base die as a stack-local control point managing cross-layer movement without host-side overhead — cross-layer DMA, layered address translation, attention-side gather/broadcast coordination, and inline quantization during migration.
• On top of that: topology-aware KV placement, workload-aware eviction, and bounded replication to keep hot KV near PIM compute while colder state moves to dense layers.

Reported results, on production-derived traces across four models, multi-QPS runs, versus AttAcc:

This is the most concrete published answer yet to the KB's standing question of what C-HBM4E level-3 near-memory compute is actually for — the answer being hot-KV attention. It is an academic architecture proposal, not a shipping product, and the baseline is another academic design rather than a deployed system.

Two July-2026 additions: shrink the object (software), or remove the GPU from it (hardware)

The KV object is now being attacked from both layers at once, and the two compound. (Both sources were ingested 2026-07-23 with raw source files; both were deepened to full text 2026-07-24 via arXiv HTML — the magnitudes below are now primary reads, not abstract-level.)

C²KV — the software layer shrinks and reuses the object itself. Du, Chen, Tang et al., "C²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference", arXiv:2607.17715, 20 Jul 2026 (Shanghai Jiao Tong University, Alibaba Group). C²KV reframes the long-context bottleneck as the cost of storing and accessing the KV cache — not just the compute to build it — which is exactly the memory-wall framing this page uses. Prior KV-reuse work saves computation but overlooks storage/transfer cost; naively bolting compression onto non-prefix reuse collapses accuracy. C²KV learns a position-agnostic, composable, compressed KV manifold via a lightweight sidecar Extractor (learnable compression tokens + structured attention flow) trained with a compression-concatenation co-training strategy — all on a frozen base model, no base retraining. Compression ratio: 4× primary (8× and 16× also evaluated). The payoff: modular KV blocks concatenate and reuse without accuracy collapse, up to 17× long-context inference speedup while preserving generation quality — per-token decode latency scaling is largely flattened even at 128k tokens. On accuracy (Llama3.1-8B), C²KV-4× beats even Full Recompute: MuSiQue 0.3587 vs 0.3198, 2WikiMQA 0.4477 vs 0.4018, SAMSum 0.3904 vs 0.3652 — while CacheBlend, EPIC and naive reuse trail Full Recompute or collapse outright (naive reuse falls to 0.2602 on 2WikiMQA). This extends the object along the lifetime / ownership axes the survey named: KV blocks become durable, composable, shareable assets rather than per-request scratch.

AMMA — the hardware layer removes the GPU from decode attention entirely. Yu, Ye, Zhou et al., "AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving", arXiv:2604.26103, 28 Apr 2026 (UC San Diego, Columbia, Yonsei, NVIDIA, Samsung Semiconductor). Where TokenStack adds PIM logic alongside the GPU, AMMA replaces the GPU compute dies with 16 HBM-PNM cubes in a 4×4-mesh multi-chiplet package (UCIe 3.0 D2D links), and names NVIDIA Rubin's GPU-LPU platform as the GPU-centric baseline it argues against. Package-level: 44 TB/s aggregate bandwidth at 240 W, against H100's 3.35 TB/s at 700 W. On 1M-token-context decode attention it reports 15.5× lower attention latency and 6.9× lower energy vs H100 (also 1.1–2.5× vs Rubin, 1.4–3.4× vs NeuPIMs, depending on sequence length and workload). The design-space exploration finds per-cube compute saturates at ~96 TFLOPS for GQA decode, after which bandwidth becomes the bottleneck — and that the reordered-collective communication scheme (ReduceScatter replacing AllReduce+AllGather) is what makes 1M-context latency tractable, with its speedup growing from 2.7× at 8K tokens to 65.4× at 1M tokens. This is the most aggressive statement of the memory-centric thesis for the specific KV-driven workload of decode attention — the hardware counterpart to C²KV's software attack on the same object. See Processing-In-Memory for the full near-memory spectrum.

Why they compound. A smaller, reusable KV cache (C²KV) changes what a KV-targeted PIM stack (TokenStack) or a GPU-free attention chiplet (AMMA) has to hold and move. The software layer sets the working-set size; the hardware layer decides where that working set is computed. Neither is a shipping product — C²KV is an algorithm/systems-software preprint (authors' own benchmarks), AMMA a simulated architecture vs H100/Rubin/NeuPIMs baselines, not independently replicated.

Key Claims

• KV cache is a first-order memory object with locality, lifetime, ownership and substrate — not a temporary per-request tensor. Evidence: moderate — survey framing, abstract-level only (arXiv:2607.02574)
• 30+ KV-management systems collapse into five archetypes (local-paged, disaggregated-pipeline, shared-store, memory-pool, hybrid-tier). Evidence: moderate — survey, abstract-level (arXiv:2607.02574)
• Once workload and hardware are fixed, ownership explains much of the remaining design variance. Evidence: moderate — the survey's own headline finding, not independently checked (arXiv:2607.02574)
• Seven KV-specific measurements are missing from current evaluations, linked to fault tolerance, isolation, tiered eviction, speculative decoding, MoE serving and shared-cache semantics. Evidence: moderate (arXiv:2607.02574)
• Only hot KV blocks benefit from near-memory compute; uniform HBM-PIM makes every layer pay for PIM logic. Evidence: moderate — the motivating argument for TokenStack (arXiv:2605.05639)
• HBM4's logic base die can act as a stack-local controller for cross-layer DMA, address translation, gather/broadcast coordination and inline quantization. Evidence: moderate — architecture proposal (arXiv:2605.05639)
• TokenStack: 1.62× geomean token throughput, 1.70× SLO-compliant capacity, 30–47% lower per-token energy vs AttAcc. Evidence: moderate — simulated/academic result vs an academic baseline, abstract-level, full text not read; NOT a product measurement (arXiv:2605.05639)
• C²KV: a position-agnostic, composable + compressed KV manifold lets non-prefix KV blocks be reused without accuracy collapse — 4× compression (8×/16× also evaluated), up to 17× long-context speedup, and C²KV-4× accuracy beats even Full Recompute (MuSiQue 0.3587 vs 0.3198; 2WikiMQA 0.4477 vs 0.4018), via a lightweight sidecar Extractor on a frozen base model. Frames the real bottleneck as KV storage/transfer cost, not just prefill compute; decode-time latency scaling flattened even at 128k tokens. Evidence: moderate — preprint, authors' own benchmarks (Shanghai Jiao Tong University, Alibaba Group); full text ingested 2026-07-24 (C²KV)
• AMMA: replacing GPU compute dies with 16 HBM-PNM cubes gives 44 TB/s / 240 W package bandwidth (vs H100's 3.35 TB/s / 700 W), 15.5× lower attention latency and 6.9× lower energy vs H100 on 1M-token decode attention (also 1.1–2.5× vs Rubin, 1.4–3.4× vs NeuPIMs), removing the GPU from the center of long-context attention serving. DSE: per-cube compute saturates ~96 TFLOPS then bandwidth-bound; comms-reordering speedup scales 2.7×→65.4× from 8K→1M tokens. Evidence: moderate — preprint, simulated vs H100/Rubin/NeuPIMs baselines (UC San Diego, Columbia, Yonsei, NVIDIA, Samsung Semiconductor); full text ingested 2026-07-24 (AMMA)

Open Questions

• What are the seven missing measurements, specifically? (Requires ingesting the survey's full text — the abstract names only the problem areas they map to.)
• Does the "ownership dominates" finding hold outside the paper's sample of 30+ systems?
• Does TokenStack's heterogeneous-stack idea survive contact with DRAM yield and thermal reality, or is per-layer differentiation a packaging non-starter?
• Is AttAcc the right baseline, or does TokenStack's advantage shrink against a well-tuned local-paged GPU serving stack?
• Do any of the three HBM makers show a heterogeneous-layer stack on a public roadmap, or does this stay academic?
• Which archetype do the hyperscaler custom stacks (TPU, Trainium, MTIA, Maia) actually use?
• Does C²KV's non-prefix reuse hold up under adversarial RAG / multi-document retrieval, and at what compression ratio above 4× (8×/16× are evaluated but not headlined) does the accuracy-beats-Full-Recompute result start to erode?
• Does AMMA's GPU-free attention chiplet attract any silicon commitment, or does removing the GPU compute die from the attention path stay a paper architecture — and does the ~96 TFLOPS/cube DSE saturation point hold on a real UCIe 3.0 implementation rather than a simulated one?

Related Concepts

• Processing-In-Memory & Memory-Centric Computing — TokenStack is the first concrete HBM4-era PIM architecture with a named workload
• HBM4 Memory Architecture — the logic base die is what makes stack-local control possible
• Prefill/Decode Disaggregation — KV re-reading is the decode-phase cost
• Memory Scarcity & Inference Economics — KV compression is the efficiency term those models assume

Backlinks

Pages that reference this concept:

• Processing-In-Memory
• HBM4 Memory Architecture
• C²KV — Compressed & Composable KV Reuse
• AMMA — Multi-Chiplet Memory-Centric Attention

Changelog

• 2026-07-24 — Deepened C²KV (arXiv:2607.17715) and AMMA (arXiv:2604.26103) from abstract-level to full text (arXiv HTML). C²KV: compression ratio 4× (8×/16× also evaluated), the accuracy table (C²KV-4× beats Full Recompute on MuSiQue/2WikiMQA/SAMSum), decode-latency scaling flattened to 128k tokens. AMMA: package-level 44 TB/s / 240 W (vs H100 3.35 TB/s / 700 W), per-baseline detail (vs Rubin 1.1–2.5×, vs NeuPIMs 1.4–3.4×), and the DSE finding (per-cube compute saturates ~96 TFLOPS then bandwidth-bound; comms-reordering speedup 2.7×→65.4× from 8K→1M tokens). Institutions resolved: C²KV = Shanghai Jiao Tong University + Alibaba Group; AMMA = UC San Diego, Columbia, Yonsei, NVIDIA, Samsung Semiconductor. No new sources.
• 2026-07-23 — Added C²KV (arXiv:2607.17715, software layer — composable + compressed non-prefix KV reuse, up to 17× long-context speedup) and AMMA (arXiv:2604.26103, hardware layer — HBM-PNM cubes replace GPU compute dies for decode attention, 15.5× lower latency vs H100 at 1M context). Framed them as the two-sided attack on the KV object: software shrinks/reuses it, hardware removes the GPU from computing over it, and the two compound. Both are preprints with raw source files; abstract-level claims capped at moderate. +2 sources.
• 2026-07-22 — Initial compilation from the KV-cache-management survey (arXiv:2607.02574) and TokenStack (arXiv:2605.05639). Both abstract-level only — no raw source file exists for either; provenance noted inline and all claims capped at moderate.