# KV Cache Management

Canonical URL: https://menfem.com/kb/hardware/concepts/kv-cache-management
Knowledge base topic: [Hardware & Computing](https://menfem.com/kb/hardware)
Frontier status: active
Tags: kv-cache, memory-hierarchy, llm-serving, ai-memory-wall, processing-in-memory, hbm4, kv-compression, kv-reuse

---

The KV cache has stopped being a per-request scratch tensor and become **a first-order memory object
in its own right** — one with locality, a lifetime, an owner, and a storage substrate. That
reclassification is the single most useful frame this KB has acquired for the serving-software layer,
because it is the exact point where the hardware memory wall this topic tracks (HBM bandwidth,
processing-in-memory, $/PB) meets the software layer that decides which bytes move.

The mechanism is simple and brutal: during **decode**, every new token re-reads all prior KV state.
Attention therefore becomes a bandwidth- *and* capacity-heavy memory task rather than a compute task
— which is why decode is the phase where memory architecture decides the outcome (see
[Prefill/Decode Disaggregation](./prefill-decode-disaggregation.md)) and why bytes-per-token is the
efficiency term in the inference-cost models on
[Memory Scarcity & Inference Economics](./memory-scarcity-inference-economics.md).

Two 2026 preprints cover the two halves of the response — a taxonomy of what the software layer has
built, and a concrete hardware architecture for putting compute where the hot KV lives.

> **Provenance note.** Both sources are registered in `raw/_sources.json` from a discovery pass with
> **abstracts verified against arXiv but full text not ingested**. There is no `raw/*.md` file for
> either; links below go to arXiv. Claims are therefore capped at what the abstracts state — none of
> the internal methodology, baselines, or workload definitions has been read.

## The survey: four axes, five archetypes

*Li, Wang & Chen, "From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache
Management for LLM Serving", [arXiv:2607.02574](https://arxiv.org/abs/2607.02574), 30 June 2026
(cs.DC / cs.AI / cs.LG).*

The survey classifies **more than thirty** KV-management systems and frameworks along four design
axes, and finds those axes collapse into five recurring architectural archetypes.

| Design axis | What it decides |
|---|---|
| **Locality** | Where cached KV physically sits relative to the compute that needs it |
| **Lifetime** | How long a KV block persists — request-scoped, session-scoped, or durable |
| **Ownership** | Which component holds authority over a block (engine, scheduler, external store) |
| **Substrate** | What medium backs it — HBM, host DRAM, SSD, remote memory pool |

| Archetype | Shape |
|---|---|
| **Local-paged** | KV paged within a single engine's own memory |
| **Disaggregated-pipeline** | KV handed across separated prefill/decode stages |
| **Shared-store** | A common store several engines read from |
| **Memory-pool** | KV drawn from a pooled, disaggregated memory tier |
| **Hybrid-tier** | Mixed tiers with movement between them |

**The survey's sharpest finding:** once workload and hardware are fixed, **ownership accounts for
much of the remaining design variance** among distributed systems. That is a non-obvious claim — it
says the interesting differences between serving systems are governance/authority differences, not
placement differences.

It also **audits current evaluations and identifies seven missing KV-specific measurements**, tying
them to open problems in fault tolerance, isolation, tiered eviction, speculative decoding, MoE
serving, and shared-cache semantics. For a KB that keeps hitting vendor-claimed inference numbers,
an explicit evaluation-methodology gap list is worth more than another benchmark.

## The hardware answer: TokenStack (heterogeneous HBM-PIM)

*Li, Bian, Huang, Sun, Liang & Zhuo, "TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime
for Efficient LLM Inference", [arXiv:2605.05639](https://arxiv.org/abs/2605.05639), 7 May 2026
(cs.AR). Author affiliations not listed on the arXiv abstract page; the source registry attributes
Peking University per author affiliations.*

TokenStack's premise is that uniform HBM-PIM stacks waste resources: **only hot KV blocks benefit
from near-memory compute**, while weights, activations and cold KV mainly need dense storage and
GPU-visible bandwidth. A uniform PIM stack makes every layer pay for PIM logic; a dedicated-PIM
design (their named baseline is AttAcc) recovers capacity but shrinks the HBM bandwidth left for
GPU-side work.

The proposal is **vertical heterogeneity inside the stack**, and it lands directly on the HBM4
architectural shift this KB already tracks:

- Separate each stack into **dense capacity layers** and **PIM-enabled compute layers**.
- Use **HBM4's logic base die as a stack-local control point** managing cross-layer movement without
  host-side overhead — cross-layer DMA, layered address translation, attention-side gather/broadcast
  coordination, and inline quantization during migration.
- On top of that: topology-aware KV placement, workload-aware eviction, and bounded replication to
  keep hot KV near PIM compute while colder state moves to dense layers.

Reported results, on production-derived traces across four models, multi-QPS runs, **versus AttAcc**:

| Metric | Claim |
|---|---|
| Geometric-mean token throughput | **1.62×** |
| SLO-compliant serving capacity | **1.70×** |
| Per-token energy | **−30 to −47%** |

This is the most concrete published answer yet to the KB's standing question of *what C-HBM4E
level-3 near-memory compute is actually for* — the answer being hot-KV attention. It is **an academic
architecture proposal, not a shipping product**, and the baseline is another academic design rather
than a deployed system.

## Two July-2026 additions: shrink the object (software), or remove the GPU from it (hardware)

The KV object is now being attacked from both layers at once, and the two compound. *(Both sources
were ingested 2026-07-23 with raw source files; both were deepened to full text 2026-07-24 via arXiv
HTML — the magnitudes below are now primary reads, not abstract-level.)*

**C²KV — the software layer shrinks and reuses the object itself.**
*Du, Chen, Tang et al., "C²KV: Compressed and Composable KV Cache Reuse for Efficient LLM
Inference", [arXiv:2607.17715](https://arxiv.org/abs/2607.17715), 20 Jul 2026 (Shanghai Jiao Tong
University, Alibaba Group).* C²KV reframes the long-context bottleneck as **the cost of storing and
accessing** the KV cache — not just the compute to build it — which is exactly the memory-wall
framing this page uses. Prior KV-reuse work saves computation but overlooks storage/transfer cost;
naively bolting compression onto *non-prefix* reuse collapses accuracy. C²KV learns a
**position-agnostic, composable, compressed KV manifold** via a lightweight **sidecar Extractor**
(learnable compression tokens + structured attention flow) trained with a compression-concatenation
co-training strategy — all on a **frozen base model**, no base retraining. **Compression ratio: 4×
primary (8× and 16× also evaluated).** The payoff: modular KV blocks concatenate and reuse without
accuracy collapse, up to **17× long-context inference speedup** while preserving generation quality —
per-token decode latency scaling is largely flattened even at 128k tokens. On accuracy (Llama3.1-8B),
**C²KV-4× beats even Full Recompute**: MuSiQue 0.3587 vs 0.3198, 2WikiMQA 0.4477 vs 0.4018, SAMSum
0.3904 vs 0.3652 — while CacheBlend, EPIC and naive reuse trail Full Recompute or collapse outright
(naive reuse falls to 0.2602 on 2WikiMQA). This extends the object along the **lifetime / ownership**
axes the survey named: KV blocks become durable, composable, shareable assets rather than per-request
scratch.

**AMMA — the hardware layer removes the GPU from decode attention entirely.**
*Yu, Ye, Zhou et al., "AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context
Attention Serving", [arXiv:2604.26103](https://arxiv.org/abs/2604.26103), 28 Apr 2026 (UC San Diego,
Columbia, Yonsei, NVIDIA, Samsung Semiconductor).* Where TokenStack adds PIM logic *alongside* the
GPU, AMMA **replaces the GPU compute dies with 16 HBM-PNM cubes** in a 4×4-mesh multi-chiplet package
(UCIe 3.0 D2D links), and names NVIDIA Rubin's GPU-LPU platform as the GPU-centric baseline it argues
against. Package-level: **44 TB/s aggregate bandwidth at 240 W**, against H100's 3.35 TB/s at 700 W.
On 1M-token-context decode attention it reports **15.5× lower attention latency and 6.9× lower energy
vs H100** (also 1.1–2.5× vs Rubin, 1.4–3.4× vs NeuPIMs, depending on sequence length and workload).
The design-space exploration finds per-cube compute **saturates at ~96 TFLOPS** for GQA decode, after
which **bandwidth becomes the bottleneck** — and that the reordered-collective communication scheme
(ReduceScatter replacing AllReduce+AllGather) is what makes 1M-context latency tractable, with its
speedup growing from 2.7× at 8K tokens to **65.4× at 1M tokens**. This is the most aggressive
statement of the memory-centric thesis for the specific KV-driven workload of decode attention — the
hardware counterpart to C²KV's software attack on the same object. See
[Processing-In-Memory](./processing-in-memory.md) for the full near-memory spectrum.

**Why they compound.** A smaller, reusable KV cache (C²KV) changes *what a KV-targeted PIM stack
(TokenStack) or a GPU-free attention chiplet (AMMA) has to hold and move.* The software layer sets
the working-set size; the hardware layer decides where that working set is computed. Neither is a
shipping product — C²KV is an algorithm/systems-software preprint (authors' own benchmarks), AMMA a
simulated architecture vs H100/Rubin/NeuPIMs baselines, not independently replicated.

## Key Claims

- **KV cache is a first-order memory object with locality, lifetime, ownership and substrate — not a
  temporary per-request tensor.** *Evidence: moderate — survey framing, abstract-level only*
  ([arXiv:2607.02574](https://arxiv.org/abs/2607.02574))
- **30+ KV-management systems collapse into five archetypes** (local-paged, disaggregated-pipeline,
  shared-store, memory-pool, hybrid-tier). *Evidence: moderate — survey, abstract-level*
  ([arXiv:2607.02574](https://arxiv.org/abs/2607.02574))
- **Once workload and hardware are fixed, ownership explains much of the remaining design variance.**
  *Evidence: moderate — the survey's own headline finding, not independently checked*
  ([arXiv:2607.02574](https://arxiv.org/abs/2607.02574))
- **Seven KV-specific measurements are missing from current evaluations**, linked to fault tolerance,
  isolation, tiered eviction, speculative decoding, MoE serving and shared-cache semantics.
  *Evidence: moderate* ([arXiv:2607.02574](https://arxiv.org/abs/2607.02574))
- **Only hot KV blocks benefit from near-memory compute; uniform HBM-PIM makes every layer pay for
  PIM logic.** *Evidence: moderate — the motivating argument for TokenStack*
  ([arXiv:2605.05639](https://arxiv.org/abs/2605.05639))
- **HBM4's logic base die can act as a stack-local controller** for cross-layer DMA, address
  translation, gather/broadcast coordination and inline quantization. *Evidence: moderate —
  architecture proposal* ([arXiv:2605.05639](https://arxiv.org/abs/2605.05639))
- **TokenStack: 1.62× geomean token throughput, 1.70× SLO-compliant capacity, 30–47% lower per-token
  energy vs AttAcc.** *Evidence: moderate — simulated/academic result vs an academic baseline,
  abstract-level, full text not read; NOT a product measurement*
  ([arXiv:2605.05639](https://arxiv.org/abs/2605.05639))
- **C²KV: a position-agnostic, composable + compressed KV manifold lets non-prefix KV blocks be
  reused without accuracy collapse — 4× compression (8×/16× also evaluated), up to 17× long-context
  speedup, and C²KV-4× accuracy beats even Full Recompute** (MuSiQue 0.3587 vs 0.3198; 2WikiMQA 0.4477
  vs 0.4018), via a lightweight sidecar Extractor on a frozen base model. Frames the real bottleneck as
  KV storage/transfer cost, not just prefill compute; decode-time latency scaling flattened even at
  128k tokens. *Evidence: moderate — preprint, authors' own benchmarks (Shanghai Jiao Tong University,
  Alibaba Group); full text ingested 2026-07-24* ([C²KV](../../raw/c2kv-compressed-composable-kv-cache-reuse.md))
- **AMMA: replacing GPU compute dies with 16 HBM-PNM cubes gives 44 TB/s / 240 W package bandwidth**
  (vs H100's 3.35 TB/s / 700 W), **15.5× lower attention latency and 6.9× lower energy vs H100** on
  1M-token decode attention (also 1.1–2.5× vs Rubin, 1.4–3.4× vs NeuPIMs), removing the GPU from
  the center of long-context attention serving. DSE: per-cube compute saturates ~96 TFLOPS then
  bandwidth-bound; comms-reordering speedup scales 2.7×→65.4× from 8K→1M tokens. *Evidence: moderate —
  preprint, simulated vs H100/Rubin/NeuPIMs baselines (UC San Diego, Columbia, Yonsei, NVIDIA, Samsung
  Semiconductor); full text ingested 2026-07-24* ([AMMA](../../raw/amma-multichiplet-memory-centric-1m-context-attention.md))

## Open Questions

- What are the seven missing measurements, specifically? (Requires ingesting the survey's full text —
  the abstract names only the problem areas they map to.)
- Does the "ownership dominates" finding hold outside the paper's sample of 30+ systems?
- Does TokenStack's heterogeneous-stack idea survive contact with DRAM yield and thermal reality, or
  is per-layer differentiation a packaging non-starter?
- Is AttAcc the right baseline, or does TokenStack's advantage shrink against a well-tuned
  local-paged GPU serving stack?
- Do any of the three HBM makers show a heterogeneous-layer stack on a public roadmap, or does this
  stay academic?
- Which archetype do the hyperscaler custom stacks (TPU, Trainium, MTIA, Maia) actually use?
- Does C²KV's non-prefix reuse hold up under adversarial RAG / multi-document retrieval, and at what
  compression ratio above 4× (8×/16× are evaluated but not headlined) does the accuracy-beats-Full-Recompute
  result start to erode?
- Does AMMA's GPU-free attention chiplet attract any silicon commitment, or does removing the GPU
  compute die from the attention path stay a paper architecture — and does the ~96 TFLOPS/cube DSE
  saturation point hold on a real UCIe 3.0 implementation rather than a simulated one?

## Related Concepts

- [Processing-In-Memory & Memory-Centric Computing](./processing-in-memory.md) — TokenStack is the first concrete HBM4-era PIM architecture with a named workload
- [HBM4 Memory Architecture](./hbm4-memory-architecture.md) — the logic base die is what makes stack-local control possible
- [Prefill/Decode Disaggregation](./prefill-decode-disaggregation.md) — KV re-reading is *the* decode-phase cost
- [Memory Scarcity & Inference Economics](./memory-scarcity-inference-economics.md) — KV compression is the efficiency term those models assume

## Backlinks

*Pages that reference this concept:*
- [Processing-In-Memory](./processing-in-memory.md)
- [HBM4 Memory Architecture](./hbm4-memory-architecture.md)
- [C²KV — Compressed & Composable KV Reuse](../../raw/c2kv-compressed-composable-kv-cache-reuse.md)
- [AMMA — Multi-Chiplet Memory-Centric Attention](../../raw/amma-multichiplet-memory-centric-1m-context-attention.md)

## Changelog

- **2026-07-24** — Deepened **C²KV** (arXiv:2607.17715) and **AMMA** (arXiv:2604.26103) from
  abstract-level to full text (arXiv HTML). C²KV: compression ratio 4× (8×/16× also evaluated), the
  accuracy table (C²KV-4× beats Full Recompute on MuSiQue/2WikiMQA/SAMSum), decode-latency scaling
  flattened to 128k tokens. AMMA: package-level 44 TB/s / 240 W (vs H100 3.35 TB/s / 700 W),
  per-baseline detail (vs Rubin 1.1–2.5×, vs NeuPIMs 1.4–3.4×), and the DSE finding (per-cube compute
  saturates ~96 TFLOPS then bandwidth-bound; comms-reordering speedup 2.7×→65.4× from 8K→1M tokens).
  Institutions resolved: C²KV = Shanghai Jiao Tong University + Alibaba Group; AMMA = UC San Diego,
  Columbia, Yonsei, NVIDIA, Samsung Semiconductor. No new sources.
- **2026-07-23** — Added **C²KV** (arXiv:2607.17715, software layer — composable + compressed non-prefix KV reuse, up to 17× long-context speedup) and **AMMA** (arXiv:2604.26103, hardware layer — HBM-PNM cubes replace GPU compute dies for decode attention, 15.5× lower latency vs H100 at 1M context). Framed them as the two-sided attack on the KV object: software shrinks/reuses it, hardware removes the GPU from computing over it, and the two compound. Both are preprints with raw source files; abstract-level claims capped at *moderate*. +2 sources.
- **2026-07-22** — Initial compilation from the KV-cache-management survey (arXiv:2607.02574) and TokenStack (arXiv:2605.05639). Both abstract-level only — no raw source file exists for either; provenance noted inline and all claims capped at *moderate*.

## Sources

- kv-cache-management-survey-tensor-buffer-distributed-memory
- tokenstack-heterogeneous-hbm-pim-llm-inference
- c2kv-compressed-composable-kv-cache-reuse
- amma-multichiplet-memory-centric-1m-context-attention

---

Cite as: MenFem Knowledge Base — https://menfem.com/kb/hardware/concepts/kv-cache-management