From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
First systematic taxonomy of 30+ KV-cache-management systems across four design dimensions (locality, lifetime, ownership, substrate) and five architectural patterns (local-paged, disaggregated-pipeline, shared-store, memory-pool, hybrid-tier); identifies seven evaluation-methodology gaps tied to fault tolerance, tiered eviction, speculative decoding, and MoE serving. Best current map of how the serving-software layer answers the memory-wall problem this topic already tracks at the hardware layer.
KV Cache Management: From Tensor Buffer to Distributed Memory Hierarchy
The framing that matters
"The key-value (KV) cache has become a first-order memory object in LLM serving rather than a temporary per-request tensor."
That sentence is the whole argument for a serving rung. The KV cache stopped being an implementation detail inside a model and became a distributed-systems problem with its own architecture — which is why serving mechanics could not stay filed under models or hardware.
The taxonomy
Thirty-plus KV-management systems classified on four axes: locality, lifetime, ownership, substrate. Those yield five architectural archetypes — local-paged, disaggregated-pipeline, shared-store, memory-pool, hybrid-tier.
The load-bearing finding is about which axis dominates: once workload and hardware are fixed, ownership accounts for much of the remaining design variance among distributed systems. Not locality, not substrate — who owns the cache. That is a design-level claim, and it is the kind of thing a survey earns its place with.
The gap it names
The survey audits current evaluations and identifies seven missing KV-specific measurements, tying them to open problems in fault tolerance, isolation, tiered eviction, speculative decoding, MoE serving, and shared-cache semantics.
That connects straight to evals: the field is measuring KV systems without seven of the measurements it needs. A benchmark that cannot see tiered eviction cannot rank tiered-eviction designs.
Provenance
preprint, arXiv 2607.02574, submitted 2026-06-30. A survey — structure rather than a new measurement, and graded accordingly. Ingested 2026-08-06 to close a gap: this source had been listed since before the rung split with no close read, so its taxonomy was invisible to everything downstream.