C²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

Abstract (verbatim)

"Long-context inference is central to modern large language model (LLM) applications such as retrieval-augmented generation and multi-document reasoning. To mitigate the growing inference cost, recent work has explored key-value (KV) cache reuse to reduce redundant prefill computation. However, existing reuse methods primarily focus on computation savings and overlook a critical bottleneck in long-context LLM serving: the cost of storing and accessing large KV caches. While KV compression appears to be a natural complement, naively combining compression with non-prefix KV reuse often leads to severe accuracy degradation. In this work, we propose C²KV, a unified framework for non-prefix KV reuse that jointly optimizes KV extraction and inference-time concatenation. C²KV learns a composable and compressed KV cache manifold that is explicitly designed to be position-agnostic. Our approach introduces a lightweight sidecar Extractor with learnable compression tokens and a structured attention flow, enabling modular KV representations that can be flexibly reused and concatenated without modifying the frozen base model. We further employ a compression-concatenation co-training strategy to align extraction-time representations with their downstream reuse behavior. Extensive experiments across multiple long-context benchmarks and model families demonstrate that C²KV significantly reduces KV cache storage and transfer costs, achieving up to 17× inference speedup under long contexts, while preserving generation quality."

Key Contributions

• Frames the real long-context bottleneck as the cost of storing and accessing the KV cache — not just the compute to build it. That is the memory-wall framing at the algorithm layer, matching how this KB treats the KV cache as a first-order memory object.
• Non-prefix reuse that survives compression: a position-agnostic, composable KV manifold lets modular KV blocks be concatenated and reused without the accuracy collapse that naive compression-plus-reuse causes.
• Mechanism: a lightweight sidecar Extractor with learnable compression tokens + structured attention flow + a compression-concatenation co-training strategy — all on a frozen base model (no base retraining).

Results (full-text magnitudes — deepened 2026-07-24 from the arXiv HTML)

(Preprint; authors' own benchmarks, Llama3.1-8B on long-context tasks.)

• Compression ratio: 4× primary; 8× and 16× also evaluated — the "how aggressive" number the abstract-level ingest was missing.
• Up to 17× inference speedup under long contexts; per-token decode latency shows "only a mild increase even at 128k tokens" (decode-time scaling largely flattened).
• Minimal TTFT while holding accuracy — occupies the "optimal upper-left" of the efficiency-accuracy frontier.
• Accuracy vs baselines (Llama3.1-8B, C²KV-4× vs Full Recompute and prior reuse methods):
  • MuSiQue: 0.3587 (C²KV-4×) vs 0.3198 (Full) — beats even full recompute; CacheBlend 0.2720, EPIC 0.2498.
  • 2WikiMQA: 0.4477 vs 0.4018 (Full); naive reuse collapses to 0.2602.
  • SAMSum: 0.3904 vs 0.3652 (Full).
• Outperforms CacheBlend, EPIC, naive reuse, and naive SnapKV+reuse across LongBench tasks — the position-agnostic manifold is what avoids the accuracy collapse that naive compression-plus-non-prefix-reuse suffers.

Why it matters for this KB

• A software-layer complement to the hardware KV work (TokenStack, AMMA, StreamDQ): the same memory-wall pressure the near-memory architectures attack in silicon, C²KV attacks by shrinking and reusing the KV object itself. The two layers compound — a smaller, reusable KV cache changes what a KV-targeted PIM stack has to hold.
• Extends the KV-cache-as-memory-object frontier along the "lifetime / ownership" axes the survey named: KV blocks become durable, composable, shareable assets rather than per-request scratch.

Scope note: this is an algorithm/systems-software paper, not a hardware architecture. Ingested because KV-cache storage/transfer cost is squarely on the memory↔compute spine this topic tracks, and it is genuinely new (2026-07-20). Deepened 2026-07-24 with full-text magnitudes (4/8/16× compression, the accuracy table, decode-scaling behaviour).

Source: C²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference — Du, Chen, Tang et al. (Shanghai Jiao Tong University, Alibaba Group). Full text ingested 2026-07-24 via the arXiv HTML (2607.17715v1); compression ratios and accuracy table captured. Provenance [preprint — authors' own benchmarks].