VeriCache: Turning Lossy KV Cache into Lossless LLM Inference

Abstract

KV-cache compression (token dropping and quantization) cuts the memory footprint that scales linearly with context length, but compressed-KV decoding diverges from full-cache decoding as generation proceeds — breaking exact-output tasks like code generation and tool calling. VeriCache turns any such lossy compressor into a lossless system by treating compressed-KV decoding as a drafter and verifying its tokens against the full KV cache, recovering identical output while keeping most of the compression speedup.

Key Contributions

• Compression-as-drafter, full-KV-as-verifier. Draft tokens one at a time from the compressed KV in GPU HBM (HBM-bandwidth-bound), then verify multiple tokens in parallel against the full KV loaded from CPU/remote storage (interconnect + FLOPs bound). Accept tokens matching full-KV predictions; correct mismatches. Output is exact.
• Cross-resource staggering. Rather than lock-step (all draft, then all verify), VeriCache interleaves different requests' draft and verify phases so HBM and PCIe/network links stay saturated simultaneously — the parallelization that makes verification nearly free.
• Uniform compressor interface supporting 7 existing methods as drop-in drafters: token-dropping (KVzip, KVzap, ExpectedAttention, SnapKV) and quantization (KIVI, KVQuant, RotateKV). The interface exposes only dropped-token positions and bits-per-element.
• Two deployment pipelines: long-context decoding (full KV on CPU) and remote prefix caching (compressed cache over slow network, full cache over fast local link).
• Composes with small-model speculative decoding (EAGLE, MTP) as an orthogonal axis.

Methodology

Draft/verify with the same model weights, differing only in KV fidelity — which is why acceptance is extraordinarily high (the drafter is a faithful low-memory copy, not a separate small model). Verification uses standard parallel checking and is exact/distribution-preserving. Hardware tested: NVIDIA RTX PRO 6000 (96GB HBM, PCIe Gen5 ×16 @ 64GB/s) and 2× H100 NVL (94GB); remote setup uses a 40GB/s local vs 1.2GB/s remote split. Models: Mistral-24B, Qwen-32B (+8B/14B ablations), Llama-70B, Llama-3.1-8B-1M; contexts up to ~500K tokens in the bandwidth analysis.

Results

Throughput vs full-KV baseline — long-context decoding (Pipeline 1):

• Llama-70B 2.73× (256 vs 102 tok/s), Mistral-24B 2.41×, Qwen-32B 1.92×; peak 4.26× (Qwen-32B) when composed with a traditional drafter.

Remote prefix caching (Pipeline 2):

• Llama-70B 2.11× (485 vs 240 tok/s), Mistral-24B 1.75×, Qwen-32B 1.33×.

Quality preservation:

• Output identical to full-KV decoding, KL < 0.01 nats (attributed to hardware nondeterminism), 100% completion on long-generation tasks. Contrast: KVzip at 4× compression shows ~14.4 nats KL on Llama-70B, and lossy methods lose 10–30 pp on function-call accuracy.

Acceptance:

• At 4× compression, draft length 30: ~0.8 acceptance rate, ~25–40 accepted tokens per verification round (vs only 2–3 for typical small-model drafters), so verification amortizes far better.

Composition:

• With EAGLE/MTP: up to 4.35× ideal speedup (vs 3.50× VeriCache alone, 1.78× EAGLE alone) — orthogonal optimization dimensions.

Memory example: Qwen-32B at 100K context ≈ 15GB full KV (on CPU) + ~1.5–3.75GB compressed KV (on GPU).

Limitations

Author-stated: keeps the full KV on CPU/storage in addition to the compressed GPU copy, so storage overhead rises; draft length is static per workload (adaptive per-request policies would help heterogeneous compressors); existing compressors optimize serving accuracy, not acceptance-rate at long horizons; current scope assumes uniform bit-precision across layers/tokens, one compression method per deployment, and identical per-layer-head token-drop counts; other lossy techniques (e.g. CacheBlend KV reuse) are not yet covered by the draft-verify approach. Preprint, self-measured, no independent reproduction.

Relation to the KB

Opens KV-cache coverage, previously a named-but-empty thesis area (the KB held Speculative Decoding and Low-Bit Quantization concepts but no KV-cache source). Reinforces the §3 "levers compound" pattern in a stronger form: VeriCache makes compression and speculation the same mechanism, and further stacks with EAGLE. Also a concrete instance of the §4 attribution point — a pure software/serving change (no new weights, no new silicon) that recovers exact quality at multiples of throughput.

Source: arXiv:2605.17613 — VeriCache: Turning Lossy KV Cache into Lossless LLM Inference, Yao, Shen, Du, Feng, Seo, Zhang, Huang, Liu, Lu, Jiang, 17 May 2026. Abstract + full HTML5 retrieved 2026-07-23.