Quantized KV Cache: Stored Small, Computed Large

When a model generates text it keeps a running memory of everything it has read so far — the key-value (KV) cache. At long context that memory gets large: DeepSeek gives the size of one 128K-token request on DeepSeek-V3.2 as 8.72 GiB of cache, which is enough to run a GPU out of memory or force small batches (FlashMLA FP8 deep dive). The obvious answer is to store the cache in fewer bits. That decision belongs to the serving rung. What belongs here is what happens next: the smaller numbers have to be turned back into bigger ones before the maths can use them, and that conversion is not free.

DeepSeek's format keeps 512 of each token's 576 values in 8-bit floating point (FP8) with four scale factors, and leaves the 64 position-encoding values in BF16 because they "are sensitive to precision loss". That is 656 bytes per token instead of 1,152 — about 43% smaller (the percentage is derived in the close-read from the source's own layout). But inside the kernel every FP8 value is converted back to BF16 and both matrix multiplications run in BF16. On H800, which has no direct FP8→BF16 instruction, the conversion takes four steps per value and costs more cycles than the matrix maths it feeds. The cache is compressed; the maths is not. That gap is where the non-matmul bottleneck came from on Hopper.

ReQAT points the other way: go all the way down to 4-bit floating point for weights, activations and the KV cache (W4A4KV4), on hardware that has a native FP4 path. There the maths itself runs in the small format, and the problem moves from conversion cost to accuracy — see FP4 reasoning quantization.

Key Claims

• A long-context KV cache is big enough to set batch size. 576 × 2 × 62 × 128 × 1024 bytes = 8.72 GiB for one 128K-token request; the arithmetic reproduces exactly. Evidence: moderate (vendor technical report, arithmetic checked) (FlashMLA)
• Tile-wise FP8 with an unquantized position slice cuts cache bytes ~43%. 656 vs 1,152 bytes per token. The README's wire-format description agrees with the blog's. Evidence: moderate (FlashMLA)
• No accuracy number was published for the FP8 cache format. The source argues it preserves quality by design (fine 1×128 tiles, position values kept in BF16) and reports no perplexity or benchmark score. A cache-quantization change without a quality figure is half a result. Evidence: this is an absence, recorded from the close-read's limitations (FlashMLA)
• Going to 4-bit KV (with 4-bit weights and activations) can hold reasoning accuracy — on one of two models tested. See FP4 reasoning quantization. Evidence: strong for the measured results (ICML 2026, peer-reviewed), limited to two distilled models (ReQAT)

Benchmarks & Data

Item	Value	As-of	Source
KV cache, one 128K-token request (DeepSeek-V3.2)	8.72 GiB	2025-09-29	FlashMLA
Bytes per token, FP8 format vs BF16	656 vs 1,152 (≈43% smaller)	2025-09-29	FlashMLA
Conversion cost vs matmul cost per token (H800)	≥50 vs 34 cycles	2025-09-29	FlashMLA
Crossover kernel beats dense decode past	~3,000 tokens of context	2025-09-29	FlashMLA
Q-FIT overhead over native NVFP4	4–5%	2026-06-14	ReQAT

Open Questions

• What does DeepSeek's FP8 cache cost in accuracy? Nothing in this KB says.
• Where is the crossover point between "store in FP8, compute in BF16" and "compute in FP4 natively"? The two sources run on different chips and different models, so they cannot be set side by side as a comparison.

Related Concepts

• The non-matmul bottleneck — the conversion step is one of its causes
• FP4 reasoning quantization — the 4-bit, compute-in-the-small-format route
• KV-Cache Compression (serving rung) — the storage decision one layer up
• KV Cache Management (serving rung) — eviction, reuse and placement

Backlinks

Pages that reference this concept:

• The non-matmul bottleneck
• FP4 reasoning quantization
• DeepSeek