FP4 Reasoning Quantization

Running a model in 4-bit floating point (FP4) instead of 16-bit makes it several times faster on chips built for it, but reasoning models have been hurt badly by it. ReQAT's contribution is to say where the damage lands. It is not spread evenly across the answer. It lands on the tokens the model is most sure of — a single digit, an operator, a variable name — where the model has one clear choice and a small nudge from rounding noise is enough to tip it onto a wrong one. Prose tokens, where several continuations are all fine, absorb the same noise harmlessly. One wrong digit early in a chain of reasoning then carries through to a wrong final answer (ReQAT).

The paper tests that directly, which is what makes it a mechanism rather than a speed table: add noise only at the confident positions and accuracy drops sharply; add the same noise only at the uncertain positions and it barely moves. The fix follows from the finding — a training method that spends its updates on those confident positions (trace-aligned quantization-aware training), sharpens confidence there, and starts from a quantization-friendly calibration of the KV cache.

The honest summary is narrower than the abstract. On R1-Qwen-14B, full 4-bit weights, activations and KV cache (NVFP4 W4A4KV4) with ReQAT scored 65.94% on AIME-120 against 65.46% for full-precision fine-tuning — at parity or slightly above. On R1-Llama-8B it trailed full precision on all three benchmarks, by 6.9 points on AIME-120. "Recovers and surpasses full precision" is a 14B result, not a general one.

Why it moves the token price — and the task price

Throughput went up 3.9× on NVIDIA DGX Spark and 3.1× on B200 against BF16. More tokens per second from the same box is a cheaper token. But this source matters for the through-line for a second reason: a quantized model that is wrong more often can cost more per finished task even while each token is cheaper, because the task has to be retried or checked. ReQAT is one of the few quantization results that reports the accuracy leg alongside the speed leg, on reasoning tasks — which is what lets a reader ask about cost per correct answer rather than cost per token.

DGX Spark is a machine a person can buy. A 3.9× speedup measured on it is a local-inference number, which is the cloud-versus-local question this site's direction is about.

Key Claims

• FP4 reasoning failure concentrates on low-entropy tokens. Quantization flattens the model's probability distribution at confident positions (tail-mass ratio above 1) even when the top choice is unchanged — greedy decoding hides the damage, sampling exposes it. The noise-injection experiment isolates the effect. Evidence: strong (ICML 2026, peer-reviewed, with a separable test) (ReQAT)
• 4-bit weights, activations and KV cache can match full-precision fine-tuning on a 14B reasoning model. 65.94% vs 65.46% on AIME-120 (R1-Qwen-14B, NVFP4 W4A4KV4); 68.02% with MXFP4 weight-only (W4A16). Evidence: strong (peer-reviewed), single model (ReQAT)
• It does not hold on the 8B model. R1-Llama-8B, NVFP4 W4A4KV4, 350M training tokens: GSM8K 89.85% vs 91.15%, MATH-500 90.53% vs 92.18%, AIME-120 41.85% vs 48.75% (ReQAT vs BF16 fine-tuning). Evidence: strong (peer-reviewed; the paper's own table) (ReQAT)
• The speedups belong partly to the chip. 3.9× (DGX Spark) and 3.1× (B200) are measured on NVIDIA Blackwell hardware with native microscaled FP4 support; they are as much a property of that silicon path as of the method. Evidence: strong for the measurement, the attribution caveat is the close-read's (ReQAT)
• The method inherits the quality of its training traces. The authors state that when reasoning traces are weak or noisy, trace-aligned training may give limited gains. Evidence: strong (authors' own stated limitation) (ReQAT)

Benchmarks & Data

Model / setting	Result	As-of	Source
R1-Qwen-14B, BF16 baseline, AIME-120	56.83%	2026-06-14	ReQAT
R1-Qwen-14B, BF16 fine-tuning, AIME-120	65.46%	2026-06-14	ReQAT
R1-Qwen-14B, NVFP4 W4A4KV4 + ReQAT, AIME-120	65.94%	2026-06-14	ReQAT
R1-Qwen-14B, MXFP4 W4A16 + ReQAT, AIME-120	68.02%	2026-06-14	ReQAT
R1-Llama-8B, NVFP4 W4A4KV4 + ReQAT vs BF16 FT, AIME-120	41.85% vs 48.75%	2026-06-14	ReQAT
Throughput vs BF16, DGX Spark / B200	3.9× / 3.1×	2026-06-14	ReQAT

Throughput setting: 1K requests, 512-token prompts, batch up to 256, generation up to 8K/16K tokens.

Open Questions

• Why does it work at 14B and fail at 8B? The paper gives two data points, not a trend.
• Both models are DeepSeek-R1 distills. Nothing here speaks to frontier or non-distilled reasoning models.
• What does 3.9× on DGX Spark come to in cost per correct answer, once the accuracy gap on smaller models is counted? The source gives the two legs separately; nobody has joined them.

Related Concepts

• Quantized KV cache: stored small, computed large — the FP8 store-small-compute-large route this contrasts with
• The non-matmul bottleneck — the speed side of numeric-format choices
• Low-Bit Quantization (serving rung) — weight-only post-training quantization and how it interacts with speculative decoding

Backlinks

Pages that reference this concept:

• Quantized KV cache
• NVIDIA