ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Training
FP4 reasoning failure concentrates on LOW-ENTROPY tokens (digits, operators) — shown by a logit-noise experiment where perturbing only low-entropy predictions collapses accuracy and perturbing high-entropy ones barely moves it. W4A4KV4 NVFP4 + ReQAT hits 65.94% on AIME-120 vs BF16 FT 65.46% on R1-Qwen-14B — but TRAILS BF16 FT on R1-Llama-8B (41.85% vs 48.75%). 3.9x throughput on DGX Spark, 3.1x on B200. ICML 2026.
ReQAT: full-precision reasoning accuracy at 4-bit floating point
Abstract
Verbatim from the arXiv abstract page (fetched 2026-09-11):
Large Reasoning Models (LRMs) achieve strong problem-solving through long chain-of-thought, but their deployment is constrained by the high cost of full-precision inference and growing KV cache footprints. Microscaled FP4 formats enable efficient FP4 deployment; however, fully quantizing weights, activations, and KV caches (W4A4KV4) causes severe reasoning degradation that existing PTQ and QAT fail to recover. We identify that FP4 failures concentrate on low-entropy tokens--precise symbolic commitments such as digits and operators--where quantization noise inflates sampling errors that cascade through reasoning traces. Based on this insight, we propose ReQAT, a reasoning-centric FP4 training framework with three components: (i) Trace-Aligned QAT (TAQ), which revisits identical reasoning traces to focus updates on critical low-entropy decisions; (ii) Selective Entropy Minimization (SEM), which reinforces confidence at low-entropy positions; and (iii) Q-FIT, a quantization-friendly initialization that jointly calibrates RoPE-consistent KV cache transformations to stabilize QAT. Under the same training budget, ReQAT not only recovers but surpasses BF16 fine-tuning accuracy, while delivering up to 3.9x throughput speedup on NVIDIA DGX Spark and 3.1x on B200.
The mechanism, stated plainly
Quantization noise does not damage a reasoning trace uniformly. It lands hardest where the model is most certain — the single digit, the operator, the variable name — because at those positions the distribution is sharp and a small perturbation in the logits is enough to move probability mass onto a wrong alternative. High-entropy positions (where several continuations are all plausible prose) absorb the same noise harmlessly.
The paper's evidence for this, beyond the claim:
- Tail-mass ratio ρ > 1 at low-entropy positions — quantization reduces the top-1 probability and raises alternatives, flattening the distribution, even where the argmax ranking is unchanged. Greedy decoding hides the damage; sampling does not.
- Logit-noise perturbation experiment — injecting noise only on low-entropy token predictions causes a large accuracy drop across models; injecting the same noise only on high-entropy predictions has a much smaller effect.
That second experiment is the falsifier, and it is the reason this belongs on the rung rather than being one more FP4 speedup table: the mechanism is separable and testable.
Methodology
Three components under a fixed training budget:
- TAQ (Trace-Aligned QAT) — re-runs identical reasoning traces so updates concentrate on the critical low-entropy decisions rather than being spread over the whole sequence.
- SEM (Selective Entropy Minimization) — explicitly sharpens confidence at low-entropy positions.
- Q-FIT — a quantization-friendly initialisation that jointly calibrates RoPE-consistent KV-cache transformations to stabilise QAT.
Models: R1-Qwen-14B and R1-Llama-8B (DeepSeek-R1 distills). Formats: NVFP4 W4A4KV4 and MXFP4 W4A16.
Results
AIME-120, R1-Qwen-14B:
| Configuration | Accuracy |
|---|---|
| BF16 baseline | 56.83% |
| BF16 full fine-tuning | 65.46% |
| NVFP4 W4A4KV4 + ReQAT | 65.94% |
| MXFP4 W4A16 + ReQAT | 68.02% |
The headline claim holds on this benchmark: 4-bit weights, activations and KV cache beat BF16 fine-tuning at the same budget.
R1-Llama-8B, NVFP4 W4A4KV4, 350M tokens — where it does not:
| Benchmark | ReQAT | BF16 FT |
|---|---|---|
| GSM8K | 89.85% | 91.15% |
| MATH-500 | 90.53% | 92.18% |
| AIME-120 | 41.85% | 48.75% |
On the smaller model ReQAT trails full-precision fine-tuning on all three, and by 6.9 points on AIME-120. The abstract's "surpasses" is a 14B result, not a general one — record it that way.
Throughput (1K requests, 512-token prompts, batch up to 256, max generation 8K/16K):
| Hardware | Speedup vs BF16 |
|---|---|
| NVIDIA DGX Spark | 3.9× (up to 3.93×) |
| B200 | 3.1× (up to 3.13×) |
Q-FIT adds only 4–5% overhead over native NVFP4.
Limitations
- The "surpasses BF16" result is model-dependent — it holds for R1-Qwen-14B and fails for R1-Llama-8B (see above). This is the single most important caveat and the abstract does not carry it.
- Authors' own stated limitation: TAQ sits on top of SFT and inherits the supervision quality. "When the reasoning traces themselves are weak or noisy, TAQ may provide limited gains."
- Two models, both DeepSeek-R1 distills. Not evidence about frontier reasoning models or non-distilled training.
- Throughput figures are on NVIDIA Blackwell hardware with microscaled FP4 support; the speedups are as much a property of the silicon's FP4 path as of the method.
Why this is on the compilers rung, and where it points
Quantization is the compiler/kernel layer deciding what numeric format the graph actually runs in, and KV-cache quantization is where the serving cost lives. The teachable line — quantization error lands on the tokens that carry the commitment — is a mechanism, not a leaderboard.
It reads hard onto the ruled direction (personal AI, docs/plans/direction-2026-08-19.md): DGX Spark is a box a person can buy, and 3.9× on it is the local-inference number. That is the cloud-versus-local arithmetic moving, measured on the hardware the direction is about.
No live market call rests on this rung.
Source: ReQAT by Janghwan Lee, Sihwa Lee, Jinseok Kim, Yongjik Kim, Jieun Lim, Jinwook Oh and Jungwook Choi — arXiv:2606.15682, ICML 2026, submitted 2026-06-14.