Low-Bit Quantization

Post-training weight-only quantization reduces the bit-width of a trained model's weights, shrinking model size and accelerating token generation by relieving the memory-bound character of autoregressive decoding. Unlike speculative decoding, quantization is not exactness-preserving — it trades accuracy for throughput, so the research question is always how much accuracy per bit removed.

The dominant obstacle at low bit-widths is systematic outliers in the weight distribution: a small number of large-magnitude weights that dominate the quantization range and force everything else into too few levels. Prior approaches attack this with scaling or rotation transformations. The 2026 contribution compiled here takes a different route — exploiting structure in the second-order (Hessian) geometry to suppress outliers without touching the task loss.

The commercially interesting property is that these methods increasingly carry zero inference overhead. OSAQ's transformation is additive and folds into the weights offline, requiring no inter-layer transformations at runtime; and it is constructed by closed-form solution rather than iterative optimization, so it is cheap to apply. An efficiency technique that costs nothing at serving time and nothing meaningful to apply is the kind that diffuses fast.

Evidence caveat: both sources are arXiv preprints, not peer-reviewed, with self-reported benchmarks. Neither has independent reproduction recorded in this KB.

Key Claims

• A stable low-rank null space in the Hessian can be used to absorb weight outliers with no task-loss cost and no inference overhead — OSAQ observes that the Hessian exhibits low-rank consistency across different inputs, with certain directions consistently showing vanishing curvature. It identifies that stable null space and builds an additive weight transformation by linearly combining vectors within it, suppressing outliers "without affecting the task loss." The transformation absorbs into the weights offline. Evidence: moderate (preprint; mechanism is principled and the claim is internally coherent, but unreproduced) (OSAQ, arXiv:2605.04738)
• At 2-bit, OSAQ integrated with GPTQ achieves over 40% lower perplexity than vanilla GPTQ — Note the framing precisely: OSAQ is a complement to GPTQ, not a replacement for it. Full text confirms it verbatim — "serves as a plug-and-play component that complements existing methods" and "always applied before them." The comparison is OSAQ+GPTQ vs. GPTQ alone. Full text (ingested 2026-07-23) pins the headline: LLaMA2-7B / WikiText2 at 2-bit (W2A16 g128), perplexity 36.8 → 21.2 = 42.4% lower; the gain holds across LLaMA2-13B (28.1→13.4) and 70B (19.2→10.7). Evidence: moderate (preprint, single-paper claim, baseline explicitly named) (OSAQ, arXiv:2605.04738)
• Quantization is training-free and closed-form here, not an optimization loop — OSAQ requires "no resource-intensive training or iterative procedures." This matters for adoption economics: the cost of applying the technique is near-zero, so there is no capex barrier between the paper and deployment. Evidence: moderate (preprint) (OSAQ, arXiv:2605.04738)
• Quantizing the drafter as well as the target is itself an efficiency lever — Because the drafter is invoked at every speculative-decoding step, its overhead is on the hot path. Quantizing the drafter (plus sliding-window attention) preserved draft-token acceptance while improving long-context decoding latency. Recovering target-model accuracy required quantization-aware distillation while retaining the original quantization grid, and the drafter was trained in two stages — first from the high-precision target, then adapted to the low-precision one. Evidence: moderate (preprint competition report; 3rd overall, externally adjudicated against quality thresholds) (Quantize the Target, Quantize the Drafter, arXiv:2607.04244)
• At 4-bit weights, tree-draft speculation can cost the quantization win — the fix is sequence drafts (2.78×) — Low-bit weight quantization pushes decoding hard into the memory-bound regime, and in that regime verifying a tree-style speculative draft (EAGLE-2) incurs significantly more time overhead than a single-token forward pass — so the memory benefit of 4-bit quantization is diminished by the speculative-decoding compute rather than compounded by it. SpecMQuant restores it by converting tree drafts → sequence drafts via a small intermediate model, so the quantized target keeps its memory-access advantage: 2.78× on 4-bit-weight Llama-3-70B (A100), +1.31× over EAGLE-2. The interaction is with drafter geometry, not bit-width alone — and this is a weight-quantization finding; activation and KV quantization interact differently. Evidence: moderate (preprint; abstract + reported metrics; single target family/size on one accelerator; no independent reproduction) (Speculative Decoding Meets Quantization, arXiv:2505.22179)

Compound vs. conflict — MEASURED + RESOLVED (2026-07-24)

The Quantize-the-Target result (6.978×) is this KB's cleanest datapoint that quantization and speculative decoding compound, and VeriCache reinforces the compounding pattern from the KV-cache side. But the pattern is not universal, and the counter-case lives on the quantization side specifically — now measured, not flagged.

arXiv:2505.22179 "Speculative Decoding Meets Quantization" (Zhang et al., May 2025 — ingested 2026-07-24) measures directly that on a 4-bit-weight target, tree-draft speculative decoding (EAGLE-2 style) conflicts with the low-bit-quantization gain and diminishes it: verifying a tree-style draft incurs significantly more time overhead than a single-token forward pass once the target is 4-bit-weight. Low-bit weight quantization shifts decoding further into the memory-bound regime; the parallel tree verification that pays off on a full-precision target relies on spare arithmetic intensity that a quantized target no longer has, so the tree stops paying once the weights are 4-bit.

Resolved — sequence drafts recover the win. SpecMQuant's hierarchical framework uses a small intermediate model to convert tree drafts → sequential drafts, so the quantized target keeps its memory-access advantage: 2.78× on 4-bit-weight Llama-3-70B (A100), +1.31× over EAGLE-2. So the two KB findings do not contradict — Quantize-the-Target quantizes both target and drafter with a block-diffusion (non-tree) drafter and gets 6.978×; 2505.22179 uses a tree drafter on a 4-bit target and must go sequential. The load-bearing correction stands: "quantize the model and it composes freely with any speculative decoder" is false. Drafter geometry (tree vs sequential/block) interacts with target bit-width, and at 4-bit weights that interaction can erase the gain until the drafts are made sequential. Evidence: moderate (preprint; abstract + reported metrics; single target family Llama-3-70B on A100; weight-quantization only — activation/KV quantization uncharacterized). Directly resolves the tension previously held here as evidence-pending. Mirror note on Speculative Decoding.

Benchmarks & Data

Gap (largely closed 2026-07-23): the full-text raw file now holds OSAQ's WikiText2 perplexity grid at 2/3/4-bit plus commonsense-QA and MMLU tables. The 42.4% headline is LLaMA2-7B / WikiText2 at 2-bit specifically — real, but a single (model, corpus, bit-width) point, not a universal figure. The 6.978× is likewise an average; at long context it falls to 2.85×, which is the honest read on where quantization×speculation compounding thins out.

Why this matters for the house thesis

Quantization is the efficiency lever that most looks like it should belong to the hardware layer — it is fundamentally about memory bandwidth and numeric formats — and yet the gain here comes from a closed-form linear-algebra insight about the Hessian, applied offline, running on unchanged silicon at unchanged cost. That is the house thesis in miniature: the constraint is physical, the edge is in software. See Inference & Token Economics for the empirical cost decomposition that this instantiates.

Open Questions

• At what bit-width does the compounding with speculative decoding stop paying off — i.e. where does drafter quality collapse faster than the throughput gain accrues? The now-ingested arXiv:2505.22179 result (2026-07-24) confirms the boundary is not just "low bit-width" but "low bit-width and tree-shaped drafts": at 4-bit weights, tree verification stops paying and sequential drafts recover the gain (2.78×). Still open: does this hold at 2-bit, and for activation/KV quantization rather than weight-only? 2505.22179 is a weight-quantization, single-family (Llama-3-70B / A100) result.
• Does the Hessian's "low-rank consistency across different inputs" hold at frontier (trillion-parameter, MoE) scale, or is it an artifact of dense models at the tested sizes?
• OSAQ suppresses outliers "without affecting the task loss" — verified against which task distribution? Outlier weights are implicated in long-tail and rare-capability behaviour; a perplexity-neutral transform is not automatically a capability-neutral one.
• Does quantization-aware distillation with a retained original quantization grid generalize beyond the competition setting, or is grid retention a constraint-specific choice?

Related Concepts

• Speculative Decoding — the compounding partner; quantizing the drafter is a shared optimization surface — but tree drafts on a 4-bit-weight target conflict, resolved by tree → sequence drafts (measured 2.78×; arXiv:2505.22179)
• KV-Cache Compression — the other memory-bound lever; KV quantization is one of the compression families VeriCache wraps
• Inference & Token Economics — where per-token cost reductions do or do not surface as price

Backlinks

Pages that reference this concept:

• Speculative Decoding
• Inference & Token Economics

Changelog

• 2026-07-22 — Created from two 2026 preprints (OSAQ, Quantize the Target). Records the >40% 2-bit perplexity claim with its GPTQ-integration caveat, the zero-inference-overhead / closed-form properties, and the drafter-quantization result. All claims flagged preprint-grade; abstracts verified directly at arXiv (no local raw files — these sources were discovered, not ingested).
• 2026-07-23 — Full text of both preprints read via arXiv HTML5 and ingested as local raw files (osaq-*, quantize-target-*). Closed the OSAQ perplexity-grid gap (LLaMA2-7B/WikiText2 2-bit 36.8→21.2) and added the 6.978× hardware nuance (A10G = competition target, reproduced on RTX 5000 Ada; 2.85× at long context). No claim reversed — the abstract-only compile was directionally correct, including the OSAQ complement-not-replacement framing.
• 2026-07-23 — Recorded the compound-vs-conflict tension (mirror of the Speculative Decoding note): added a "Compound vs. conflict" section flagging arXiv:2505.22179 — tree-draft speculative decoding on a 4-bit-weight target can negate the quantization gain unless drafts go sequential. Held as a flagged tension (evidence pending full read), and updated the lead open question to name drafter geometry, not just bit-width, as the boundary. Added VeriCache/KV-cache cross-links.
• 2026-07-24 — Resolved the compound-vs-conflict tension. Ingested arXiv:2505.22179 "Speculative Decoding Meets Quantization" (SpecMQuant) as a local raw file; flipped the section from flagged / evidence-pending to MEASURED + RESOLVED and added a Key Claim: on a 4-bit-weight target, tree-draft EAGLE-2 diminishes the quant benefit (tree-draft verification costs significantly more than a single-token forward pass), and a hierarchical small-model tree→sequence drafter recovers it — 2.78× on 4-bit-weight Llama-3-70B (A100), +1.31× over EAGLE-2. Updated the lead open question and the Speculative-Decoding cross-link; added the source to frontmatter. Preprint-grade (abstract + reported metrics); weight-quantization only, so 2-bit / activation-quant / non-A100 behaviour stays open.