OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization

Abstract

Verbatim (opening): "Large Language Models (LLMs) have demonstrated remarkable capabilities. However, their massive parameter scale leads to significant resource consumption and latency during inference. Post-training weight-only quantization offers a promising solution by reducing model size and accelerating token generation through alleviating the memory-bound issue. Nevertheless, the presence of inherent systematic outliers in weights continues to be a major obstacle. While existing methods, such as scaling and rotation, attempt to address this issue, the performance remains unsatisfactory. In this paper, we propose Outlier Self-Absorption Quantization (OSAQ), which performs additive weight suppression guided by the second-order low-rank property for low-bit weight-only quantization of LLMs..."

THE LOAD-BEARING FRAMING: complement to GPTQ, NOT a replacement

This is the claim the earlier abstract-only pass got wrong (it implied replacement / ">40% lower perplexity than vanilla GPTQ" read as OSAQ-vs-GPTQ). The full text is explicit that OSAQ is a plug-and-play complement applied before other quantizers:

• Verbatim: "The proposed [OSAQ] serves as a plug-and-play component that complements existing methods to further enhance performance."
• Verbatim: "OSAQ is used as a plug-and-play component, and when combined with other methods, it is always applied before them."
• The ">40%" claim, verbatim: "For instance, in 2-bit quantization, OSAQ, when integrated with GPTQ, achieves over 40% lower perplexity compared to vanilla GPTQ."

So the comparison is OSAQ+GPTQ vs GPTQ-alone, at 2-bit weight-only. OSAQ does not stand in for GPTQ; it preconditions the weights so GPTQ (or AWQ, QuIP, etc.) quantizes them better.

Key Contributions (verbatim)

1. "We propose an outlier self-absorption method that performs additive transformations by exploiting the low-rank properties of the second-order Hessian."
2. "The additive transformation is constructed by linearly combining the vectors in the null space of the Hessian. Thanks to the Softmax-∞ objective approximation, the combination coefficients can be derived in closed form."
3. "Extensive experiments... OSAQ significantly outperforms the existing methods in low-bit quantization."

Methodology

Three stages: (1) eigendecompose the Hessian to identify directions of vanishing curvature (its stable null space); (2) construct an additive weight transformation as a linear combination of null-space vectors — because it lives in the null space, it suppresses outliers "without affecting the task loss"; (3) solve the combination coefficients in closed form via a Softmax-∞ approximation (minimize ||W + βN||∞). The transform is absorbed into the weights offline, so there is zero inference overhead and no iterative optimization at apply time. Null-space stability across input batches is high (max singular values 0.965-0.981, Table 6).

Results

Perplexity, WikiText2 (Table 1) — OSAQ+GPTQ vs GPTQ-alone

2-bit (W2A16 g128): LLaMA2-7B 36.8 → 21.2 (42.4% lower, the headline); LLaMA2-13B 28.1 → 13.4; LLaMA2-70B 19.2 → 10.7. 3-bit (W3A16): LLaMA2-7B 8.37 → 6.75; LLaMA2-13B 6.44 → 5.72. 4-bit (W4A16): LLaMA2-7B 5.83 → 5.73.

The ">40% lower perplexity" headline is specifically LLaMA2-7B on WikiText2 at 2-bit ((36.8−21.2)/36.8 = 42.4%). (This closes the KB gap that previously said "specific models and corpora not captured.")

Commonsense QA zero-shot accuracy (Table 2)

LLaMA3-8B 3-bit avg: GPTQ 63.6% → OSAQ+GPTQ 65.2%. LLaMA3-8B 2-bit avg: 38.3% → 44.4%.

MMLU (Tables 3-4)

LLaMA2-7B 4-bit: 38.7% → 39.6%. LLaMA3-8B 4-bit: 57.6% → 58.2%. LLaMA-3.1-405B 4-bit: 85.7% → 86.1%.

KV-cache + weights (Table 8)

LLaMA2-7B W4 + 4-bit KV: WKVQuant 5.64 → OSAQ+WKVQuant 5.59 (also complements weight-activation methods).

Overhead (Tables 13-14)

Apply cost: LLaMA2-7B GPTQ 22 min → OSAQ+GPTQ 24 min (A100). Inference speedup unchanged: W4A16 1.89x (7B) / 2.41x (13B) maintained — OSAQ adds no runtime overhead.

Baselines compared against: GPTQ, AWQ, QuIP, MagR, OmniQuant (weight-only); QuaRot, DuQuant, SpinQuant, WKVQuant (weight-activation).

Verification note (against prior KB record)

• OSAQ "integrated with GPTQ" framing: CONFIRMED as a complement, not a replacement — verbatim quotes above. This corrects the original registry error where the ">40% lower perplexity than vanilla GPTQ" line implied OSAQ replaced/beat GPTQ standalone. It is OSAQ+GPTQ vs GPTQ-alone.
• The >40% figure = 42.4%, LLaMA2-7B/WikiText2, 2-bit W2A16 g128. Baseline = vanilla GPTQ. Confirmed.

Limitations

The paper enumerates no formal limitations. Implicit: requires a Hessian computation (cost mitigated by the closed-form solution); null-space stability depends on calibration data; a few hyperparameters (γ, τ, μ₁, μ₂) though grid-search robustness is shown (Figure 5). "Suppresses outliers without affecting the task loss" is a perplexity/task-loss statement — not automatically a capability-neutral one, since outlier weights are implicated in rare-capability behaviour. Preprint, not peer-reviewed, self-reported.

Source: arXiv:2605.04738 — OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization, Li, Dong, Liu, Zhang & Gu, 6 May 2026. Full text retrieved via arXiv HTML5 on 2026-07-23.