Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B
Combines quantization and speculative decoding in one system for constrained hardware (NVIDIA A10G): quantization-aware distillation for the target model + a two-stage-trained block-diffusion drafter, plus sliding-window attention. Achieves 6.978x average speedup over baseline while meeting quality thresholds (3rd place, Efficient Qwen Competition). Concrete evidence the two inference-efficiency techniques this lane tracks (quantization, speculative decoding) compound rather than substitute.
Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B
Abstract
The paper accelerates Qwen3.5-4B inference on resource-constrained hardware by combining a quantized target model with speculative decoding. It applies quantization-aware distillation (QAD) to recover accuracy lost to post-training quantization, and pairs the quantized target with a diffusion-based (DFlash) drafter that is itself optimized via quantization and sliding-window attention. Submitted to the Efficient Qwen Competition.
Key Contributions
- Quantization-Aware Distillation (QAD) — applied to an AWQ INT4 checkpoint while retaining the original quantization grid; per-token distillation minimizing forward KL between BF16 teacher and INT4 student, weights updated via straight-through estimator with quantization scales frozen. Recovers most of the accuracy lost to PTQ.
- Speculative decoding with a quantized drafter — a 5-layer DFlash block-diffusion drafter (hidden size 2,560, block size 16, ~537M BF16 params) trained in two stages: Stage 1 from the BF16 target, Stage 2 fine-tuned against the QAD'd INT4 target so the drafter adapts to the quantized target's output distribution.
- Drafter optimization — GPTQ applied to the drafter to cut per-step cost, plus sliding-window attention (SWA) restricting attention to a local window, reducing attention compute while preserving draft-token acceptance.
Results
Headline
- 6.978x average speedup over the baseline — this is the average of three length settings (Table 1): Short (64/128) 11.1x, Medium (2048/256) 6.96x, Long (8192/256) 2.85x. The long-context setting is where the gain is smallest (2.85x) — relevant to the open question of where quantization×speculation stops compounding.
- Placement: 3rd of more than 40 teams.
Hardware — important nuance
- Competition target hardware: a single NVIDIA A10G GPU (24 GB VRAM) on an AWS g5.xlarge instance. The 6.978x is reported against that target.
- But the authors' own latency reproduction was measured on a different GPU: "Due to the unavailability of an NVIDIA A10G GPU," evaluation was run on an NVIDIA RTX 5000 Ada Generation GPU, "which may not perfectly replicate the competition's hardware characteristics." So "6.978x on a single A10G" is the competition-target framing; the reproduced wall-clock numbers in the paper are RTX 5000 Ada.
Quality thresholds (had to be met, and were)
| Benchmark | Required | Achieved |
|---|---|---|
| MMLU-Pro | ≥ 0.621 | 0.659 |
| IFEval | ≥ 0.814 | 0.845 |
| GPQA-Diamond | ≥ 0.630 | 0.667 |
Target accuracy recovery (Table 3)
| Metric | BF16 | INT4 PTQ | INT4 PTQ + QAD |
|---|---|---|---|
| MMLU-Pro | 0.6804 | 0.6557 | 0.6610 |
| IFEval | 0.8249 | 0.8046 | 0.8285 |
| GPQA-Diamond | 0.6798 | 0.6536 | 0.6626 |
Cumulative contribution (Table 2 — the compounding ledger)
Baseline 1.00x → +INT4 target (QAD) 2.16x → +BF16 drafter 3.32x → +INT4 drafter (GPTQ) 3.55x → +drafter SWA 3.57x. (Table 2's ablation baseline differs slightly from Table 1's, which is why its top figure ~3.57x is below the 6.978x Table-1 average — different reference latencies and setting weighting.) Drafter mean acceptance length ~5.03 tokens (Stage-2 INT4).
Verification note (against prior KB record)
- Compounding claim — "~6.978x on a single A10G, quality threshold": CONFIRMED with two refinements the abstract-only record lacked: (1) 6.978x is an average over short/medium/long, with the long-context component only 2.85x; (2) "single A10G" is the competition target — the paper's reproduced latencies were on an RTX 5000 Ada because an A10G was unavailable. Quality thresholds (MMLU-Pro/IFEval/GPQA) are named and cleared. Institution now identified as Nota Inc. (prior record said only "arXiv preprint").
- The compounding-not-substitution reading holds: both target and drafter are INT4-quantized and speculative decoding is layered on top; each stage adds multiplicatively (Table 2).
Limitations
The authors flag only the hardware substitution (RTX 5000 Ada vs the A10G competition target) as an explicit limitation; no other formal limitations are enumerated. Beyond that: preprint / competition report, self-reported latencies, single model (Qwen3.5-4B) and a single constrained-GPU regime.
Source: arXiv:2607.04244 — Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B, Kim, Lee & Kim (Nota Inc.), 5 July 2026. Full text retrieved via arXiv HTML5 on 2026-07-23.