StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration

Abstract

StreamDQ addresses a bottleneck in weight-only-quantized LLM inference: quantized weights must be dequantized back to FP16/BF16 before matrix multiply, and doing that on GPU CUDA cores consumes on-chip datapath and, at large batch sizes, forces extra HBM write-back and reload of the dequantized weights. StreamDQ relocates dequantization into the base die of custom HBM via compact DeQuantization Blocks (DQBs) that perform inline dequantization on standard memory loads. A lightweight 3-bit sideband tag on each read request selects the dequantization mode (or bypass), preserving conventional load semantics with no ISA change on the GPU side.

Key Contributions

• Near-memory dequantization on the HBM base die — one DQB sits on the read path of each pseudo-channel memory controller, so weights are unpacked on the memory side and returned through standard load-response paths. The GPU compute datapath is bypassed entirely.
• Format coverage: INT4→FP16, INT4→BF16, INT8→FP16, INT8→BF16, FP8→FP16/BF16. Floating-point conversion uses wire-mapping; integer conversion uses shared lookup tables.
• Design of the DQB itself: tag parser + scale/zero-point (S/Z) request generator with buffer + data-processing block.
• A concrete, silicon-costed answer to "what does near-memory compute on a custom HBM base die actually buy you" — from a memory maker, not an academic group.

Results

(All figures are the paper's own simulated/synthesized results — preprint, not independently replicated.)

• Mixed-precision GEMM: up to 7.08× speedup and 90.23% lower energy vs fused-kernel software dequantization (compared against GPTQ, AWQ-v1, AWQ-v2, TorchAO).
• Per-DQB hardware overhead: 0.127 mm² area and 0.355 W power (at 20% toggle) in a 12 nm CMOS process; 11.36 W total per stack across 32 DQBs.
• End-to-end LLM inference: up to 54.68% latency reduction and 2.20× higher decode throughput.

Limitations

The authors state the advantage shrinks at larger batch sizes, where dequantization is a smaller fraction of total runtime. In small-batch memory-bound regimes (batch ≤ 4) StreamDQ can be competitive with or slightly slower than fused kernels. This is the mirror image of TokenStack's regime dependence — near-memory wins are concentrated where memory movement dominates.

Why it matters for this KB

• This is a primary source from a memory maker (SK Hynix). The topic's single largest structural gap is that no primary memory-maker source exists at all — every HBM claim traces to third-party outlooks. StreamDQ closes that on the architecture/near-memory side (SK Hynix showing what it intends to build into custom HBM base dies), though not on the supply/pricing side, which the SK Hynix 2026-07-29 and Samsung 2026-07-30 earnings prints are expected to address.
• It gives C-HBM4E level-3 near-memory-compute a second named workload alongside TokenStack's hot-KV attention: inline weight dequantization for weight-only-quantized inference. Two independent 2026 works now argue the HBM base die should do arithmetic, not just move bytes.
• The regime story is consistent across the near-memory literature: gains concentrate in memory-bound / small-batch decode, not compute-bound prefill or large-batch GEMM.

Source: StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration — Jeong, Yoon, Ahn et al., SK Hynix. Affiliation confirmed via arXiv HTML author block. Full text read at abstract + HTML level; not independently benchmarked.