Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode
At batch 1, three 7-8B models across four NVIDIA GPUs (44 cells): H100 reaches ~27% of its memory-bandwidth floor vs ~81% on L4. Launch overhead is the gap — CUDA Graphs 1.259x on H100 (95% CI 1.253-1.267, N=10, pre-registered thresholds) vs 1.028x on L4. On L4 the int4 KERNEL is the lever: ExLlamaV2 17.36 ms vs AutoAWQ+Marlin 45.24 ms vs bf16 62.32 ms per token. H100+Graphs 11.78 ms vs L4+ExLlamaV2 17.36 ms at $3.50 vs $0.30/hr (May 2026): paper says ~6x cheaper per token on L4; its own inputs give ~7.9x (~$11.45 vs ~$1.45 per Mtok, KB arithmetic). Preprint, single author, no Blackwell/AMD/Apple.
How it was read: the arXiv abstract page and the full HTML text (arxiv.org/html/2605.30571, v1, 28 May 2026) were fetched on 2026-09-24 and read in full, including the tables, limitations and appendix list. Figures rendered as maths were recovered from the HTML's alt-text. Single author; a preprint, not peer-reviewed. Nothing was re-run here.
What it measures
Most research on serving language models is about batches: many users' requests packed together so the GPU stays busy. This paper measures the other case — one request at a time, which is how a robot, a car's assistant, or a model running for one person on their own machine actually works. The textbook account of that case is that it is limited by memory bandwidth: each new token means reading every weight (and the growing KV cache) from memory once, so a GPU with faster memory should produce tokens proportionally faster.
The setup: three 7–8B models (Qwen-2.5-7B, Mistral-7B-v0.3, Llama-3.1-8B; all grouped-query attention, head_dim 128), four NVIDIA GPUs (H100 SXM5 at 3,350 GB/s, A100-80GB at 2,039 GB/s, L40S at 864 GB/s, L4 at 300 GB/s), four context lengths (2,048–16,384), bf16, batch 1 — 44 valid cells (4 ran out of memory on the 24 GB L4). Median of 30 measured decode steps per cell, on Modal cloud hosts.
Correction to the ingest proposal: COMP-1 was described as "one model (Qwen-2.5-7B) and one context length". The sweep covers three models and four context lengths; only the headline cell and the follow-up experiments use Qwen-2.5-7B at 2,048 tokens.
Findings
| Finding | Figure | Where |
|---|---|---|
| Share of the memory-bandwidth floor reached, Qwen-2.5-7B, ctx 2048 | L4 0.810 · L40S 0.723 · A100 0.311 · H100 0.270 | Table 1 |
| Median across all cells per GPU | L4 0.694 · L40S 0.516 · A100 0.275 · H100 0.260 | §4.3 |
| H100 step time, eager → with CUDA Graphs (N=10 sessions) | 14.83 ms → 11.78 ms = 1.259× (95% CI 1.253–1.267; CV 0.9%) | Table 2 |
| Launch-side time removed on H100 | 3.05 ms ≈ 20.6% of the step | §5 |
| Same intervention on L4 | 64.48 → 62.50 ms = 1.028× (a null, as predicted) | §5, Table 3 |
| Graphs speedup on H100 at batch 4 | 1.110× (ctx 2048) falling to 1.036× (ctx 16384) | Table 4 |
| Best attention kernel at H100 batch-1 shape | PyTorch's default SDPA, 36.05 µs/layer — faster than FlashAttention-3 (79.25), FlashInfer (48.20), pinned FLASH backend (44.35); cuDNN attention "not supported" for this shape | Table 6 |
| L4 quantisation, Qwen-2.5-7B, ctx 2048 (ms per token) | bf16 62.32 · bnb-nf4 59.36 · AutoAWQ+Marlin 45.24 · ExLlamaV2 EXL2 4.25bpw 17.36 (int4 floor 13.09) | Table 7 |
| Best lever per GPU | H100 + Graphs 11.78 ms vs L4 + ExLlamaV2 17.36 ms — H100 ~1.47× faster | §7 |
The mechanism. A decode step for Qwen-2.5-7B launches roughly ten kernels per layer, ~280 in all. Each launch costs the CPU a roughly fixed amount of time. On the slow L4 each kernel's GPU work is long, so launch time hides underneath it; on the fast H100 the GPU work shrinks and the launches become a large share of the step. That is why the faster the memory, the smaller the share of it that gets used. The paper tested this with pre-registered kill thresholds (an H100 speedup under 1.05×, or an L4 speedup over 1.10×, would have falsified it); neither was crossed.
Quantisation, on the cheap GPU. Four-bit weights should cut the bytes read per token about 4×. Two common routes barely deliver: bitsandbytes nf4 converts weights back to bf16 before the maths (59.36 ms, ~5% faster than bf16); AutoAWQ's Marlin kernel was tuned for the Ampere generation (SM80) and gets 1.38× on the Ada-generation L4. ExLlamaV2, with int4 kernels tuned for Ada, gets 3.59× (62.32 → 17.36 ms). "The lever was the kernel implementation, not the bit width."
Self-correction in the paper. Earlier drafts reported a 1.72× Graphs speedup from one session; the 10-session run replaced it with 1.259×. An earlier claim about which backend default SDPA dispatches to was withdrawn after a backend-pinned test.
What number this moves
This is a kernel-layer result with a price attached. The paper prices both setups at Modal's list rates as of May 2026 — $3.50/hr (H100) and $0.30/hr (L4) — and states the L4 serves this workload at "roughly 6× less $/token".
That figure does not match the paper's own inputs. Recomputed here from the stated rates and step times: H100 + Graphs = 3,600 s ÷ 0.01178 s ≈ 305,600 tokens an hour → ~$11.45 per million tokens; L4 + ExLlamaV2 = 3,600 ÷ 0.01736 ≈ 207,400 tokens an hour → ~$1.45 per million tokens — a ratio of ~7.9×, not ~6×. The direction and the lesson hold either way; the exact multiple should be cited as "6–8×, depending on how it is computed", and the per-million figures as this KB's arithmetic on single-stream decode, excluding idle time, networking and storage.
For the through-line: at batch 1, the cheapest token came from the right kernel on the cheaper GPU, not from the faster GPU. A buyer sizing hardware for one-user workloads by memory bandwidth would overpay.
Limitations
- One author, preprint, not peer-reviewed; measured on shared cloud hosts with undisclosed drivers (the author reports Nsight Compute was blocked, so kernel timing uses torch.profiler, which itself inflated H100 step time 3.2×).
- Scope is narrow and stated: 7–8B GQA models, head_dim 128, bf16 (except the L4 quantisation test), batch 1 (batch 4 as a check), NVIDIA only — no Blackwell, no AMD, no Apple, no Jetson.
- The launch-overhead finding is, as the author says, folk knowledge among kernel engineers; what is new is the controlled cross-GPU sweep and the falsification test.
- For anyone not forced to batch 1, batching is the stronger lever: batch 4 on H100 raised the graphed step from 11.78 ms to 14.75 ms while producing four tokens.
- The per-token prices are list-rate arithmetic, not a measured bill.
Standing
I only read about it. Nothing here was run. The workload — one request at a time on a single GPU — is the personal-AI case this KB is pointed at, so it is a strong candidate to run first-hand (CUDA Graphs on/off, and int4 kernel against int4 kernel on the same card); until then a piece built on it teaches how it works, not how to do it.
No live market call rests on this rung.
Source: arXiv:2605.30571 — Josef Chen, 28 May 2026 (v1).