Launch Overhead: Why a Faster GPU Doesn't Speed Up One User

When a model answers one person at a time — on a laptop, in a car, in a robot — each new token means reading every weight from memory once. So the standard advice is that memory bandwidth sets the speed: buy a GPU with faster memory, get tokens proportionally faster.

That is true on a slow GPU and wrong on a fast one. Measured across four NVIDIA GPUs and three 7–8B models, an L4 reaches ~81% of its memory-bandwidth limit on this workload while an H100 reaches only ~27% (batch-1 decode study). The H100 has 11× the memory bandwidth and is nowhere near 11× faster.

The missing time is launch overhead. Producing one token runs roughly 280 small programs (kernels) on the GPU, and the CPU has to start each one. On a slow GPU each kernel runs long enough to hide its launch; on a fast GPU the kernels finish so quickly that the GPU spends much of each step waiting to be told what to do next. Recording the whole sequence once and replaying it (CUDA Graphs) removes most of that: 1.259× faster on H100, measured over ten sessions, and 1.028× on L4, where there was little to remove. The author set the thresholds that would have disproved this before running it, and neither was crossed.

The cheaper-GPU lesson

On the slow GPU, where memory really is the limit, the obvious fix is smaller weights. But the saving only arrives if the kernel actually reads the smaller weights. On an L4, the same 4-bit model ran at 59.36 ms per token through bitsandbytes (it converts back to 16-bit before the maths), 45.24 ms through AutoAWQ's Marlin kernel (tuned for the previous chip generation), and 17.36 ms through ExLlamaV2 (tuned for this one) — against 62.32 ms unquantised. Same bit width, up to a 3.4× spread from the kernel alone.

Put the best lever on each GPU and the H100 (11.78 ms) is only ~1.47× faster than the L4 (17.36 ms), on hardware ~12× dearer per hour at the paper's quoted cloud rates.

How it reaches the token price

Single-stream decode is the personal-AI case, and here the kernel layer decides the price more than the chip does. At the paper's list rates ($3.50/hr H100, $0.30/hr L4, May 2026), this KB's arithmetic from its step times is ~$11.45 per million tokens on the H100 against ~$1.45 on the L4 — roughly 8× cheaper on the cheaper card (the paper itself says "roughly 6×"; see the source's note on the mismatch). These are list-rate, single-stream figures excluding idle, networking and storage — not a bill anyone paid. The general point survives any rate: for one user, the ladder of GPUs by memory bandwidth is not the ladder of cost per token.

Key Claims

• At batch 1, a fast GPU uses a small share of its memory bandwidth: H100 ~27%, L4 ~81%. Monotone across four GPUs and 44 cells. Evidence: moderate (single preprint, careful method, cloud hosts) (batch-1 decode study)
• The gap is launch overhead: CUDA Graphs gives 1.259× on H100 (95% CI 1.253–1.267), 1.028× on L4. Pre-registered falsification test; 3.05 ms of a 14.83 ms step removed. Evidence: moderate to strong for this shape (batch-1 decode study)
• The attention kernel is not the lever at batch 1. PyTorch's default attention path beat FlashAttention-3, FlashInfer and the pinned flash backend at this shape; swapping in FlashAttention-2 made decode slower (17.07 → 24.16 ms). Evidence: moderate (batch-1 decode study)
• On a bandwidth-bound GPU, the quantisation kernel matters as much as the bit width. 4-bit weights on L4: 59.36 / 45.24 / 17.36 ms depending on the kernel. Evidence: moderate (one model, one context length) (batch-1 decode study)
• The effect shrinks with batching. At batch 4 the H100 Graphs gain falls to 1.04–1.11×; for anyone not forced to batch 1, batching is the stronger lever. Evidence: moderate (batch-1 decode study)

Benchmarks & Data

Setup (Qwen-2.5-7B, ctx 2048, batch 1)	ms per token	As-of	Source
H100, eager	14.83	2026-05	study
H100, CUDA Graphs	11.78	2026-05	study
L4, bf16	62.32	2026-05	study
L4, bnb-nf4 (4-bit)	59.36	2026-05	study
L4, AutoAWQ + Marlin (4-bit)	45.24	2026-05	study
L4, ExLlamaV2 EXL2 4.25 bpw	17.36	2026-05	study

Open Questions

• Does it hold on Blackwell, on Apple silicon, or on AMD — the machines a person actually buys to run models at home? The study is NVIDIA data-centre and workstation cards only.
• Do fused "megakernels" (fewer, bigger kernels) beat CUDA Graphs as the fix? The Triton roadmap lists megakernels as a direction (Triton autoWS); the closest prior work, FlashDecoding++, has no public code to compare against.
• What does it cost per token on hardware Connor owns? Nothing here has been run first-hand.

Related Concepts

• The non-matmul bottleneck — the batched-case version: the fast unit waiting on slower ones
• Kernel portability — Marlin tuned for one generation, slow on the next
• Quantized KV cache — storing small is not the same as computing small

Backlinks

Pages that reference this concept:

• The non-matmul bottleneck
• Kernel portability
• NVIDIA