inference-bench — the memory roofline, verified
Turn "fast inference is a memory problem" from a claim into a measurement: reproduce the Qwen3-8B roofline (ridge point, KV bytes per token, prefill vs decode), then benchmark vLLM / SGLang against the theoretical ceilings and see exactly where the real stack lands.
Where it runs
In-labThe interactive roofline runs right here in your browser; the vLLM / SGLang benchmark half runs on a GPU box (lab-notes).
Live demo
A mixture-of-experts model routes each token through only a fraction of its weights, so compute is charged on the active parameters while memory is still charged on all of them — leave this at 0 for a dense model, where the two are the same number.
Decode runs at 0.74 FLOP/byte — far left of the 562 ridge, so the B200 spends decode ≈0.13% of its peak FLOPs and starves waiting on memory. The lever is fewer bytes per token (KV compression, quantization), not more FLOPs.
Model shapes were read from each model's own config.json, as of 2026-09-05 — a dated snapshot, not a maintained series. Defaults reproduce the Qwen3-8B worked example from Alex Smola's “Efficiency in LLMs” (KV/token ≈ 147 KB, ridge ≈ 562 FLOP/byte on B200, decode well under 1% of peak). Peak is the GPU's FP8-dense figure; the memory-bound conclusion holds across precisions.
Stack
The problem
Before you rent a model by the token, the question is whether your own machine can run it at all — and nearly every published KV-cache figure is derived from a formula that does not hold for hybrid-attention models. A reader deciding between a subscription and a box needs the real number, not the naive one.
The approach
Derive the roofline by hand from the model shape, then instrument a real serving run — batching, KV cache, quantization — and plot measured throughput against the compute / bandwidth ceilings.
Decisions & trade-offs
- 01Use the exact model from the Smola deck (Qwen3-8B) so theory and measurement line up.
- 02Report ops:byte and where decode sits on the roofline, not just tokens/sec.
What it has found
- The standard KV-cache formula charges every layer for the whole context, and a hybrid-attention model does not work that way. Muse Glimmer 30B keeps 13 of its 52 layers at full length and caps the other 39 at a 2,048-token window: at 32k context and fp16 that is 0.52 GB of cache. Sized instead from the nearest plain-GQA row of comparable size — Qwen3 32B, 64 layers and 8 KV heads, which is exactly what a shape lookup borrowed before this model had a row — the same context is charged 8.59 GB. A factor of 16.6, and at int4 weights it moved the verdict from a 24 GB card to a 32 GB one for a model its maker ships as a 24 GB model.
- The cache is a property of the layer TYPES, not the layer COUNT. Two models with the same parameter count and different attention geometry hold different amounts per token, which is why no rule derived from parameter count can be scaled into correctness.
Where it stands
Half of it is live. The in-browser calculator answers "will this model fit on my machine, and how fast" for named models — the shapes load from the same dated table the /tools sizer runs on — and the hybrid-attention correction is built in rather than promised: pick a model with a sliding window and watch the cache figure fall. Still to come: the vLLM / SGLang benchmark half on a GPU box, and the /notebook piece showing what the standard formula gets wrong and by how much.
- You can run it here — metA live demo on the page itself, in your own browser — not a screenshot of one, and not a promise that one is coming.
- Its core is pinned by tests — metThe arithmetic underneath is a plain module with its own test suite, so every number the interface prints is a number something checks.
- The code is public — not yetA link to the source, so a reader who disagrees with a figure can go and read the line that produced it.
- It has found something — metAt least one stated finding — a thing the instrument taught that the build could not have written down before it existed.
Not shipped yet: one criterion is still open — the code is public.
The running diary appears here as /notebook entries once the build starts.