inference-bench — the memory roofline, verified
Turn "fast inference is a memory problem" from a claim into a measurement: reproduce the Qwen3-8B roofline (ridge point, KV bytes per token, prefill vs decode), then benchmark vLLM / SGLang against the theoretical ceilings and see exactly where the real stack lands.
In-labThe interactive roofline runs right here in your browser; the vLLM / SGLang benchmark half runs on a GPU box (menfem-lab).
Live demo
Decode runs at 0.74 FLOP/byte — far left of the 562 ridge, so the B200 spends decode ≈0.13% of its peak FLOPs and starves waiting on memory. The lever is fewer bytes per token (KV compression, quantization), not more FLOPs.
Defaults reproduce the Qwen3-8B worked example from Alex Smola's “Efficiency in LLMs” (KV/token ≈ 147 KB, ridge ≈ 562 FLOP/byte on B200, decode well under 1% of peak). Peak is the GPU's FP8-dense figure; the memory-bound conclusion holds across precisions.
Stack
The problem
The serving-economics thesis (why value flees the commoditised serving layer) is only as sharp as the numbers under it. Commentary is cheap; a measured roofline is not.
The approach
Derive the roofline by hand from the model shape, then instrument a real serving run — batching, KV cache, quantization — and plot measured throughput against the compute / bandwidth ceilings.
Decisions & trade-offs
- —Use the exact model from the Smola deck (Qwen3-8B) so theory and measurement line up.
- —Report ops:byte and where decode sits on the roofline, not just tokens/sec.
Where it stands
Target: a reproducible benchmark + a /notebook explainer sharpening the CoreWeave / Nebius dossiers.
Build log
The running diary will appear here as /notebook entries once the build starts.