Compilers & Kernels

This rung is the layer between a model and the chip: the kernels — small, hand-tuned programs — that turn "do attention over this context" into instructions a specific GPU can run, and the tools used to write them. It is the least visible rung in the stack and one of the most direct. A kernel change moves how many tokens a GPU produces per second with nothing else changing: same chip, same model, same maths. That makes it the cleanest place to see a token get cheaper for a reason you can name.

The rung holds five sources, compiled first on 2026-09-24 from three and extended the same day: one peer-reviewed conference paper (ReQAT, ICML 2026), two preprints (FlashAttention-4, and a batch-1 decode study across four GPUs), and two vendor write-ups — DeepSeek's FlashMLA, checked against its shipped code, and Meta's design note on automatic warp specialization in Triton, which is labelled as the feature team describing its own work. Every performance number here is the authors' own, and none has been independently reproduced in this KB. Standing across all five is read it.

What this rung prices

The number this rung owns: 1.64× on decode throughput from a kernel change alone — DeepSeek's FP8 sparse decoding kernel went from 250 to 410 TFLOPS on the same H800 GPU with the same model, algorithm and number format (as of 2025-09-29, vendor-measured, one configuration) (FlashMLA FP8 deep dive). Decode throughput is the bottom half of output-token cost: the same GPU-hour spread over 1.64× more tokens. How much of that reaches a price someone pays is set on the serving and economics rungs, not here — this rung supplies the multiplier, not the price.

The binding constraint: the tensor cores are no longer the limit. On Blackwell, the work around the matrix maths — moving data through shared memory, computing exponentials — takes 25–60% longer than the maths itself (FlashAttention-4). On Hopper, converting an FP8 cache back to BF16 costs at least 50 cycles per token against 34 cycles of matrix work (FlashMLA). Two teams, two chip generations, one conclusion: the fast part of the chip is waiting on the slow parts. See the non-matmul bottleneck.

For one user at a time, the binding constraint is different again. At batch 1 — the personal-AI case — an H100 running a 7B model reaches only ~27% of its memory-bandwidth limit against ~81% on a far cheaper L4, because the CPU's launching of ~280 small kernels per token dominates on the fast card; CUDA Graphs recovers 1.259× on H100 and 1.028× on L4 (batch-1 decode study). With the best kernel on each card, the H100 is only ~1.47× faster, and this KB's arithmetic at the paper's list rates puts the L4 at ~$1.45 per million tokens against ~$11.45 on the H100 (as of May 2026, single-stream, excluding idle). See launch overhead at batch 1.

Why it sits here and not next door. hardware would claim the FlashMLA result — the bottleneck exists only because H800 has no direct FP8→BF16 instruction — but that is a claim on the next chip; on the chip that shipped, the gain came from software. serving would claim it because FP8 caching is a serving decision, but serving owns the choice to store fewer bytes; this rung owns what happens when those bytes have to become numbers again fast enough to matter.

The load-bearing findings

• The tensor core is not the binding constraint in attention kernels any more. Moderate evidence, from two sources that agree on the shape of the problem while naming different culprits. The non-matmul bottleneck.
• A newer chip ran the same kernel slower. DeepSeek's sparse FP8 decode kernel: 410 TFLOPS on H800, up to 350 on B200, the B200 path self-described as "not really optimized yet" (as of 2025-10-01). Kernels are written for a generation; new silicon does not inherit old wins. Kernel portability.
• 4-bit reasoning is possible on one model and not yet on another. ReQAT puts weights, activations and the KV cache in FP4 and matches full-precision fine-tuning on R1-Qwen-14B (65.94% vs 65.46% on AIME-120), but trails it on R1-Llama-8B by 6.9 points. Its mechanism — rounding noise lands on the tokens the model is most sure of, like digits and operators — is tested directly. 3.9× throughput on DGX Spark, a machine a person can buy. FP4 reasoning quantization.
• For one user, a faster GPU is not a cheaper token. Launch overhead caps the H100 at ~27% of its memory bandwidth at batch 1; on the cheaper L4 the 4-bit kernel choice alone spans 45.24 to 17.36 ms per token. Launch overhead at batch 1.
• The portable compiler is catching up, on one kernel. Triton's automatic warp specialization reaches 1.5–2× over stock Triton on B200 attention forward, 10–20% behind cuDNN (vendor-reported, no configurations published). Kernel portability.
• Compressing the cache is not the same as compressing the maths. FP8 storage saves ~43% of cache bytes per token and then costs cycles to undo. Quantized KV cache.

Frontier — What's Moving Now

• Kernel design organised around whichever chip unit did not scale (exponential units, shared memory) rather than around peak FLOPs.
• Native low-precision compute (FP4 on Blackwell) moving the problem from conversion cost to accuracy on the tokens that matter.
• Python-embedded kernel languages (CuTe-DSL) cutting compile times 20–30×, making per-chip tuning cheaper; and compilers (Triton autoWS) doing the warp-level scheduling hand-written kernels do.
• Single-stream decode, where launch overhead and the choice of quantisation kernel matter more than the GPU's memory bandwidth.
• See the open questions for open problems and gaps.

Concept Map

Concepts

Concept	Sources	Evidence	Last Updated
The Non-Matmul Bottleneck	3 (2 preprint, 1 technical report)	Moderate	2026-09-24
Quantized KV Cache: Stored Small, Computed Large	2 (1 conference, 1 technical report)	Moderate	2026-09-24
FP4 Reasoning Quantization	1 (conference)	Strong, single paper	2026-09-24
Kernel Portability Across Chip Generations	4 (2 preprint, 2 technical report)	Weak to moderate	2026-09-24
Launch Overhead: Why a Faster GPU Doesn't Speed Up One User	1 (preprint)	Moderate, single preprint	2026-09-24

Entities

Entity	Type	Sources
DeepSeek	Lab	1
FlashAttention	Product	1
NVIDIA	Company	5
Triton	Product	2

Timeline

See the timeline for chronological developments.

Sources

#	Title	Type	Date	Status
1	A Deep Dive Into The Flash MLA FP8 Decoding Kernel on Hopper	technical-report	2025-09-29	compiled
2	FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling	preprint	2026-03-05	compiled
3	ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Training	conference	2026-06-14	compiled
4	Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode	preprint	2026-05-28	compiled
5	Warp Specialization in Triton: Design and Roadmap	technical-report	2026-01-08	compiled

Standing

Read it, for all five. The FlashMLA close-read went further than most — the repository was cloned at a named commit, every arithmetic step recomputed, and the crossover mechanism verified in the shipped CUDA — but nothing was run. The batch-1 decode study is the most runnable source on the rung (a single consumer-class GPU, open tools) and the obvious first candidate to do first-hand. Until then anything built on this rung teaches at the level of how it works, not how to do it. No live market call rests on this rung.