Compilers & Kernels — Research Frontier

What's genuinely new and where the field is heading.

State of the rung as of 2026-09-24 (second compile, same day): 5 sources, 5 concepts, 4 entities. One peer-reviewed paper, two preprints, two vendor write-ups (one of them the Triton team describing its own feature). Every performance number is self-measured by its authors and none is reproduced in this KB. The rung can now explain how a kernel bottleneck is found and removed at both large batch and batch 1, and has its first cost-per-token figure — list-rate arithmetic from one preprint, not a measured bill.

Active Frontiers

1. Feeding the tensor cores

Status: Rapid progress Key papers: FlashAttention-4, FlashMLA FP8 deep dive Key players: FlashAttention, DeepSeek, NVIDIA

The tensor cores got faster than everything around them. On Blackwell, shared-memory traffic and exponentials exceed the matrix work by 25–60%; on Hopper, FP8→BF16 conversion costs more cycles than the maths it feeds. The response is to redistribute the slow work — software exponentials on the FMA units, two thread blocks splitting a conversion job — rather than to chase peak FLOPs. See the non-matmul bottleneck.

Open problems:

• Does the Blackwell penalty hold for batch-1 decode, the shape that matters for a single user? Still open: the batch-1 study (frontier 4) finds launch overhead binding on Hopper, and excludes Blackwell.
• Only one throughput result here has been converted into a cost per token (frontier 4), and that is list-rate arithmetic.

2. Low-precision numbers without losing the answer

Status: Early stage Key papers: ReQAT, FlashMLA FP8 deep dive Key players: NVIDIA, DeepSeek

Two routes: store in FP8 and convert back for the maths (DeepSeek, Hopper), or compute natively in FP4 (ReQAT, Blackwell). The second is faster — 3.9× on DGX Spark — but moves the risk to accuracy, and ReQAT locates that risk on the tokens the model is most certain of. It matched full precision on a 14B model and missed on an 8B one. See FP4 reasoning quantization and quantized KV cache.

Open problems:

• Why the 14B model holds and the 8B model does not.
• DeepSeek's FP8 cache format has no published accuracy figure at all.
• Nobody has joined the speed leg and the accuracy leg into cost per correct answer.

3. Portability across chip generations

Status: Steady Key papers: FlashMLA FP8 deep dive, FlashAttention-4, Triton autoWS, batch-1 decode study Key players: DeepSeek, FlashAttention, NVIDIA, Triton

The best kernels are written for one chip. DeepSeek's sparse decode kernel ran slower on the newer B200 (up to 350 TFLOPS) than on H800 (410); FlashAttention-4 is Blackwell-specific by design and beats the portable Triton by up to 2.7×. Python-embedded kernel languages cut compile time 20–30×, which lowers the cost of re-tuning per generation. See kernel portability.

Two additions on 2026-09-24. On a cheap Ada-generation L4, a 4-bit kernel tuned for the previous generation (Marlin) ran at 45.24 ms per token against 17.36 ms for one tuned for Ada (ExLlamaV2) — same bits, 2.6× apart (batch-1 decode study). And the portable compiler is doing some of the hand-tuning itself: Triton's automatic warp specialization reports 1.5–2× over stock Triton on B200 attention forward, within 10–20% of cuDNN — vendor-reported, no configurations (Triton autoWS).

Open problems:

• Has the B200 gap closed? The figures have not been restated since 2025-10-01.
• How long does a new chip take to reach its predecessor's kernel utilisation?
• Does autoWS hold up outside attention forward, and does anyone independent reproduce the 10–20%?

4. One user at a time: batch-1 decode

Status: Early stage — one preprint Key papers: Memory-bound but not bandwidth-limited Key players: NVIDIA

The personal-AI case. At batch 1 a 7B model on an H100 uses ~27% of its memory bandwidth against ~81% on an L4; kernel launch from the CPU is the missing time (CUDA Graphs: 1.259× on H100, 1.028× on L4, pre-registered test). On the L4, the 4-bit kernel matters more than the bit width. Best lever per card: H100 11.78 ms vs L4 17.36 ms per token — at May 2026 list rates, ~$11.45 against ~$1.45 per million tokens by this KB's arithmetic (the paper states "roughly 6×"; its own inputs give ~7.9×). See launch overhead at batch 1.

Open problems:

• Apple silicon, AMD, Blackwell and Jetson — the machines people run models on at home — are all out of the study's scope.
• Fused megakernels versus CUDA Graphs as the fix; no open FlashDecoding++ code to compare.
• Run it first-hand: this is the most reproducible result on the rung.

Recent Breakthroughs

Date	Breakthrough	By	Source
2026-06	4-bit weights, activations and KV cache match full-precision fine-tuning on a 14B reasoning model	ReQAT authors (ICML 2026)	ReQAT
2026-05	At batch 1, launch overhead — not memory bandwidth — caps the H100 at ~27% of its bandwidth; the cheaper L4 with the right int4 kernel serves tokens ~6–8× cheaper	Josef Chen (preprint)	Batch-1 decode study
2026-03	Non-matmul work measured at 25–60% over matmul on Blackwell; 1,613 TFLOPs/s attention	FlashAttention-4 authors	FlashAttention-4
2026-01	Compiler-automated warp specialization: 1.5–2× over stock Triton, within 10–20% of cuDNN (vendor-reported)	Meta Triton team	Triton autoWS
2025-09	1.64× decode throughput from a kernel change alone, via Distributed Shared Memory	DeepSeek	FlashMLA

Predictions & Trends

• Each new chip generation will move the bottleneck to a different non-matmul unit, and the first year of kernels on it will underperform the previous generation's mature kernels. One data point supports this (FlashMLA on B200); it is a pattern to test, not an established trend.

Knowledge Gaps

Areas where the KB needs more sources (each still needs Connor's individual approval):

• Batch-1 decode on the hardware people buy — Apple silicon, AMD, DGX Spark-class Blackwell. The batch-1 study covers NVIDIA data-centre and workstation cards only.
• Compiler stacks (XLA, TVM, Inductor) — Triton now has one vendor-authored design note; the others still have zero sources.
• Non-NVIDIA accelerators — all five sources run on NVIDIA; portability across vendors is untouched (autoWS lists AMD support as roadmap only).
• Independent reproduction of any headline number on this rung.
• A measured throughput-to-price bridge — the one cost-per-token figure here is list-rate arithmetic on one preprint's step times, not a served bill.
• TritonForge (arXiv:2512.09196), held since the rung had no sources, was skipped on 2026-09-24: it reports speedups without naming which limit moved, and the two sources added that day do.