Rung 07 Compilers & KernelsSwitch rungClose
In scope: CUDA and Triton, kernel fusion and autotuning, compiler stacks (XLA, TVM, Inductor), custom kernels for attention and quantization, portability across accelerators. Out: the serving system above it (serving), the chip below it (hardware).
| Type | Source | Published |
|---|---|---|
| PAPER | FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, Tri Dao · Not stated on the arXiv abstract page (FlashAttention lineage) Names which bound moved: on Blackwell, shared-memory traffic and exponential operations exceed MMA compute by 25-60% for typical attention workloads — the tensor cores are no longer the constraint. Reaches 1,613 TFLOPs/s at 71% utilisation on B200 BF16 (1.3x cuDNN 9.13, 2.7x Triton); CuTe-DSL compiles 20-30x faster than C++ templates (2.5s vs 55s fwd). Deterministic backward costs up to 25% of speed. | 2026-03-05 |
| PAPER | Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode Josef Chen · KAIKAKU At batch 1, three 7-8B models across four NVIDIA GPUs (44 cells): H100 reaches ~27% of its memory-bandwidth floor vs ~81% on L4. Launch overhead is the gap — CUDA Graphs 1.259x on H100 (95% CI 1.253-1.267, N=10, pre-registered thresholds) vs 1.028x on L4. On L4 the int4 KERNEL is the lever: ExLlamaV2 17.36 ms vs AutoAWQ+Marlin 45.24 ms vs bf16 62.32 ms per token. H100+Graphs 11.78 ms vs L4+ExLlamaV2 17.36 ms at $3.50 vs $0.30/hr (May 2026): paper says ~6x cheaper per token on L4; its own inputs give ~7.9x (~$11.45 vs ~$1.45 per Mtok, KB arithmetic). Preprint, single author, no Blackwell/AMD/Apple. | 2026-05-28 |