Kernel Portability Across Chip Generations

The fastest kernels are written for one chip. They lean on features that chip has and the next one may not, or that the next one has in a different shape. So a new, faster GPU does not automatically run yesterday's best code faster — and the clearest example in this KB runs the other way.

DeepSeek's FP8 sparse decoding kernel scores 410 TFLOPS on H800 (Hopper) and up to 350 TFLOPS on B200 (Blackwell), a newer and faster part — with the README describing the B200 path as "not really optimized yet" (FlashMLA FP8 deep dive). The trick that won the Hopper gain uses Distributed Shared Memory, a Hopper-generation feature, and the bottleneck it solves exists only because H800 lacks a direct FP8→BF16 conversion instruction. A newer part running an older part's kernel more slowly is the plainest statement that the kernel, not the silicon, is what is being measured.

FlashAttention-4 is the same story from the Blackwell side. Its authors say the implementation is optimised for Blackwell; the principle — optimise for whichever unit did not scale — carries over, the code does not (FlashAttention-4). It also shows how much the choice of authoring tool matters: written in CuTe-DSL inside Python, it compiles 20–30× faster than the equivalent C++ templates (2.5 s vs 55 s forward), and it runs up to 2.7× faster than Triton, the portable kernel language, on the same chip.

The same story, one generation back and on a cheap card. On an NVIDIA L4 (Ada generation), 4-bit weights through AutoAWQ's Marlin kernel — tuned for the previous, Ampere generation — ran a 7B model at 45.24 ms per token; through ExLlamaV2, whose 4-bit kernels are tuned for Ada, at 17.36 ms. Same card, same bit width, 2.6× apart (batch-1 decode study). A kernel tuned for the wrong generation leaves most of quantisation's saving on the table.

The compiler closing the gap itself. Meta's Triton team describes automatic warp specialization (autoWS) — the compiler splitting a kernel's work between warp groups the way hand-written Blackwell kernels do. On B200 flash-attention forward it reports 1.5–2× over stock Triton, near cuDNN and Gluon, with cuDNN still ahead by 10–20% (Triton autoWS). That is the vendor's own benchmark with no configurations published, and it is not comparable to FlashAttention-4's 2.7× (different kernels and baselines). It does answer the shape of this page's open question: the portable path can close much of the gap, for the one kernel where it works so far.

Why it moves the token price

A new GPU's list price buys its peak FLOPs. What a buyer actually gets per GPU-hour is whatever the kernels on it reach, and that lags the hardware. In the gap, the cost per token on the new part is higher than its spec sheet implies — and a lab with its own kernel team is buying tokens more cheaply than one relying on portable code on the same hardware. The 2.7× FlashAttention-4 vs Triton gap is one measure of how large that difference can be.

Key Claims

• A newer GPU ran the same kernel slower: 350 TFLOPS on B200 vs 410 on H800. The two figures come from the repository README (as of 2025-10-01); the portability reading is the close-read's, not the source's. Evidence: weak-to-moderate (vendor figures, single configuration, B200 path self-described as unoptimised) (FlashMLA)

• Kernel wins are tied to a chip generation by construction. FlashMLA's crossover needs Hopper Distributed Shared Memory; FlashAttention-4 is optimised for Blackwell's tensor memory and 2-CTA MMA mode. Evidence: moderate (both sources state it) (FlashMLA, FlashAttention-4)

• Hand-tuned beats portable by a wide margin on the same chip. FlashAttention-4 reports up to 2.7× over Triton and up to 1.3× over NVIDIA's own cuDNN 9.13 on B200, BF16. Kernel baselines age in weeks, so these multiples are dated to the paper. Evidence: moderate (preprint) (FlashAttention-4)

• Authoring in a Python DSL cut compile time 20–30×. CuTe-DSL: 2.5 s vs 55 s forward, 1.4 s vs 45 s backward, "while preserving full functionality". Faster compiles make per-chip tuning cheaper to do. Evidence: moderate (preprint) (FlashAttention-4)

• Reproducibility has a price at this layer. FlashAttention-4's deterministic backward pass runs at up to 75% of the non-deterministic version's speed, because reductions are serialised. Evidence: moderate (preprint) (FlashAttention-4)

• A compiler can recover much of the hand-tuned advantage automatically — for one kernel, so far. Triton autoWS: 1.5–2× over stock Triton on B200 flash-attention forward, 10–20% behind cuDNN; experimental, attention forward only, Hopper and Blackwell only. Evidence: weak to moderate (vendor-authored, one sentence of results, no configurations) (Triton autoWS)

• A kernel tuned for the previous generation wastes the next one's quantisation gain. Marlin (tuned for Ampere) 45.24 ms vs ExLlamaV2 (tuned for Ada) 17.36 ms per token, 4-bit, on an L4. Evidence: moderate (single preprint, one model) (batch-1 decode study)

Open Questions

• The FlashMLA B200 figure is labelled "not really optimized yet" and has not been restated since 2025-10-01. Has the gap closed? The repository has moved on (HEAD 2026-07-27); its published numbers have not.
• How long, typically, between a GPU shipping and its kernels reaching the old part's utilisation? This KB has one anecdote, not a series.
• Does a portable language (Triton) close the 2.7× gap over time, or is the gap structural? Partial answer (2026-01, vendor-reported): autoWS brings Triton to within 10–20% of cuDNN on B200 attention forward. Independent reproduction, and anything beyond attention forward, are still missing.

Related Concepts

• The non-matmul bottleneck — what each generation-specific kernel is fighting
• Launch overhead at batch 1 — where the int4 kernel mismatch was measured

Backlinks

Pages that reference this concept:

• The non-matmul bottleneck
• DeepSeek
• FlashAttention
• NVIDIA
• Triton
• Launch overhead at batch 1