FlashAttention

Type: Product (open attention-kernel line; FlashAttention-4 by Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar and Tri Dao)

FlashAttention is the line of attention kernels that each GPU generation gets re-optimised for. Its fourth version, published on arXiv on 2026-03-05, is written for NVIDIA Blackwell and opens with the finding this rung is built around: on Blackwell the tensor cores are no longer what limits attention. Blackwell roughly doubled tensor-core throughput while shared-memory bandwidth and the exponential units scaled "minimally or not at all", so the non-matrix work now takes 25–60% longer than the matrix work for typical workloads.

The paper's techniques all follow from that: bigger tiles and fully asynchronous matrix operations so softmax overlaps with the matmuls, exponentials computed partly in software on the FMA units to relieve the hardware exponential unit, softmax rescaling skipped when it is not needed, and a paired-block (2-CTA) mode that halves one operand's shared-memory traffic in the backward pass and cuts global atomic adds by half. It is written in CuTe-DSL inside Python rather than C++ templates.

Key Contributions

• 1,613 TFLOPs/s at 71% utilisation on B200, BF16 — up to 1.3× cuDNN 9.13 and up to 2.7× Triton (FlashAttention-4)
• Non-matmul over matmul cost: +25% to +60% on Blackwell attention (FlashAttention-4)
• 20–30× faster compilation in CuTe-DSL vs C++ templates (2.5 s vs 55 s forward) (FlashAttention-4)
• Deterministic backward at up to 75% of full speed — the measured price of reproducible training (FlashAttention-4)

Evidence caveat

A preprint, not peer-reviewed; the institution is not stated on the arXiv abstract page. "Up to" multiples are against a single cuDNN version, and utilisation is measured against a vendor peak. No independent reproduction is recorded here.

Mentioned In

• The non-matmul bottleneck — the Blackwell half of the evidence
• Kernel portability — Blackwell-specific by construction; the Triton gap
• FlashAttention-4 — the source

Related Entities

• NVIDIA — Blackwell's asymmetric scaling is the problem the kernel answers
• Triton — the portable kernel language it beats by up to 2.7×, now closing part of the gap automatically