The Non-Matmul Bottleneck

A modern AI chip has one part that is enormously fast — the tensor cores, which do matrix multiplication — and a lot of smaller parts around it that load data, move it between memories, convert number formats and run the few non-matrix steps of attention, like the exponentials in softmax. For years the job of a kernel writer was to keep the tensor cores busy. Both sources on this rung say that is no longer the whole job: on current NVIDIA chips, the tensor cores sit waiting on the other parts. The limit is not how fast the chip can multiply; it is whether everything else can keep up.

The two sources arrive at the same place from different directions. FlashAttention-4 measures it on Blackwell (B200): for typical attention workloads, shared-memory traffic and exponential operations take 25–60% longer than the matrix maths itself, because Blackwell roughly doubled tensor-core throughput while the shared-memory bandwidth and the exponential units barely grew (FlashAttention-4). DeepSeek's FlashMLA write-up finds it on Hopper (H800), in a different place: converting an FP8 key-value cache back into BF16 costs at least 50 cycles per token on the ordinary CUDA cores, against 34 cycles of tensor-core work — so the tensor cores idle for roughly a third of the time (FlashMLA FP8 deep dive).

The useful point for a reader is the gap between compute-bound and compute-limited. By the usual test (arithmetic per byte moved), the FlashMLA kernel is comfortably compute-bound, and it still was not running at the tensor cores' speed. A roofline drawn at the level of the model or the chip cannot see this. Only the kernel layer can, which is why it belongs on this rung.

At batch 1, the waiting moves off the GPU entirely

Both sources above measure kernels doing large amounts of work at once. For a single user generating one token at a time, a third source finds the fast chip waiting on something further away still: the CPU. On an H100 running a 7B model at batch 1, the GPU reaches only ~27% of its memory-bandwidth limit because each of ~280 kernels per token has to be launched from the host; replaying them as a CUDA Graph recovers 1.259× (batch-1 decode study). It is the same pattern — the fastest part idle, waiting on a slower one — at a different scale. See launch overhead at batch 1.

Why it moves the token price

Decode throughput on a given GPU is the denominator of output-token cost: the same GPU-hour divided over more tokens makes each token cheaper, and a cheaper token makes a finished task cheaper. The fixes in both sources change that denominator without new silicon, a new model or a new algorithm — FlashMLA's "crossover" lifted decode throughput from 250 to 410 TFLOPS (1.64×) on the same H800 with the same numerics. That multiple belongs to the kernel layer alone. It is the number this rung owns today.

Key Claims

• On Blackwell, non-matmul work now exceeds matmul work in attention. Shared-memory traffic and exponential operations exceed MMA compute by 25–60% for typical workloads, so the tensor cores are not the constraint. Evidence: moderate (preprint with benchmarks, not peer-reviewed, no independent reproduction in this KB) (FlashAttention-4)
• The fix is to spread the bottlenecked work, not to add FLOPs. FlashAttention-4 computes part of the exponentials in software as a degree-3 polynomial on the FMA units, alongside the fixed hardware exponential unit, and skips softmax rescaling when the running maximum barely moves (threshold typically log₂(256) = 8.0). Larger 128×128 tiles and accumulators held in tensor memory let the softmax overlap with the matrix work instead of taking turns. Evidence: moderate (preprint) (FlashAttention-4)
• On Hopper, FP8 dequantization starves the tensor cores: 50 cycles of conversion vs 34 of MMA per KV token. The 50 is a lower bound built from NVIDIA's published instruction throughputs, not a profiled figure; every step of the arithmetic reproduces. Evidence: moderate (vendor technical report; derivation recomputed in the close-read) (FlashMLA FP8 deep dive)
• Two CTAs sharing one conversion job lifted decode throughput 1.64×. "Crossover" launches two thread blocks as a cluster of 2; each dequantizes half the KV block and writes it straight into its partner's shared memory over Hopper's Distributed Shared Memory, so each does half the conversion and both end up with the whole block. 250 → 410 TFLOPS on H800 SXM5. The mechanism was verified in the shipped CUDA source down to the PTX instruction; the throughput is DeepSeek measuring its own kernel at one configuration, with no error bars. Evidence: moderate for the mechanism, weak-to-moderate for the number (vendor self-measurement, single configuration) (FlashMLA FP8 deep dive)
• Two independent teams, two chip generations, one conclusion. FlashAttention-4 (Blackwell, exponentials and shared memory) and FlashMLA (Hopper, format conversion) name different non-matmul culprits but the same shape of problem. Agreement between two sources lifts the general claim — the tensor core is no longer the binding constraint in attention kernels — to Evidence: moderate.

Benchmarks & Data

Result	Value	Condition	As-of	Source
Non-MMA cost over MMA cost	+25% to +60%	typical attention workloads, Blackwell	2026-03-05	FA4
Peak attention throughput	1,613 TFLOPs/s, 71% utilisation	B200, BF16	2026-03-05	FA4
vs cuDNN 9.13 / vs Triton	up to 1.3× / up to 2.7×	B200, BF16	2026-03-05	FA4
Dequant vs MMA cycles per KV token	50 vs 34	H800, FP8 KV cache, 64 heads per CTA	2025-09-29	FlashMLA
Sparse FP8 decode, without → with crossover	250 → 410 TFLOPS (1.64×)	H800 SXM5, batch 128, 128 heads, s_q 2, topk 2048	2025-09-29	FlashMLA
Same kernel at topk 32768	up to 460 TFLOPS	H800 SXM5	2025-09-29	FlashMLA
Dense BF16 decode, for reference	640 TFLOPS	H800 SXM5	2025-09-29	FlashMLA

The FlashMLA figures have not been restated in the repository since 2025-10-01, although the code kept moving (HEAD read at 2026-07-27). Treat them as an October 2025 snapshot.

Open Questions

• Neither source reports cost per token. The 1.64× is a throughput multiple; how much of it reaches a served price depends on the serving stack above it, which this rung does not hold.
• Does the Blackwell non-matmul penalty (25–60%) hold for decode at batch 1, or only for the training-shaped and prefill-shaped workloads FlashAttention-4 targets? Still open for Blackwell. On Hopper at batch 1 the binding cost turns out to be kernel launch, not the non-matmul units (batch-1 decode study) — and the study excludes Blackwell.
• Silicon that adds a direct FP8→BF16 cast deletes the FlashMLA bottleneck. Does the next part have one? This rung cannot answer that; hardware would.
• No independent reproduction of either headline number exists in this KB.

Related Concepts

• Quantized KV cache: stored small, computed large — the FlashMLA bottleneck exists because the cache is compressed and the maths is not
• Kernel portability — the fixes here are tied to one chip generation
• Launch overhead at batch 1 — the one-user case, where the GPU waits on the CPU
• KV-Cache Compression (serving rung) — the decision to store fewer bytes, one layer up

Backlinks

Pages that reference this concept:

• DeepSeek
• FlashAttention
• NVIDIA
• Kernel portability
• Quantized KV cache
• Launch overhead at batch 1