NVIDIA

Type: Company (the chips every source on this rung was measured on)

All five sources on this rung run on NVIDIA hardware, and on this rung NVIDIA appears as the thing kernels are written against: each chip generation changes which part of the chip is the bottleneck, and the kernel layer has to catch up. None of the five sources is by NVIDIA; the figures here are other people's measurements on its parts.

• Hopper (H800). Has Distributed Shared Memory, which DeepSeek's crossover trick uses, and lacks a direct FP8→BF16 conversion instruction, which is what made FP8 dequantization the bottleneck in the first place (FlashMLA).
• Blackwell (B200). Roughly doubled tensor-core throughput while shared-memory bandwidth and the exponential units barely scaled — the "asymmetric scaling" that FlashAttention-4 is written around (FlashAttention-4). Adds tensor memory and a 2-CTA MMA mode. Native microscaled FP4 support is what ReQAT's 3.1× runs on (ReQAT).
• DGX Spark. A desktop machine a person can buy; ReQAT measured 3.9× throughput over BF16 on it with 4-bit weights, activations and KV cache (ReQAT).
• H100 and L4, at batch 1. One user at a time: the H100 reaches ~27% of its memory-bandwidth limit, the L4 ~81%, because kernel launch dominates on the faster card; CUDA Graphs gives 1.259× on H100 and 1.028× on L4. With the best lever on each, H100 11.78 ms vs L4 17.36 ms per token, at list rates of $3.50/hr against $0.30/hr (May 2026) (batch-1 decode study).
• cuDNN 9.13. NVIDIA's own attention library, which FlashAttention-4 reports beating by up to 1.3× on B200 (FlashAttention-4). On B200 flash-attention forward, Triton's automatic warp specialization still trails cuDNN by 10–20% (vendor-reported by Meta's Triton team) (Triton autoWS). At the H100 batch-1 decode shape, cuDNN's attention path is "not supported" (batch-1 decode study).

Mentioned In

• The non-matmul bottleneck — Hopper and Blackwell each move the bound
• Kernel portability — the same kernel at 410 TFLOPS on H800 and 350 on B200
• FP4 reasoning quantization — the DGX Spark and B200 speedups
• Launch overhead at batch 1 — H100 against L4, one user at a time

Related Entities

• DeepSeek — wrote the Hopper FP8 kernel
• FlashAttention — the Blackwell attention kernel
• Triton — the portable compiler measured against cuDNN
• NVIDIA on the hardware rung — the silicon itself