Is a Compute Crunch Coming?

What this is

Epoch AI's serving-side model (2026-05-25) — the modelled counterpart to DigitalOcean's measured H200 benchmark. Where DigitalOcean measures $/token on one box, Epoch models how many tokens the world's chips can serve and whether supply keeps up with demand. It is the topic's authority source on price trends (its LLM-price-trends dataset is already ingested here), and this piece extends that into serving capacity. Crucially, it is calibrated against real measured experiments, so it is a modelled-but-grounded serving-cost methodology rather than a pure projection.

Methodology (modelled, calibrated)

Inference is modelled in two stages with distinct hardware bottlenecks:

• Prefill — all input tokens processed in parallel to populate the KV cache; compute-bound. Time = operations (weight multiplications + attention) ÷ hardware FLOP/s at the relevant precision (4-bit weights, 8-bit attention).
• Decode — output tokens generated sequentially, each attending to cached KVs; bandwidth-bound. Time governed by memory bandwidth, not FLOPs.
• For each stage, total time = max(compute time, bandwidth time), which is what makes chunked-prefill and speculative decoding optimizations pay off.

Calibration: the model is fitted to 111 SemiAnalysis InferenceX runs of Kimi K2.5 across varied GPU configurations. Three fitted parameters: compute efficiency 65%, bandwidth efficiency 30%, per-step latency 5ms. (This is the single most important methodological detail — it means the numbers are anchored to real serving experiments, and it names the ~30% bandwidth-efficiency reality that makes decode so expensive.)

Key quantitative findings

• Per-chip throughput: ~400,000 tokens/sec on a GB200 NVL72 serving Kimi K2.6 at an 8,000:1,000 input:output ratio.
• Global serving capacity: ~500 million to 20 billion output tokens/sec depending on request context length (long context collapses effective throughput).
• Capacity growth: ~3.4x/year, from more infrastructure + more efficient chips.
• Demand growth: ~10x/year (from Google and industry metrics).
• Central claim: a compute crunch is "likely near" — demand (~10x/yr) is outgrowing serving supply (~3.4x/yr).

Why it matters here

• Serving-cost methodology gap — the KB now holds a rigorous, InferenceX-calibrated model of the physics that set the cost floor (compute-bound prefill vs bandwidth-bound decode, ~30% bandwidth efficiency). This is the "how a token is actually served" layer underneath every price number.
• Supply-demand context for sticky pricing — a 3.4x-supply / 10x-demand gap is a mechanistic reason list-price rungs can stay sticky (Token Price Index) even as the price of fixed capability falls fast: scarce serving capacity is being rationed by price. It also gives a supply-side reading of Matsuoka's memory-scarcity thesis — bandwidth, not FLOPs, is the decode bottleneck, and bandwidth is exactly what HBM scarcity gates.
• Bridges to $/PB — decode being bandwidth-bound is the physical basis for Matsuoka's proposal to denominate inference in $/PB of bandwidth delivered rather than $/token.

Limitations

Denominated in tokens/sec, not dollars — it does not convert capacity to $/token, so it complements but does not replace a measured cost series. Kimi-K2-family calibration; generalization to other model shapes/precisions is assumed, not shown. Demand-growth (~10x/yr) is drawn from disclosed industry metrics of varying reliability.

Source: Is a Compute Crunch Coming? by Luke Emberson & Jaime Sevilla, Epoch AI, 2026-05-25