Inference & Token-Pricing Economics

The economics of serving AI models: what a token actually costs to produce, how fast that cost is falling, whether the margin on top of it is durable or transitory, and how upstream cost structure (memory, optics, foundry) feeds through into the price a buyer pays. Created 2026-07-14 because no existing KB topic tracked token-pricing/inference-margin economics as a first-class subject.

As of the 2026-07-23 compile, the topic has four layers. At the top, the umbrella question — the inference-margin question: durable-margin business or commodity infrastructure? Beneath it, two mirror-image price empirics: the price-decline distribution (buy-side — how fast the price of fixed capability falls: Epoch's median 50x/yr, 9x-900x range; now with Du's tier half-lives, economy 1.10yr / mid 1.55yr) and price-rung persistence (sell-side — providers hold nominal per-token rungs while capability rises: GPT-5.6 held two of three prior-generation rungs to the cent). Underneath both, the cost layer: the cost-measurement problem (now holding the topic's first measured cost-to-serve series) and the depreciation conveyor (whose cost is lower, why, and what the buildout needs to stay solvent). And a unit layer that says the token is the wrong denominator: cost per task (price a correct answer, not a token). A seventh concept, inference cost as a macro input, holds the speculative branch: what if inference cost passes through to the general price level.

The 2026-07-23 compile's headline is that the topic's defining gap is partly closed — and the remaining gap is now sharply stated. DigitalOcean measured a 70B open model (Llama-3.3-70B FP8) on a single H200 at $20.32/M at batch=1 → $0.45/M at batch=128 (~44x from traffic shape alone; 2026-07-08). That is the topic's first measured cost-to-serve series, and it independently reproduces Patil's utilization thesis in dollars. But it is a POINT, not a series (one SKU / one open model / one framework / one date), and — the sharpest remaining gap — there is still no measured cost floor for any frontier/proprietary model, so the Evans-40% vs Patel-70% lab-margin debate stays unfalsifiable. The cheapest independent-provider list price (Together $0.65/M batch, 70B) sits on that measured floor → thin batch-tier margin.

A new first-order conflict is on the record, unresolved. Tiered Super-Moore's Law attributes ~103.7% of the token-price decline to software/total-factor productivity and –0.9% to GPU hardware, cutting directly against the memory/hardware-scarcity-as-destiny framing of Matsuoka and Patel. Memory may gate capacity while software drives price; the sources locate the cost lever in different layers. Recorded on the price-decline and depreciation-conveyor pages and as frontier Conflict 7 — not adjudicated.

Evidence discipline. Epoch AI = strong (documented methodology). The Token Price Index rung observation = strong, own dataset, primary against OpenAI's own pricing page (2026-07-16). DigitalOcean's cost floor = measured throughput × list rate, read in full, but analysis-grade and single-config. Epoch's serving-capacity model = modelled but calibrated to 111 measured InferenceX runs, read in full. Ben Evans = moderate (single analyst essay). The six arXiv sources = preprints, single-author or small-team, not peer-reviewed; the four newest (Tiered Super-Moore's, Cost-of-Pass, AI Tokenomics, and the DigitalApplied matrix's arXiv siblings) were read from abstract/article pages only — their measured ratios/trajectories are moderate, their modelled projections are weak, and that split is marked on every page. Dylan Patel's Anthropic-margin claim = weak (attributed, disclosed conflict, paywalled). Every cost or price figure carries its as-of date, and every number is labelled measured / modelled / projected / estimated / derived / observed. No directional calls.

Concept Map

Concepts

Entities

Topic-lensed stubs only — canonical company profiles live in the ai/ topic and are cross-linked, not duplicated.

Sources

All 13 registered sources are ingested and compiled; staleSources is 0. DigitalOcean and Epoch (compute-crunch) were read in full; the four newest arXiv/analysis economics sources (Tiered Super-Moore's, Cost-of-Pass, AI Tokenomics, DigitalApplied matrix) were read from abstract/article pages only. The three older arXiv preprints (Matsuoka, Patil, ICPC) have had full-text reconciliation/touch-ups. Every raw file states its read-depth at the top. Zero peer-reviewed papers.

Cross-Topic Links

• The Matsuoka preprint (arXiv 2607.07207) is also held in hardware/, with a desk close-read at hardware/raw/memory-scarcity-2607.07207-closeread.md. This topic holds the inference-economics lens on it ($/PB as a cost unit, the depreciation conveyor, the solvency corridor); hardware/ owns the memory-supply read. Deliberately not duplicated.
• Memory-shortage claim (Patel, attributed) folded into hardware/wiki/concepts/hbm4-memory-architecture.md
• CPO-ramp-timing claim (Patel, attributed) folded into optical-computing/wiki/concepts/co-packaged-optics.md
• Anthropic financial claim (Patel, attributed) canonical on ai/wiki/entities/anthropic.md; economics-lensed stub here at entities/anthropic-economics.md
• OpenAI canonical profile at ai/wiki/entities/openai.md; pricing-lensed stub here at entities/openai-pricing.md

Timeline

See timeline.md for chronological developments.

Research Frontier

See frontier.md.