Token Pricing & the Inference-Margin Question

The rung's umbrella question: does selling inference settle into a business with lasting margins, or does it become low-margin plumbing, with the value captured somewhere else in the stack — the way mobile carriers built the networks and earned little from the traffic? This page holds the evidence on both sides. As of 2026-09-24 the honest answer is still "not decidable from what this rung holds", and the reasons why are more useful than a guess.

Prices are falling fast, but unevenly. Epoch AI finds the price of reaching a fixed performance level fell at a median 50× a year across six benchmarks, with a range of 9× to 900×. The often-quoted GPT-4-level science milestone fell at 40× a year — below the median, and counting only models released after January 2024 lifts the median to 200×, a pace too recent to call durable. Evidence: strong (documented method; data insight published 2025-03-12, so the series itself is now 18 months old).

Ben Evans (2026-07-09) says the instability is the point. He puts inference at 40–50% gross margin, but that figure leaves out training, which he says is "currently far larger than revenue" — so inference profitability is not lab profitability. He frames the outcome around four open questions (how many uses need a frontier model, whether capability outruns efficiency, whether competition concentrates, and whether value sits in the model or the software around it) and reaches for the mobile-data analogy: traffic grew by orders of magnitude into a ~$1T-a-year industry on ~$200B of capex, and "the stocks have gone nowhere". Evidence: moderate (one analyst essay, no primary financial data).

A counter-claim this rung used to carry is no longer evidence. Until August this page set Evans' figure against Dylan Patel's attributed claim (Podcast Alpha, 2026-07-10) that Anthropic runs at 70%+ gross margin. That podcast was culled from the KB on 2026-08-06 — the ruling was that honest labelling of a conflicted source is not enough. It is not re-registered here and the figure is not weighed. Located web reporting from 2026-07-17 (unregistered, investor-deck lineage) put Anthropic's present-period gross margin near 50% with ~77% projected for 2028; that sits inside Evans' range and is recorded in frontier.md, not treated as a source. Whether the Anthropic (economics) page should drop the Patel claim entirely is an open editorial call, flagged since 2026-08-20.

Every margin figure fails at the same point: no cost denominator. Patil (arXiv 2606.11690, full text) measured effective cost on one H100 setup at $0.21–$15.25 per million output tokens across configurations, priced at $6.98/GPU-hour (Azure list); for a single model, cost moved 17.5–36.3× between idle and full load. Evans' 40–50% states no utilisation, cost unit or as-of date. That does not make it wrong; it makes it uncheckable. See the cost-measurement problem.

A measured cost exists for one open model — on a sibling rung, and for no frontier model. DigitalOcean measured a 70B open model on one H200 at $20.32 per million output tokens at batch 1 and $0.45 at batch 128 (2026-07-08). That source is registered on the serving rung (raw) and is context here, not evidence. It changes nothing for the lab-margin question, which is about serving GPT-5.x, Claude and Gemini; no measured cost for any frontier model exists anywhere in the KB. One cross-check does land on this rung's own shelf: the cheapest independent-provider list price for a 70B model (Together, $0.65/M batch, Q2 2026) sits on that measured cost — thin-to-zero margin at the batch tier (independent providers).

The unit is probably wrong, and there is a framework for a better one. AI Tokenomics (Zhu, arXiv 2606.24616, theoretical, abstract read) argues that tokens spent and value produced are different quantities — per-token pricing prices the input, not the output. Cost-of-Pass (Erol et al., arXiv 2504.13359, abstract read) gives the working fix: cost per correct answer = cost per attempt ÷ accuracy. That is the unit in which "lasting margin or plumbing?" should finally be settled — see cost per task. The rung does now hold a dated task-price range — $0.17–$6.55 per task across 11 models (STAGE-Claw) — but those figures were surfaced by search and still have not been confirmed from the paper.

The May-2024 break leans toward Evans' pole — weakly. Tiered Super-Moore's Law (Du, arXiv 2603.28576, full text read 2026-09-11) dates a shift from technology-driven to competition-driven price decline (Chow F = 5.74, p = 0.005) with concentration falling (HHI 4,558 → 2,086). That fits "prices compress toward cost as competition bites" better than "a few landlords keep pricing power". Two cautions from the full text: only 16 observations sit before the break, and the market-share figures are estimated without sales-volume data. The same paper attributes ~103.7% of the cost fall to productivity and −0.9% to GPU hardware — a residual, but one that cuts against the memory-scarcity story (see the price-decline distribution).

The first quantitative model of the question calls it a coin flip. Matsuoka (arXiv 2607.07207) gives five scenarios, with the two poles tied at 25% each — Rotating Landlord Oligopoly 25%, Commoditization Crash 25%, Jevons Absorption 20%, System-Layer Re-differentiation 18%, Geopolitical Bifurcation 12%. These are expert judgments revised over five rounds, not computed outputs, from a single-author preprint; treat them as a clear statement that the question is not currently decidable. His mechanism — the depreciation conveyor, under which incumbents stay cheaper than entrants because paid-off fleets keep arriving — now lives on the capital-structure rung (concept page). Note that the two questions separate: inference can become cheap plumbing overall while incumbents stay permanently cheaper than newcomers.

The sell-side rung — held, then partly broken. When GPT-5.6 was recorded into the Token Price Index on 2026-07-16, two of its three tiers landed on the prior generation's exact price ($5/$30 and $2.50/$15) — a provider keeping the capability gain instead of passing it on as a lower price. Fourteen days later (2026-07-30) OpenAI cut Terra 20% and Luna 80%; only the frontier tier held. Anthropic showed the same shape: frontier held, strong tier cut. So the surviving claim is narrower: frontier list prices hold; the tiers below them do not. That is recorded on price-rung persistence, a page kept as a dated archive since the Index was retired on 2026-08-19. OpenAI's own announcement of the cut, ingested 2026-09-24 (raw), names a 20% serving-cost reduction as the cause — and shows the narrower claim failing too: the frontier tier was cut on 2026-08-21 (Sol to a promotional $4/$20) and GPT-6 Sol launched at $2/$10 on 2026-09-22. A 20% cost saving passed on as a 20% cut (Terra) is the plumbing reading; an 80% cut (Luna) that goes far beyond it is either a margin being given up or a margin that was larger than the price implied — the page cannot tell which without a cost figure.

Key Claims

• Median inference price decline: 50×/year, range 9×–900×/year across six benchmarks. Evidence: strong (Epoch AI)
• GPT-4-level science milestone: 40×/year — below the median. Evidence: strong (Epoch AI)
• Post-January-2024 median: 200×/year (short track record). Evidence: moderate (Epoch AI)
• Inference gross margin ~40–50%, excluding training, which exceeds revenue. Evidence: moderate (Ben Evans)
• Mobile-data analogy: ~$1T/yr industry on ~$200B capex, carrier stocks flat. Evidence: moderate (historical analogy) (Ben Evans)
• TSMC's near-monopoly nets $53B, "less than half of Apple alone". Evidence: moderate (Ben Evans)
• Evans' margin figure states no utilisation, cost unit or as-of date. Evidence: strong (visible absence) (Evans)
• Effective cost on one H100 setup spans $0.21–$15.25 per million output tokens across configurations at $6.98/GPU-hr; 17.5–36.3× for one model between idle and full load (June 2026). Evidence: moderate (measured, named price basis) (Patil)
• No measured cost for any frontier or proprietary model exists in the KB. Evidence: strong (absence, checked 2026-09-24 across this rung and serving)
• The commercially right unit is cost per correct answer, not price per token. Evidence: moderate as frameworks (two abstract-read preprints) (Cost-of-Pass, AI Tokenomics)
• A dated task-price range exists: ~$0.17–$6.55 per task across 11 models (June 2026). Evidence: weak-moderate — search-surfaced from the paper's cost table, still unconfirmed (STAGE-Claw)
• A May-2024 break shifts the decline from technology- to competition-driven (HHI 4,558 → 2,086). Evidence: moderate-weak — estimated; 16 pre-break observations; shares proxy-estimated (Du)
• The first quantitative scenario model ties the two poles at 25% each. Evidence: weak (expert judgment over five rounds, single-author preprint) (Matsuoka)
• Buildout solvency is modelled as needing ~2× annual token-demand growth for four years AND sticky premium pricing. Evidence: weak-moderate (modelled) (Matsuoka)
• GPT-5.6 held two of three prior-generation price rungs at release (2026-07-16). Evidence: strong as a dated observation (own dataset, primary vs OpenAI's page) — since partly broken: Terra and Luna were cut on 2026-07-30, and Sol (promotionally) on 2026-08-21 — confirmed on OpenAI's own changelog and pricing page (TPI, OpenAI)
• Inference cost may pass through to economy-wide inflation (US slope 0.087, s.e. 0.021, data to 2026-04). Evidence: weak — unreplicated; cost series is list-price-based; the size term λ̄ = 0.18 is stipulated (ICPC preprint)

Open Questions

• What does a frontier model cost to serve? Nothing in the KB measures it. Until something does, every lab-margin figure stays uncheckable.
• Will an audited number ever arrive? Anthropic's confidential draft S-1 (reported filed ~2026-06-01) is the only dated path to audited lab financials; no public S-1 had been located as of this recompile, which has not searched for one.
• Is the frontier rung the next to break? The strong and mid tiers broke within weeks in July; the frontier tier held at both OpenAI and Anthropic. Under Matsuoka's corridor a frontier cut would be a solvency signal, not just a price move.
• Does the post-2024 200×/year pace persist? Epoch's series is from March 2025; Du's ends March 2026. Nothing on this rung measures the pace after that.
• Which of Evans' four variables is moving fastest right now?
• Are "does inference become plumbing?" and "do incumbents stay cheaper?" the same question? The depreciation conveyor says no.
• Is the token even the right unit? Cost per task says no; Matsuoka's $/PB says no for a different reason.

Related Concepts

• The Price-Decline Distribution — the buy side: how fast the price of fixed capability falls.
• Price-Rung Persistence — the sell side: providers holding list prices while capability rises, and where that broke. Dated archive.
• The Cost-Measurement Problem — the missing denominator under every margin claim on this page.
• Cost Per Task (Cost-of-Pass) — the unit in which this question should finally be settled.
• Inference Cost as a Macro Input — the "does this matter outside the lab P&L?" branch. Speculative.
• Inference & Token Economics — the July–August 2026 price moves by three labs, mostly from sibling-rung sources.
• Sibling rungs: the depreciation conveyor (capital-structure) · HBM4 memory architecture and memory scarcity & inference economics (hardware) — memory cost and supply as inputs to the cost of a token.

Backlinks

Pages that reference this concept:

• Anthropic (economics)
• The Cost-Measurement Problem
• Cost Per Task
• Inference Cost as a Macro Input
• Anthropic (models rung)

Changelog

• 2026-09-24 (source ingest) — OpenAI's GPT-5.6 price-cut announcement ingested: the July cut is now primary-sourced with a stated cost cause, and the "frontier list prices hold" narrowing is itself broken (Sol cut 2026-08-21, promotional). Removed the "proposed, awaiting approval" notes.
• 2026-09-24 (recompile, no new source) — Removed the culled Patel podcast from this page's evidence (culled 2026-08-06; the open editorial call on the entity page is flagged, not decided here). Moved DigitalOcean's measured H200 cost to sibling-rung context — it is registered on serving, and this page cited a raw file that no longer exists here (its slug stays in sources, which the site resolves to the serving ledger). Corrected Matsuoka's scenario weights from "no elicitation method" to "expert judgment over five rounds". Brought Du up to the full-text read. Added the 2026-07-30 GPT-5.6 cut to the rung paragraph so the page no longer presents a partly-falsified hold as current. Added the STAGE-Claw task-price range (graded weak-moderate). Fixed the depreciation-conveyor, Anthropic and HBM4 links; dropped a co-packaged-optics link to a page that does not exist.
• 2026-07-23 — Compiled the six-source discovery sweep against the umbrella.
• 2026-07-22 — Compiled Matsuoka, Patil and the ICPC preprint; margin figures recorded as uncheckable without a cost denominator.
• 2026-07-16 — Split out the price-decline distribution and price-rung persistence.
• 2026-07-14 — Initial compilation (Evans, Epoch AI, and the Patel podcast later culled).