# Token Pricing & the Inference-Margin Question

Canonical URL: https://menfem.com/kb/inference-economics/concepts/token-pricing-margin-question
Knowledge base topic: [Inference & Token-Pricing Economics](https://menfem.com/kb/inference-economics)
Frontier status: active
Tags: token-pricing, inference-economics, gross-margin, commodity-infrastructure

---

The rung's umbrella question: does selling inference settle into a business with lasting margins,
or does it become low-margin plumbing, with the value captured somewhere else in the stack — the way
mobile carriers built the networks and earned little from the traffic? This page holds the
evidence on both sides. As of 2026-09-24 the honest answer is still "not decidable from what this
rung holds", and the reasons why are more useful than a guess.

**Prices are falling fast, but unevenly.** Epoch AI finds the price of reaching a fixed performance
level fell at a **median 50× a year** across six benchmarks, with a range of **9× to 900×**. The
often-quoted GPT-4-level science milestone fell at **40× a year — below the median**, and counting
only models released after January 2024 lifts the median to **200×**, a pace too recent to call
durable. *Evidence: strong (documented method; data insight published 2025-03-12, so the series
itself is now 18 months old).*

**Ben Evans (2026-07-09) says the instability is the point.** He puts inference at **40–50% gross
margin**, but that figure leaves out training, which he says is "currently far larger than revenue"
— so inference profitability is not lab profitability. He frames the outcome around four open
questions (how many uses need a frontier model, whether capability outruns efficiency, whether
competition concentrates, and whether value sits in the model or the software around it) and
reaches for the mobile-data analogy: traffic grew by orders of magnitude into a ~$1T-a-year industry
on ~$200B of capex, and "the stocks have gone nowhere". *Evidence: moderate (one analyst essay, no
primary financial data).*

**A counter-claim this rung used to carry is no longer evidence.** Until August this page set
Evans' figure against Dylan Patel's attributed claim (Podcast Alpha, 2026-07-10) that Anthropic runs
at 70%+ gross margin. That podcast was **culled from the KB on 2026-08-06** — the ruling was that
honest labelling of a conflicted source is not enough. It is not re-registered here and the figure
is not weighed. Located web reporting from 2026-07-17 (unregistered, investor-deck lineage) put
Anthropic's present-period gross margin near 50% with ~77% projected for 2028; that sits inside
Evans' range and is recorded in [frontier.md](../frontier.md), not treated as a source. Whether the
[Anthropic (economics)](../entities/anthropic-economics.md) page should drop the Patel claim entirely
is an open editorial call, flagged since 2026-08-20.

**Every margin figure fails at the same point: no cost denominator.** Patil (arXiv 2606.11690, full
text) measured effective cost on one H100 setup at **$0.21–$15.25 per million output tokens** across
configurations, priced at $6.98/GPU-hour (Azure list); for a single model, cost moved **17.5–36.3×**
between idle and full load. Evans' 40–50% states no utilisation, cost unit or as-of date. That does
not make it wrong; it makes it uncheckable. See [the cost-measurement problem](./cost-measurement-problem.md).

**A measured cost exists for one open model — on a sibling rung, and for no frontier model.**
DigitalOcean measured a 70B open model on one H200 at $20.32 per million output tokens at batch 1
and $0.45 at batch 128 (2026-07-08). That source is registered on the `serving` rung
([raw](../../../serving/raw/measured-h200-token-serving-cost-digitalocean.md)) and is context here,
not evidence. It changes nothing for the lab-margin question, which is about serving GPT-5.x, Claude
and Gemini; **no measured cost for any frontier model exists anywhere in the KB.** One cross-check
does land on this rung's own shelf: the cheapest independent-provider list price for a 70B model
(Together, $0.65/M batch, Q2 2026) sits on that measured cost — thin-to-zero margin at the batch tier
([independent providers](../entities/independent-inference-providers.md)).

**The unit is probably wrong, and there is a framework for a better one.** AI Tokenomics (Zhu,
arXiv 2606.24616, theoretical, abstract read) argues that tokens spent and value produced are
different quantities — per-token pricing prices the input, not the output. Cost-of-Pass (Erol et al.,
arXiv 2504.13359, abstract read) gives the working fix: **cost per correct answer** = cost per
attempt ÷ accuracy. That is the unit in which "lasting margin or plumbing?" should finally be
settled — see [cost per task](./cost-per-task.md). The rung does now hold a dated task-price range —
**$0.17–$6.55 per task** across 11 models (STAGE-Claw) — but those figures were surfaced by search
and still have not been confirmed from the paper.

**The May-2024 break leans toward Evans' pole — weakly.** Tiered Super-Moore's Law (Du, arXiv
2603.28576, full text read 2026-09-11) dates a shift from technology-driven to competition-driven
price decline (Chow F = 5.74, p = 0.005) with concentration falling (HHI 4,558 → 2,086). That fits
"prices compress toward cost as competition bites" better than "a few landlords keep pricing power".
Two cautions from the full text: only 16 observations sit before the break, and the market-share
figures are estimated without sales-volume data. The same paper attributes ~103.7% of the cost fall
to productivity and −0.9% to GPU hardware — a residual, but one that cuts against the
memory-scarcity story (see [the price-decline distribution](./price-decline-distribution.md)).

**The first quantitative model of the question calls it a coin flip.** Matsuoka (arXiv 2607.07207)
gives five scenarios, with the two poles **tied at 25% each** — Rotating Landlord Oligopoly 25%,
Commoditization Crash 25%, Jevons Absorption 20%, System-Layer Re-differentiation 18%, Geopolitical
Bifurcation 12%. These are **expert judgments revised over five rounds**, not computed outputs, from
a single-author preprint; treat them as a clear statement that the question is not currently
decidable. His mechanism — the depreciation conveyor, under which incumbents stay cheaper than
entrants because paid-off fleets keep arriving — now lives on the `capital-structure` rung
([concept page](../../../capital-structure/wiki/concepts/depreciation-conveyor.md)). Note that the
two questions separate: inference can become cheap plumbing overall while incumbents stay
permanently cheaper than newcomers.

**The sell-side rung — held, then partly broken.** When GPT-5.6 was recorded into the Token Price
Index on 2026-07-16, two of its three tiers landed on the prior generation's exact price ($5/$30 and
$2.50/$15) — a provider keeping the capability gain instead of passing it on as a lower price.
**Fourteen days later (2026-07-30) OpenAI cut Terra 20% and Luna 80%; only the frontier tier held.**
Anthropic showed the same shape: frontier held, strong tier cut. So the surviving claim is narrower:
*frontier list prices hold; the tiers below them do not.* That is recorded on
[price-rung persistence](./price-rung-persistence.md), a page kept as a dated archive since the Index
was retired on 2026-08-19. OpenAI's own announcement of the cut, ingested 2026-09-24
([raw](../../raw/openai-gpt-5-6-price-cut-fast-mode-2026-07-30.md)), names a 20% serving-cost
reduction as the cause — and shows the narrower claim failing too: **the frontier tier was cut on
2026-08-21** (Sol to a promotional $4/$20) and GPT-6 Sol launched at $2/$10 on 2026-09-22. A 20% cost
saving passed on as a 20% cut (Terra) is the plumbing reading; an 80% cut (Luna) that goes far
beyond it is either a margin being given up or a margin that was larger than the price implied —
the page cannot tell which without a cost figure.

## Key Claims

- **Median inference price decline: 50×/year, range 9×–900×/year** across six benchmarks.
  *Evidence: strong* ([Epoch AI](../../raw/epoch-ai-llm-inference-price-trends.md))
- **GPT-4-level science milestone: 40×/year — below the median.** *Evidence: strong*
  ([Epoch AI](../../raw/epoch-ai-llm-inference-price-trends.md))
- **Post-January-2024 median: 200×/year** (short track record). *Evidence: moderate*
  ([Epoch AI](../../raw/epoch-ai-llm-inference-price-trends.md))
- **Inference gross margin ~40–50%, excluding training, which exceeds revenue.** *Evidence: moderate*
  ([Ben Evans](../../raw/ben-evans-ways-to-think-about-token-pricing.md))
- **Mobile-data analogy: ~$1T/yr industry on ~$200B capex, carrier stocks flat.** *Evidence:
  moderate (historical analogy)* ([Ben Evans](../../raw/ben-evans-ways-to-think-about-token-pricing.md))
- **TSMC's near-monopoly nets $53B, "less than half of Apple alone".** *Evidence: moderate*
  ([Ben Evans](../../raw/ben-evans-ways-to-think-about-token-pricing.md))
- **Evans' margin figure states no utilisation, cost unit or as-of date.** *Evidence: strong
  (visible absence)* ([Evans](../../raw/ben-evans-ways-to-think-about-token-pricing.md))
- **Effective cost on one H100 setup spans $0.21–$15.25 per million output tokens across
  configurations at $6.98/GPU-hr; 17.5–36.3× for one model between idle and full load (June 2026).**
  *Evidence: moderate (measured, named price basis)* ([Patil](../../raw/beyond-per-token-pricing-concurrency-aware-cost-methodology.md))
- **No measured cost for any frontier or proprietary model exists in the KB.** *Evidence: strong
  (absence, checked 2026-09-24 across this rung and `serving`)*
- **The commercially right unit is cost per correct answer, not price per token.** *Evidence:
  moderate as frameworks (two abstract-read preprints)* ([Cost-of-Pass](../../raw/cost-of-pass-economic-framework-language-models.md), [AI Tokenomics](../../raw/ai-tokenomics-economics-tokens-computation-pricing.md))
- **A dated task-price range exists: ~$0.17–$6.55 per task across 11 models (June 2026).**
  *Evidence: weak-moderate — search-surfaced from the paper's cost table, still unconfirmed*
  ([STAGE-Claw](../../raw/stage-claw-state-based-agent-benchmark-cost-per-task.md))
- **A May-2024 break shifts the decline from technology- to competition-driven (HHI 4,558 → 2,086).**
  *Evidence: moderate-weak — estimated; 16 pre-break observations; shares proxy-estimated*
  ([Du](../../raw/tiered-super-moores-law-inference-price-evolution.md))
- **The first quantitative scenario model ties the two poles at 25% each.** *Evidence: weak (expert
  judgment over five rounds, single-author preprint)* ([Matsuoka](../../raw/memory-scarcity-open-models-ai-industry-restructuring-2026-2030.md))
- **Buildout solvency is modelled as needing ~2× annual token-demand growth for four years AND sticky
  premium pricing.** *Evidence: weak-moderate (modelled)* ([Matsuoka](../../raw/memory-scarcity-open-models-ai-industry-restructuring-2026-2030.md))
- **GPT-5.6 held two of three prior-generation price rungs at release (2026-07-16).** *Evidence: strong
  as a dated observation (own dataset, primary vs OpenAI's page)* — **since partly broken: Terra and
  Luna were cut on 2026-07-30, and Sol (promotionally) on 2026-08-21** — confirmed on OpenAI's own
  changelog and pricing page ([TPI](../../raw/tpi-gpt-5-6-rung-hold-2026-07-16.md),
  [OpenAI](../../raw/openai-gpt-5-6-price-cut-fast-mode-2026-07-30.md))
- **Inference cost may pass through to economy-wide inflation (US slope 0.087, s.e. 0.021, data to
  2026-04).** *Evidence: weak — unreplicated; cost series is list-price-based; the size term λ̄ = 0.18
  is stipulated* ([ICPC preprint](../../raw/economics-of-ai-inference-phillips-curve.md))

## Open Questions

- **What does a frontier model cost to serve?** Nothing in the KB measures it. Until something does,
  every lab-margin figure stays uncheckable.
- **Will an audited number ever arrive?** Anthropic's confidential draft S-1 (reported filed
  ~2026-06-01) is the only dated path to audited lab financials; no public S-1 had been located as of
  this recompile, which has not searched for one.
- **Is the frontier rung the next to break?** The strong and mid tiers broke within weeks in July; the
  frontier tier held at both OpenAI and Anthropic. Under Matsuoka's corridor a frontier cut would be a
  solvency signal, not just a price move.
- **Does the post-2024 200×/year pace persist?** Epoch's series is from March 2025; Du's ends March
  2026. Nothing on this rung measures the pace after that.
- **Which of Evans' four variables is moving fastest right now?**
- **Are "does inference become plumbing?" and "do incumbents stay cheaper?" the same question?** The
  depreciation conveyor says no.
- **Is the token even the right unit?** Cost per task says no; Matsuoka's $/PB says no for a different
  reason.

## Related Concepts

- [The Price-Decline Distribution](./price-decline-distribution.md) — the buy side: how fast the price
  of fixed capability falls.
- [Price-Rung Persistence](./price-rung-persistence.md) — the sell side: providers holding list prices
  while capability rises, and where that broke. Dated archive.
- [The Cost-Measurement Problem](./cost-measurement-problem.md) — the missing denominator under every
  margin claim on this page.
- [Cost Per Task (Cost-of-Pass)](./cost-per-task.md) — the unit in which this question should
  finally be settled.
- [Inference Cost as a Macro Input](./inference-cost-macro-channel.md) — the "does this matter outside
  the lab P&L?" branch. Speculative.
- [Inference & Token Economics](./inference-token-economics.md) — the July–August 2026 price moves by
  three labs, mostly from sibling-rung sources.
- *Sibling rungs:* [the depreciation conveyor](../../../capital-structure/wiki/concepts/depreciation-conveyor.md)
  (`capital-structure`) · [HBM4 memory architecture](../../../hardware/wiki/concepts/hbm4-memory-architecture.md)
  and [memory scarcity & inference economics](../../../hardware/wiki/concepts/memory-scarcity-inference-economics.md)
  (`hardware`) — memory cost and supply as inputs to the cost of a token.

## Backlinks

*Pages that reference this concept:*
- [Anthropic (economics)](../entities/anthropic-economics.md)
- [The Cost-Measurement Problem](./cost-measurement-problem.md)
- [Cost Per Task](./cost-per-task.md)
- [Inference Cost as a Macro Input](./inference-cost-macro-channel.md)
- [Anthropic](../../../models/wiki/entities/anthropic.md) (`models` rung)

## Changelog

- **2026-09-24 (source ingest)** — OpenAI's GPT-5.6 price-cut announcement ingested: the July cut is
  now primary-sourced with a stated cost cause, and the "frontier list prices hold" narrowing is
  itself broken (Sol cut 2026-08-21, promotional). Removed the "proposed, awaiting approval" notes.
- **2026-09-24 (recompile, no new source)** — Removed the culled Patel podcast from this page's
  evidence (culled 2026-08-06; the open editorial call on the entity page is flagged, not decided
  here). Moved DigitalOcean's measured H200 cost to sibling-rung context — it is registered on
  `serving`, and this page cited a raw file that no longer exists here (its slug stays in `sources`, which the site resolves to the `serving` ledger). Corrected Matsuoka's scenario
  weights from "no elicitation method" to "expert judgment over five rounds". Brought Du up to the
  full-text read. Added the 2026-07-30 GPT-5.6 cut to the rung paragraph so the page no longer
  presents a partly-falsified hold as current. Added the STAGE-Claw task-price range (graded
  weak-moderate). Fixed the depreciation-conveyor, Anthropic and HBM4 links; dropped a
  `co-packaged-optics` link to a page that does not exist.
- **2026-07-23** — Compiled the six-source discovery sweep against the umbrella.
- **2026-07-22** — Compiled Matsuoka, Patil and the ICPC preprint; margin figures recorded as
  uncheckable without a cost denominator.
- **2026-07-16** — Split out the price-decline distribution and price-rung persistence.
- **2026-07-14** — Initial compilation (Evans, Epoch AI, and the Patel podcast later culled).

## Sources

- ben-evans-ways-to-think-about-token-pricing
- epoch-ai-llm-inference-price-trends
- memory-scarcity-open-models-ai-industry-restructuring-2026-2030
- beyond-per-token-pricing-concurrency-aware-cost-methodology
- economics-of-ai-inference-phillips-curve
- tiered-super-moores-law-inference-price-evolution
- cost-of-pass-economic-framework-language-models
- ai-tokenomics-economics-tokens-computation-pricing
- independent-inference-provider-pricing-matrix-q2-2026
- stage-claw-state-based-agent-benchmark-cost-per-task
- tpi-gpt-5-6-rung-hold-2026-07-16
- openai-gpt-5-6-price-cut-fast-mode-2026-07-30
- measured-h200-token-serving-cost-digitalocean

---

Cite as: MenFem Knowledge Base — https://menfem.com/kb/inference-economics/concepts/token-pricing-margin-question