Inference & Token-Pricing Economics

This rung is about what a token costs to make and what it sells for — token pricing trends, inference gross margins, the AI-lab profit question, and how memory, energy and hardware costs feed through to the price of a token. Its sharpest observation is that the two numbers are only loosely attached: Quanyan Zhu's tokenomics framework argues that tokens spent and value produced are different quantities, so pricing by tokens "prices the input, not the output" (AI Tokenomics, abstract read only, and explicitly theoretical — it carries no measured figures).

Freshness, stated plainly (2026-09-24, 12:30). Three new sources were ingested today, the first since 2026-08-06, all on approval: OpenAI's own account of its July GPT-5.6 price cut, Anthropic's Fable 5.1 cache-read cut, and Intercom's Fin price per outcome. They bring the rung's newest price observations up to September 2026. They are vendor disclosures, not measurements: the rung still holds no first-hand cost figure. What is still stale is in frontier.md.

What this rung prices

A quoted price per million tokens is the start of the sum, not the end. Three multipliers sit between it and the price of a finished job. First, the seller's side: the same GPU serving the same model has a very different cost per token depending on how busy it is, so a cost figure without a utilisation figure is not a cost figure. Second, the buyer's side: a task is only done when it is done correctly, so the unit that settles a purchase is cost per correct answer, not cost per token — a cheap model that is often wrong can cost more per finished job than an expensive one. Third, the decline: prices per token have fallen fast but very unevenly, landing on some tiers and tasks and not others, so "inference is getting cheaper" means nothing until you ask which tier. Get those three right and a token price turns into a task price; skip one and the answer is off by an order of magnitude or more.

The load-bearing findings

• Same GPU, same model: cost moved 17–36× with load alone. On one H100 setup priced at $6.98/GPU-hour (Azure H100 NVL on-demand list), effective cost ran from $0.21 to $15.25 per million output tokens across configurations; for a single model the spread from idle to full load was 17.5–36.3×, and 2.5–24× across an ordinary 1–10 requests-per-second load. An estimate that ignores utilisation understates cost by exactly 1/U, which makes self-hosting look cheaper than it is (Beyond Per-Token Pricing, full text). This is the cost-measurement problem in one number. (Earlier versions said "identical model, 73× spread"; the 73× mixes two different models.)
• The right unit is cost per correct answer, and it halves every few months. Cost-of-pass is cost per attempt divided by the chance the attempt is right; measured that way, the best available cost on hard maths-style tasks "roughly halved every few months", driven by better models rather than inference tricks (Cost-of-Pass, abstract only). It also explains why an expensive reasoning model gets bought. See cost per task.
• Agent tasks: cents at the cheap end, single-digit dollars at the top — unconfirmed. Across 40 tasks and 11 frontier models, API cost per task reportedly ran $0.17 to $6.55 (STAGE-Claw). Those figures were surfaced by search from the paper's cost table, never read directly, and are still unconfirmed two months after ingest. List prices, not what anyone paid.
• The price decline is real, fast and unequal — the median hides the number you need. The price of a fixed performance level fell at a median 50× a year across six benchmarks, range 9× to 900×; the famous GPT-4-level science milestone fell 40× a year, below the median (Epoch AI, March 2025). Split by tier, the price half-life is about 1.10 years for cheap models and 1.55 years for mid-range ones, while flagship prices barely follow a trend at all — but the full text (read 2026-09-11) shows weak fits (the cheap-tier fit explains under a fifth of the variation), tiers defined by price so models move between them, and a series that ends 2026-03-28 (Tiered Super-Moore's Law). See the price-decline distribution.
• The decline lands on the lower tiers, not the frontier. A workplace-agent benchmark re-run two years on saw best-agent completion go 43% → 98% and harmful mistakes 26% → 1.9%, and says open-weight models "drastically lowered" the cost of a once-proprietary performance level while frontier costs stayed flat (WorkBench Revisited, abstract only; a ~100× size has been attributed to the unread body). Provider prices showed that shape in July — but it did not last. OpenAI cut GPT-5.6's two lower tiers 20% and 80% three weeks after launch, then put the frontier model on a promotional $4/$20 (from $5/$30) on 2026-08-21, and launched GPT-6 Sol at $2/$10 on 2026-09-22 (OpenAI). What holds now is a price point at the very top — $10/$50 at both OpenAI and Anthropic — with the named tiers falling beneath it. See price-rung persistence, corrected 2026-09-24.
• A price cut with a named cause: software, on the same chips. OpenAI says it cut Luna 80% and Terra 20% on 2026-07-30 because GPT-5.6, set to optimise its own serving, found 20% lower serving cost from GPU kernel work and 15%+ better token generation from speculative decoding (OpenAI, vendor-stated percentages, no dollar base). The 20% explains Terra's cut; Luna's 80% is mostly a pricing choice.
• The price of an agent's task fell while the headline rate stood still. Claude Fable 5.1 kept $10/$50 per million tokens and cut only cache reads — re-used stored context — by 75%, $1.00 → $0.25. Anthropic estimates ~25% off a typical bill and up to ~45% off agent-heavy ones (Anthropic). On this rung it counts only because the cache line is the mechanism; plain list-price moves were ruled out of scope on 2026-09-13. See inference & token economics.
• The first published price per task: $0.99 per outcome. Intercom sells its Fin support agent by the finished job, charged once per conversation however many steps it takes (a sales qualification is $9.99) — the token count is hidden inside the price, and its fall accrues to the seller (Intercom, vendor pages as of 2026-09-24). See cost per task.
• Nobody on this rung can tell you what a token costs a lab to make. Benedict Evans puts inference at 40–50% gross margin but leaves out training, "currently far larger than revenue"; his conclusion is that the plumbing-or-lasting-margin question is unresolved (Ways to Think About Token Pricing). That is the margin question, still open.
• At the thin-margin layer, price is still a product choice, not a cost floor. For one model (Llama 4 70B output), Q2-2026 list prices spread ~6× — $0.65 on Together's batch tier to $4.20 on Cerebras — tracking latency, service guarantees and hardware rather than cost to serve (Q2 2026 provider pricing matrix, an analysis-grade blog, one model, list prices).

What has changed since July — from the sibling rungs

Between 2026-08-06 and 2026-09-24 this rung took in no new source. Its neighbours did, and some of what they now hold bears directly on token cost. These are context with their own links, not this rung's evidence:

• serving holds the KB's two measured cost-to-serve results: DigitalOcean's H200 at $20.32 → $0.45 per million output tokens from batch 1 to batch 128 (raw, 2026-07-08) and Epoch's capacity model — serving capacity growing ~3.4×/year against demand ~10× (raw). Both were once compiled here; they are registered on serving. It also now holds the KB's first goodput result: splitting prefill from decode on the same eight GPUs raised requests meeting their service level from 26 to 73 per 100 (raw).
• harnesses holds a controlled cost-per-task result: changing only the orchestration layer cut cost per task $0.21 → $0.12 (−41%) at equal quality (raw, 2026-07-08) — the harness moved the task price more than the choice of model did.
• compilers holds kernel-level gains on unchanged chips — FlashMLA's FP8 decode kernel 1.64× faster on Hopper (raw), FlashAttention-4 at 71% utilisation on B200 (raw) — the kind of software mechanism behind "the cost fall came from software, not silicon".
• capital-structure holds the cost of money behind GPU fleets: CoreWeave borrowed at SOFR+450 in May and SOFR+550 in August 2026 (May, August) — a widening spread that feeds straight into the per-hour cost of a GPU — and the depreciation conveyor concept this rung used to hold.
• energy holds record US generation forecasts (EIA, 2026-09-09) alongside a Texas pause on connecting new data-centre load (ERCOT, 2026-08-03): energy plentiful in total, power scarce at the site (EIA, ERCOT).
• hardware holds SK hynix's 76% operating margin for Q2 2026 — memory scarcity stated as a margin (raw).

What we do not know yet

• Almost nothing here was measured first-hand. The one own-dataset source is a Token Price Index reading of a published pricing page, not a cost measurement, and that index is retired. No cost curve on this rung has been reproduced locally; vllm-cost-meter has not been run.
• Five of the seventeen sources were read at abstract level only — Cost-of-Pass, AI Tokenomics, STAGE-Claw, WorkBench Revisited and Photons = Tokens — and two of them carry the rung's task-price numbers ($0.17–$6.55 per task; the open-weight-versus-frontier cost split).
• No measured cost for any frontier model exists anywhere in the KB. Every lab-margin figure stays uncheckable until one does.
• Two sources disagree about what drives the price fall, and the rung cannot settle it. Du attributes ~103.7% of the cost fall to productivity and −0.9% to GPU hardware — a residual, but a pointed one. OpenAI's own account of its July cut (kernels and speculative decoding, no new chips) now sides with Du on one case, as a vendor claim. Matsuoka builds his model on memory and hardware vintage (Memory Scarcity, reconciled to the hardware rung's full close read). Neither has been checked against the other on shared data.
• List prices are not paid prices, anywhere on this rung. Caching discounts, batch tiers and negotiated rates are invisible. The gap is unmeasured.
• The macro channel rests on a stipulated number. The Inference-Cost Phillips Curve's size term (λ̄ = 0.18, i.e. 18% of firms' marginal cost being AI inference over 2022–2026) has no stated source, and its cost series ignores utilisation (Phillips Curve). See the macro channel.
• The energy leg is top-down only. 326 TWh of projected 2028 US AI energy could support ~6.5×10¹⁷ tokens a year — energy is not the near-term limit on token volume — but there is no $/kWh-to-$/token conversion for any real chip (Photons = Tokens, abstract only). See the energy cost of tokens.
• The measurement vocabulary is agreed but unused. Goodput — requests per second meeting a stated service level — is defined (LLM Inference Handbook, a vendor reference with no measurements), but no cost figure on this rung is quoted against it.

Concept Map

Concepts

Concept	Sources on this rung	Evidence	Last Updated
Token Pricing & the Inference-Margin Question	11 (0 peer-reviewed)	Moderate — still undecidable	2026-09-24
The Cost-Measurement Problem	5	Moderate (one full-text measured preprint)	2026-09-24
Cost Per Task (Cost-of-Pass)	5 (4 abstract-read + 1 published price)	Weak-moderate; one published task price	2026-09-24
The Price-Decline Distribution	5	Strong for direction, weak for precision	2026-09-24
Price-Rung Persistence	4	Dated archive — every GPT-5.6 rung now broken	2026-09-24 (correction box)
Inference & Token Economics	3 (+4 on sibling rungs)	Moderate	2026-09-24
The Energy Cost of Tokens	1 (abstract-read)	Weak-moderate, early	2026-09-24
Inference Cost as a Macro Input	1	Weak, early	2026-09-24

Entities

Entity	Type	Sources
OpenAI (pricing)	Company (pricing lens)	3
Anthropic (economics)	Company (economics lens)	2 (+1 culled)
Independent Inference Providers	Market segment	1
vllm-cost-meter	Product (instrument)	1
Intercom (Fin)	Product (task-priced agent)	1

Timeline

See timeline.md for the dated sequence, and frontier.md for what is stale and what would refresh it.

Sources

#	Title	Type	Published	Read depth	Status
1	Ways to Think About Token Pricing	analysis	2026-07-09	full essay	compiled
2	LLM Inference Prices Have Fallen Rapidly but Unequally	technical-report	2025-03-12	full data insight	compiled
3	Token Price Index — GPT-5.6 rung hold	technical-report (own dataset)	2026-07-16	primary observation	compiled
4	Memory Scarcity, Open Models, and the Restructuring of the AI Industry	preprint	2026-07-08	reconciled to hardware full read	compiled
5	Beyond Per-Token Pricing	preprint	2026-06-10	full text	compiled
6	The Economics of AI Inference (Phillips Curve)	preprint	2026-05-19	full text	compiled
7	Tiered Super-Moore's Law	preprint	2026-03-30	full text (2026-09-11)	compiled
8	Cost-of-Pass	preprint	2025-04-17 (rev. 2026-02-26)	abstract	compiled
9	AI Tokenomics	preprint	2026-06-10	abstract	compiled
10	Q2 2026 Provider Pricing Matrix	analysis	2026-04-24	full article	compiled
11	STAGE-Claw	preprint	2026-06-09	abstract + search-surfaced figures	compiled
12	WorkBench Revisited	preprint	2026-06-10	abstract	compiled
13	Photons = Tokens	preprint	2026-02-23	abstract	compiled
14	LLM Inference Handbook	technical-report (vendor)	undated	landing + metrics chapter	compiled 2026-09-24
15	OpenAI GPT-5.6 price cut (Luna −80%, Terra −20%)	technical-report (vendor)	2026-07-30	forum post, changelog, pricing page in full; blog post not readable (403)	compiled 2026-09-24
16	Anthropic Claude Fable 5.1 cache-read cut	technical-report (vendor)	2026-09-01	product page + pricing page in full	compiled 2026-09-24
17	Intercom Fin, $0.99 per outcome	technical-report (vendor)	~2026-09-23 (page update)	help article + pricing FAQ in full; proposed article 401	compiled 2026-09-24

By type: 9 preprints · 6 technical reports (4 of them vendor disclosures) · 2 analyses · 0 peer-reviewed papers.

Read next

• beyond-per-token-pricing-concurrency-aware-cost-methodology — the only fully-read, measured cost study on the rung, and the reason every list-price cost claim here is soft.
• cost-of-pass-economic-framework-language-models — the framework that turns a token price into a task price.
• tiered-super-moores-law-inference-price-evolution — the price-decline paper, now read in full, with the caveats that come with it.