Inference & Token Economics

Where this page's evidence lives (added 2026-09-24). This page was built on the old ai topic and carried over. Only one of its sources is registered on this rung — Tiered Super-Moore's Law, which this rung holds as its own full-text close read and which the links below now point to. Everything else on the page — the GPT-5.6, Kimi K3 and Anthropic pricing rows and the inference-compute evaluation finding — comes from sources registered on the models and evals rungs. Those links now point at the rungs that hold them. Read those claims as sibling-rung context, not as this rung's own evidence. The tier half-lives below were also written from the abstract; the full-text caveats (weak fits, price-band tiers, a 62-milestone trend) are on the price-decline distribution.

The price of a million tokens is the price of intelligence, and between 2020 and 2026 it fell roughly 600-fold. The first systematic empirical study of that collapse (Du, arXiv:2603.28576) assembles pricing across 3,237 models from Epoch AI, 318 from OpenRouter, and 62 cross-validated milestone observations, and finds that the decline is not uniform across the stack — which is the whole point. Cheap tiers fall faster than Moore's Law; the flagship and reasoning tiers resist. The paper calls this the "Tiered Super-Moore" hypothesis.

The single most consequential finding for MenFem's thesis is the cost decomposition: approximately 103.7% of the price decline is attributable to software and architectural innovation, and approximately −0.9% to GPU hardware. Read plainly, hardware improvement did essentially none of the work; algorithmic and serving-stack efficiency did all of it (and slightly more than all of it, with hardware acting as a mild drag or noise term). The mechanisms behind that number are visible in this KB directly — see Speculative Decoding and Low-Bit Quantization, where the measured gains come from a verification scheduler, a loss-support truncation, and a closed-form Hessian transform, all running on unchanged silicon.

The study also locates a structural break around May 2024 (F = 5.74, p = 0.005) separating a technology-driven price-decline regime from a competition-driven one. That distinction is now observable in real time: July 2026 produced three simultaneous, divergent pricing moves from three labs, which a single "the market price of intelligence" number would have hidden entirely.

The July 2026 pricing divergence

Lab	Move	Direction	Reading
OpenAI (Jul 9 — settled 2026-09-24 by OpenAI's own changelog; the Token Price Index recorded the rows on Jul 16)	GPT-5.6 ships as three named tiers — Sol $5/$30, Terra $2.50/$15, Luna $1/$6 per Mtok — replacing a single-flagship cadence	Explicit tier segmentation	Productizes exactly the tier-differentiated structure the Tiered Super-Moore paper found empirically
Moonshot AI (Jul 16)	Kimi K3 launches at $3/$15 per Mtok, up from K2.6's $0.95/$4	Price increase vs. own prior generation	An open-weight Chinese lab competing on capability at Sonnet-tier price, not on cost — a reversal of the DeepSeek V4 posture
Anthropic (Jun 30, revised Aug 10)	Claude Sonnet 5 at $2/$10 per Mtok — launched as an introductory rate with $3/$15 standard to follow, cancelled 2026-08-10 so $2/$10 is now the standard price — with a new tokenizer producing ~30% more tokens for the same text	Quoted price down a third against the prior generation; effective price per unit of text down only ~13%	The tokenizer eats most of the cut — it is the story, not the price sheet

What happened next (added 2026-09-24, from sources on this rung). The July table did not last. OpenAI cut Luna 80% and Terra 20% on Jul 30, naming software savings as the cause, and put Sol on a promotional $4/$20 from Aug 21 (OpenAI). Anthropic's next move, on Sep 1, left Fable 5.1's $10/$50 headline alone and cut only the cache-read price, 75% (Anthropic).

Pricing rows are vendor-published figures. OpenAI's launch prices were journalism-mediated at the time (news, primary page returned 403) and have since been confirmed against OpenAI's own changelog and pricing page (2026-09-24); Moonshot's are vendor-press-release-via-journalism plus third-party aggregators; Anthropic's are first-party. The Anthropic row was re-read on 2026-08-13 against the pricing page and the 2026-08-10 release-notes entry, which is where the cancellation is stated.

Key Claims

• Token prices fell ~600× from 2020 to 2026, and the decline is tiered, not uniform — Economy-tier price half-life 1.10 years and mid-tier 1.55 years, both faster than the 2-year Moore's Law benchmark; flagship/reasoning tiers resist the trend, with reasoning commanding roughly a 31.5× premium over non-reasoning pricing. Evidence: moderate (full text read on this rung 2026-09-11: the direction is robust, but the fits are weak — economy R² 0.192 — the trend rests on 62 hand-compiled milestones, and the tiers are price bands models move between) (Tiered Super-Moore's Law)
• ~103.7% of the cost decline is software/architecture; ~−0.9% is GPU hardware — The single most thesis-relevant number in this KB. Hardware improvement did not meaningfully drive the price of inference down over this period; the serving stack and model architecture did. Evidence: moderate (same paper, full text: the 103.7% is a growth-accounting residual — the unexplained part — which exceeds 100% because the hardware term came out slightly negative) (Tiered Super-Moore's Law)
• A structural break in May 2024 separates a technology-driven pricing regime from a competition-driven one — F = 5.74, p = 0.005. Before: pricing behaves like a technology curve. After: like a competitive market with entrants, undercutting, and tiered product lines. Evidence: moderate (formal test, F = 5.74, p = 0.005, but only 16 observations before the break) (Tiered Super-Moore's Law)
• A tokenizer change can eat most of a headline price cut — Anthropic's Claude Sonnet 5 (with the Fable 5 / Mythos 5 generation) ships a new tokenizer producing ~30% more tokens for the same text, while the prior-generation Claude Sonnet 4.6 stays on the old tokenizer at $3/$15 per Mtok. So Sonnet 5's $2/$10 reads as a 33% cut per token but only about a 13% cut per unit of text ($2 × 1.30 ≈ $2.60 against $3.00). This makes per-token prices non-comparable across tokenizer generations and across vendors — the load-bearing point, and the one that survives any repricing. Note: this claim previously ran the arithmetic against Sonnet 5's own scheduled $3/$15 standard rate and concluded the effective price would rise ~30%. Anthropic cancelled that step-up on 2026-08-10, so the comparison has to be made against the prior generation instead — and the direction flips from up to modestly down. The mechanism is unchanged; only the counterfactual moved. Evidence: strong for the tokenizer shift and for both list prices (first-party changelog and pricing page, re-read 2026-08-13); the per-unit-of-text arithmetic is this KB's inference, not Anthropic's claim (Anthropic Platform Release Notes, Anthropic pricing page)
• Vendors are now competing on tokens-consumed, not just price-per-token — OpenAI claims GPT-5.6 is 54% more token-efficient on agentic coding than prior/rival models, with GPT-5.6 Sol reportedly matching Anthropic's "Mythos Preview" on ExploitBench while emitting about one-third as many output tokens; Terra is pitched as GPT-5.5-equivalent for "half the cost and 16% fewer tokens." Independently, Artificial Analysis reports Kimi K3 using ~21% fewer output tokens than K2.6 at ~$0.94 cost per task. Evidence: weak for the OpenAI figures (vendor claim relayed by news; primary page 403; underlying eval methodology not reproduced in the sources held here). Weak-to-moderate for the Kimi figure (third-party aggregator, but pre-open-weights so unreproducible). (GPT-5.6, Kimi K3)
• The price fall is now landing on the cache line, not the headline rate — Claude Fable 5.1 (2026-09-01) kept input/output at $10/$50 per Mtok and cut cache reads 75%, $1.00 → $0.25 — from a tenth of the input price to a fortieth. Anthropic estimates ~25% off typical workloads and up to ~45% off highly agentic ones; on an illustrative agent step that is 90% re-read context the bill falls ~23% with no change to the quoted rate. A cache read is the price of re-using stored model state (the KV cache) instead of recomputing it, so this is the mechanism the KV-cache thesis is about, priced. Evidence: strong for the prices (vendor product page + pricing page, read 2026-09-24); weak for the savings (Anthropic's own estimates, no workload mix); no engineering cause given (Anthropic)
• Caching is now a first-class pricing axis — GPT-5.6 offers a 90% discount on cached reads and charges 1.25× on cache writes; Kimi K3 prices cache-hit input at $0.30 vs. $3.00 on cache miss (a 10× spread). Headline per-Mtok rates increasingly describe a worst case rather than a realized cost. Evidence: moderate (two independent vendors, consistent structure; individual figures are vendor-reported) (GPT-5.6, Kimi K3)
• Reasoning effort is becoming a non-configurable, vendor-set cost — Kimi K3 launched with a single reasoning-effort setting ("max"), driving token consumption on trivial prompts (an observed 13,241 reasoning tokens, ~$0.25, on a hello-world-grade request). In the same window Anthropic made adaptive thinking on by default and removed manual extended-thinking budgets on Sonnet 5 (a 400 error if attempted). Two labs, opposite mechanisms, same effect: the buyer loses direct control of how much deliberation they pay for. Evidence: moderate (Anthropic side first-party and unambiguous; Kimi side single-reviewer observation) (Anthropic Platform Release Notes, Kimi K3)
• Capability is a function of inference spend, not a fixed score — which collapses "how good" and "how much" into one curve — Across up to 12 frontier models × 7 benchmarks (SWE, math, medicine, cybersecurity) under controlled inference-scaling interventions (larger token budgets, context compaction, repeated attempts), fixed single-budget evaluations increasingly understate models — and the underestimation grows for newer models, which have a higher ceiling that only opens at generous budgets. The economic reading is direct: if a model's measured capability rises with the compute you pour into it at inference time, then "how good is model X" and "how much does X cost to run" are the same question asked at two points on one curve. That is the strongest theoretical support in this KB for pricing shifting from per-token to per-task — you are buying an outcome at a budget, not a fixed-quality token. Evidence: moderate-to-strong for the qualitative claim (preprint, third-party — does not evaluate the authors' own model; methodologically careful); per-benchmark numeric curves not captured at ingest (How Inference Compute Shapes Frontier LLM Evaluation)

Benchmarks & Data

Metric	Value	Source
Token price decline, 2020–2026	~600×	Tiered Super-Moore
Economy-tier price half-life	1.10 years (~1.8× faster than Moore)	Tiered Super-Moore
Mid-tier price half-life	1.55 years (~1.3× faster than Moore)	Tiered Super-Moore
Reasoning premium over non-reasoning	~31.5× average; 0.51–83.3× by quarter	Tiered Super-Moore
Cost-decline attribution: software/architecture	~103.7%	Tiered Super-Moore
Cost-decline attribution: GPU hardware	~−0.9%	Tiered Super-Moore
Structural break	May 2024 (F = 5.74, p = 0.005)	Tiered Super-Moore
Dataset size	3,237 models (Epoch AI) + 318 (OpenRouter) + 62 milestones	Tiered Super-Moore
GPT-5.6 Sol / Terra / Luna ($ per Mtok in/out)	$5/$30 · $2.50/$15 · $1/$6	GPT-5.6
Kimi K3 ($ per Mtok in/out; cache hit)	$3/$15; $0.30 cache-hit input	Kimi K3
Kimi K2.6 → K3 price change	$0.95/$4 → $3/$15 (increase)	Kimi K3
Claude Sonnet 5 ($ per Mtok in/out)	$2/$10 standard — the introductory rate was made permanent 2026-08-10 and the scheduled 1 Sep step-up to $3/$15 cancelled	Anthropic pricing page + release notes, 2026-08-10 (read 2026-08-13)
Claude Sonnet 5 vs prior-gen Sonnet 4.6, tokenizer-adjusted	$2/$10 vs $3/$15 per token; ~$2.60/$13.00 vs $3/$15 per unit of text (Sonnet 4.6 is on the old tokenizer)	Anthropic pricing page (read 2026-08-13); adjustment is this KB's arithmetic
Claude Sonnet 5 tokenizer shift	~+30% tokens for the same text	Anthropic Release Notes

Conflicts & Cautions

• "Token efficiency" is not a measurable cross-vendor quantity right now. Anthropic's tokenizer emits ~30% more tokens for identical text than its own prior generation. Any claim of the form "model A uses N% fewer tokens than model B" is partly a statement about tokenizers, not about reasoning economy — unless normalized to characters or words, which none of the sources here do. OpenAI's 54% claim and the ~one-third-output-tokens comparison against Anthropic's "Mythos Preview" are both exposed to this.
• Vendor efficiency claims are exactly the class of claim Agent Harness Evolution says to discount — self-reported, on self-selected benchmarks, without matched-budget controls or held-out generalization. Treat 54% as marketing until independently reproduced.
• Kimi K3's own benchmark story is internally inconsistent. Moonshot's own evaluation places K3 behind Claude Fable 5 and GPT-5.6 Sol overall, while Arena.ai's Frontend Code leaderboard places it #1, ahead of Claude Fable 5. Both are reported in the same source. This is a genuine conflict, not a resolvable one — task-specific leaderboards and general eval suites are measuring different things, and the vendor has an incentive to cite whichever favours it.
• Headline prices increasingly overstate realized cost (caching discounts up to 90%, 10× cache-hit spreads) while hidden overhead understates it (an observed ~85-token implicit system prompt on Kimi K3 meant a 10-word prompt billed as 95 tokens). Neither direction is captured by a price sheet.

Open Questions

• Do the serving-layer efficiency gains now being published (60–85% production speedups, 6.978× compounded) reach published API prices, or are they retained as provider margin? The Tiered Super-Moore data covers a period ending before these disclosures.
• Does the May-2024 competition-driven regime persist, or does Kimi K3's price increase mark the start of capability-led repricing at the open-weight frontier?
• The ~31.5× reasoning premium is a cross-sectional average. Does it hold per-lab and per-generation, and does it survive the shift to non-configurable adaptive reasoning?
• If reasoning effort becomes vendor-set rather than buyer-set, does per-token pricing remain a meaningful contract at all, or does the market move to per-task pricing? (Artificial Analysis already reports Kimi K3 at ~$0.94 per task — an early sign of the latter unit.) The inference-compute-eval finding sharpens this: if capability is a function of inference budget, the natural contract is "solve this task within budget B," not "pay per token" — but nobody has published the capability-vs-cost curve that would let a buyer price it.
• The inference-compute-eval study captures the shape (newer models understated at fixed budget) but not the numbers — the per-model, per-benchmark capability-vs-budget curves are the load-bearing evidence and were not captured at ingest. Without them, "buy an outcome at budget B" stays a thesis, not a price.
• What is the effective, tokenizer-normalized price series? Nothing in this KB currently computes it.

Related Concepts

• The Price-Decline Distribution — this rung's full treatment of the tier half-lives, with the full-text caveats
• Price-Rung Persistence — the GPT-5.6 and Anthropic rung observations, dated archive
• Cost Per Task — where Kimi K3's ~$0.94 per task and the per-task contract question land Sibling rungs (context):
• Speculative Decoding — a measured mechanism behind the software-driven cost decline
• Low-Bit Quantization — the second measured mechanism
• Agent Harness Evolution — the methodological standard vendor efficiency claims fail to meet
• Agent Evaluation Benchmarks — where the inference-compute-as-evaluation-axis finding also lands; matched-budget scoring is the shared discipline
• KV-Cache Compression — a third measured software mechanism behind the cost decline
• Chain-of-Thought Reasoning — token-efficiency pressure is now an economic force acting on how much reasoning is emitted

Related Entities

• OpenAI — GPT-5.6 three-tier launch
• Anthropic — Sonnet 5 pricing and tokenizer shift
• Moonshot AI — Kimi K3 price increase at the open-weight frontier
• DeepSeek — the prior open-weight cost-disruption posture K3 departs from

Backlinks

Pages that reference this concept:

• Price-Rung Persistence
• Token Pricing & the Inference-Margin Question
• Cost Per Task
• Speculative Decoding, Low-Bit Quantization (serving)
• Agent Harness Evolution (harnesses)
• Moonshot AI (models)

Changelog

• 2026-09-24 (source ingest) — Compiled two sources ingested on this rung: OpenAI's GPT-5.6 price-cut announcement (settles the launch date as Jul 9 and confirms the July cuts on primary pages) and Anthropic's Fable 5.1 cache-read cut (new key claim: the decline landing on the cache line item). Added a short "what happened next" under the July table.
• 2026-09-24 (recompile, no new source) — Every relative link on this page was broken: it was carried over from the old ai topic and pointed at raw files and concept pages that live on other rungs. Re-pointed the Tiered Super-Moore's citations to this rung's own full-text close read and the rest to models, evals, serving and harnesses; kept them in sources, where the site resolves each slug to the rung that owns it. Downgraded the Tiered claims from "strong" to "moderate" in line with the 2026-09-11 full-text read. Flagged the GPT-5.6 launch-date conflict (Jul 9 in the models news source vs Jul 16 in the Token Price Index). Linked the page into the rung's index.
• 2026-08-13 — CORRECTION. This page was publishing Claude Sonnet 5's scheduled 1 September step-up to $3/$15 per Mtok as fact in three places (the July-divergence table, the tokenizer key claim, and the data table). Anthropic cancelled that step-up on 2026-08-10 and made the $2/$10 introductory rate the standard price; verified against the pricing page and the 2026-08-10 release-notes entry on 2026-08-13. The tokenizer claim needed rewriting rather than a number swap: its arithmetic ran against the now-dead $3/$15 rate and concluded effective price per unit of text would rise ~30%. Re-anchored on the prior generation (Sonnet 4.6, $3/$15, old tokenizer), the direction flips — a 33% per-token cut is only ~13% per unit of text. The mechanism the claim exists to make (per-token prices are non-comparable across tokenizer generations) is unchanged and is now stated as the load-bearing part. Added a tokenizer-adjusted comparison row. No new source registered and last_compiled deliberately not bumped — this is a correction, not an ingest.
• 2026-07-22 — Created from four sources spanning the full authority range (Tiered Super-Moore paper, Anthropic first-party release notes, Kimi K3 analysis, GPT-5.6 news). Records the ~600× decline, tiered half-lives, the ~103.7%-software / ~−0.9%-hardware decomposition, the May-2024 structural break, and the July-2026 three-lab pricing divergence. Flags the tokenizer-comparability problem and the Kimi K3 internal benchmark conflict.
• 2026-07-23 — Compiled How Inference Compute Shapes Frontier LLM Evaluation (arXiv:2606.17930). Added a key claim that capability is a function of inference spend (fixed-budget scoring understates newer models, 12 models × 7 benchmarks) — the strongest theoretical support in this KB for per-token → per-task repricing — plus two open questions (the missing capability-vs-cost curve) and cross-links to Agent Evaluation Benchmarks and KV-Cache. Evidence graded moderate-to-strong for the qualitative claim; per-benchmark numeric grid owed.