Inference & Token Economics

The price of a million tokens is the price of intelligence, and between 2020 and 2026 it fell roughly 600-fold. The first systematic empirical study of that collapse (Du, arXiv:2603.28576) assembles pricing across 3,237 models from Epoch AI, 318 from OpenRouter, and 62 cross-validated milestone observations, and finds that the decline is not uniform across the stack — which is the whole point. Cheap tiers fall faster than Moore's Law; the flagship and reasoning tiers resist. The paper calls this the "Tiered Super-Moore" hypothesis.

The single most consequential finding for MenFem's thesis is the cost decomposition: approximately 103.7% of the price decline is attributable to software and architectural innovation, and approximately −0.9% to GPU hardware. Read plainly, hardware improvement did essentially none of the work; algorithmic and serving-stack efficiency did all of it (and slightly more than all of it, with hardware acting as a mild drag or noise term). The mechanisms behind that number are visible in this KB directly — see Speculative Decoding and Low-Bit Quantization, where the measured gains come from a verification scheduler, a loss-support truncation, and a closed-form Hessian transform, all running on unchanged silicon.

The study also locates a structural break around May 2024 (F = 5.74, p = 0.005) separating a technology-driven price-decline regime from a competition-driven one. That distinction is now observable in real time: July 2026 produced three simultaneous, divergent pricing moves from three labs, which a single "the market price of intelligence" number would have hidden entirely.

The July 2026 pricing divergence

Pricing rows are vendor-published figures. OpenAI's are journalism-mediated (news, primary page returned 403); Moonshot's are vendor-press-release-via-journalism plus third-party aggregators; Anthropic's are first-party.

Key Claims

• Token prices fell ~600× from 2020 to 2026, and the decline is tiered, not uniform — Economy-tier price half-life 1.10 years and mid-tier 1.55 years, both faster than the 2-year Moore's Law benchmark; flagship/reasoning tiers resist the trend, with reasoning commanding roughly a 31.5× premium over non-reasoning pricing. Evidence: strong (arXiv paper, largest known cross-validated pricing dataset, DEA + Chow structural-break methodology — but this KB holds abstract + reported findings only, not the full PDF) (Tiered Super-Moore's Law)
• ~103.7% of the cost decline is software/architecture; ~−0.9% is GPU hardware — The single most thesis-relevant number in this KB. Hardware improvement did not meaningfully drive the price of inference down over this period; the serving stack and model architecture did. Evidence: strong (same paper; decomposition methodology stated, underlying regression not inspected at this KB's fetch depth) (Tiered Super-Moore's Law)
• A structural break in May 2024 separates a technology-driven pricing regime from a competition-driven one — F = 5.74, p = 0.005. Before: pricing behaves like a technology curve. After: like a competitive market with entrants, undercutting, and tiered product lines. Evidence: strong (formal statistical test with reported statistic and p-value) (Tiered Super-Moore's Law)
• A tokenizer change can raise effective per-word price while the quoted per-token price falls — Anthropic's Claude Sonnet 5 (and the Fable 5 / Mythos 5 generation before it) ship a new tokenizer producing ~30% more tokens for the same text. Introductory pricing of $2/$10 per Mtok is nominally below the $3/$15 standard rate, but ~30% more tokens are billed for identical input. This makes per-token prices non-comparable across tokenizer generations and across vendors. Evidence: strong for the fact that it shipped (first-party changelog); the pricing implication is this KB's inference, not Anthropic's claim (Anthropic Platform Release Notes)
• Vendors are now competing on tokens-consumed, not just price-per-token — OpenAI claims GPT-5.6 is 54% more token-efficient on agentic coding than prior/rival models, with GPT-5.6 Sol reportedly matching Anthropic's "Mythos Preview" on ExploitBench while emitting about one-third as many output tokens; Terra is pitched as GPT-5.5-equivalent for "half the cost and 16% fewer tokens." Independently, Artificial Analysis reports Kimi K3 using ~21% fewer output tokens than K2.6 at ~$0.94 cost per task. Evidence: weak for the OpenAI figures (vendor claim relayed by news; primary page 403; underlying eval methodology not reproduced in the sources held here). Weak-to-moderate for the Kimi figure (third-party aggregator, but pre-open-weights so unreproducible). (GPT-5.6, Kimi K3)
• Caching is now a first-class pricing axis — GPT-5.6 offers a 90% discount on cached reads and charges 1.25× on cache writes; Kimi K3 prices cache-hit input at $0.30 vs. $3.00 on cache miss (a 10× spread). Headline per-Mtok rates increasingly describe a worst case rather than a realized cost. Evidence: moderate (two independent vendors, consistent structure; individual figures are vendor-reported) (GPT-5.6, Kimi K3)
• Reasoning effort is becoming a non-configurable, vendor-set cost — Kimi K3 launched with a single reasoning-effort setting ("max"), driving token consumption on trivial prompts (an observed 13,241 reasoning tokens, ~$0.25, on a hello-world-grade request). In the same window Anthropic made adaptive thinking on by default and removed manual extended-thinking budgets on Sonnet 5 (a 400 error if attempted). Two labs, opposite mechanisms, same effect: the buyer loses direct control of how much deliberation they pay for. Evidence: moderate (Anthropic side first-party and unambiguous; Kimi side single-reviewer observation) (Anthropic Platform Release Notes, Kimi K3)
• Capability is a function of inference spend, not a fixed score — which collapses "how good" and "how much" into one curve — Across up to 12 frontier models × 7 benchmarks (SWE, math, medicine, cybersecurity) under controlled inference-scaling interventions (larger token budgets, context compaction, repeated attempts), fixed single-budget evaluations increasingly understate models — and the underestimation grows for newer models, which have a higher ceiling that only opens at generous budgets. The economic reading is direct: if a model's measured capability rises with the compute you pour into it at inference time, then "how good is model X" and "how much does X cost to run" are the same question asked at two points on one curve. That is the strongest theoretical support in this KB for pricing shifting from per-token to per-task — you are buying an outcome at a budget, not a fixed-quality token. Evidence: moderate-to-strong for the qualitative claim (preprint, third-party — does not evaluate the authors' own model; methodologically careful); per-benchmark numeric curves not captured at ingest (How Inference Compute Shapes Frontier LLM Evaluation)

Benchmarks & Data

Conflicts & Cautions

• "Token efficiency" is not a measurable cross-vendor quantity right now. Anthropic's tokenizer emits ~30% more tokens for identical text than its own prior generation. Any claim of the form "model A uses N% fewer tokens than model B" is partly a statement about tokenizers, not about reasoning economy — unless normalized to characters or words, which none of the sources here do. OpenAI's 54% claim and the ~one-third-output-tokens comparison against Anthropic's "Mythos Preview" are both exposed to this.
• Vendor efficiency claims are exactly the class of claim Agent Harness Evolution says to discount — self-reported, on self-selected benchmarks, without matched-budget controls or held-out generalization. Treat 54% as marketing until independently reproduced.
• Kimi K3's own benchmark story is internally inconsistent. Moonshot's own evaluation places K3 behind Claude Fable 5 and GPT-5.6 Sol overall, while Arena.ai's Frontend Code leaderboard places it #1, ahead of Claude Fable 5. Both are reported in the same source. This is a genuine conflict, not a resolvable one — task-specific leaderboards and general eval suites are measuring different things, and the vendor has an incentive to cite whichever favours it.
• Headline prices increasingly overstate realized cost (caching discounts up to 90%, 10× cache-hit spreads) while hidden overhead understates it (an observed ~85-token implicit system prompt on Kimi K3 meant a 10-word prompt billed as 95 tokens). Neither direction is captured by a price sheet.

Open Questions

• Do the serving-layer efficiency gains now being published (60–85% production speedups, 6.978× compounded) reach published API prices, or are they retained as provider margin? The Tiered Super-Moore data covers a period ending before these disclosures.
• Does the May-2024 competition-driven regime persist, or does Kimi K3's price increase mark the start of capability-led repricing at the open-weight frontier?
• The ~31.5× reasoning premium is a cross-sectional average. Does it hold per-lab and per-generation, and does it survive the shift to non-configurable adaptive reasoning?
• If reasoning effort becomes vendor-set rather than buyer-set, does per-token pricing remain a meaningful contract at all, or does the market move to per-task pricing? (Artificial Analysis already reports Kimi K3 at ~$0.94 per task — an early sign of the latter unit.) The inference-compute-eval finding sharpens this: if capability is a function of inference budget, the natural contract is "solve this task within budget B," not "pay per token" — but nobody has published the capability-vs-cost curve that would let a buyer price it.
• The inference-compute-eval study captures the shape (newer models understated at fixed budget) but not the numbers — the per-model, per-benchmark capability-vs-budget curves are the load-bearing evidence and were not captured at ingest. Without them, "buy an outcome at budget B" stays a thesis, not a price.
• What is the effective, tokenizer-normalized price series? Nothing in this KB currently computes it.

Related Concepts

• Speculative Decoding — a measured mechanism behind the software-driven cost decline
• Low-Bit Quantization — the second measured mechanism
• Agent Harness Evolution — the methodological standard vendor efficiency claims fail to meet
• Agent Evaluation Benchmarks — where the inference-compute-as-evaluation-axis finding also lands; matched-budget scoring is the shared discipline
• KV-Cache Compression — a third measured software mechanism behind the cost decline
• Chain-of-Thought Reasoning — token-efficiency pressure is now an economic force acting on how much reasoning is emitted

Related Entities

• OpenAI — GPT-5.6 three-tier launch
• Anthropic — Sonnet 5 pricing and tokenizer shift
• Moonshot AI — Kimi K3 price increase at the open-weight frontier
• DeepSeek — the prior open-weight cost-disruption posture K3 departs from

Backlinks

Pages that reference this concept:

• Speculative Decoding
• Low-Bit Quantization
• Agent Harness Evolution
• Moonshot AI

Changelog

• 2026-07-22 — Created from four sources spanning the full authority range (Tiered Super-Moore paper, Anthropic first-party release notes, Kimi K3 analysis, GPT-5.6 news). Records the ~600× decline, tiered half-lives, the ~103.7%-software / ~−0.9%-hardware decomposition, the May-2024 structural break, and the July-2026 three-lab pricing divergence. Flags the tokenizer-comparability problem and the Kimi K3 internal benchmark conflict.
• 2026-07-23 — Compiled How Inference Compute Shapes Frontier LLM Evaluation (arXiv:2606.17930). Added a key claim that capability is a function of inference spend (fixed-budget scoring understates newer models, 12 models × 7 benchmarks) — the strongest theoretical support in this KB for per-token → per-task repricing — plus two open questions (the missing capability-vs-cost curve) and cross-links to Agent Evaluation Benchmarks and KV-Cache. Evidence graded moderate-to-strong for the qualitative claim; per-benchmark numeric grid owed.