# Inference & Token Economics

Canonical URL: https://menfem.com/kb/ai/concepts/inference-token-economics
Knowledge base topic: [Artificial Intelligence](https://menfem.com/kb/ai)
Frontier status: active
Tags: inference-economics, token-pricing, market-structure, model-tiering

---

The price of a million tokens is the price of intelligence, and between 2020 and 2026 it fell roughly **600-fold**. The first systematic empirical study of that collapse (Du, arXiv:2603.28576) assembles pricing across 3,237 models from Epoch AI, 318 from OpenRouter, and 62 cross-validated milestone observations, and finds that the decline is **not uniform across the stack** — which is the whole point. Cheap tiers fall faster than Moore's Law; the flagship and reasoning tiers resist. The paper calls this the **"Tiered Super-Moore"** hypothesis.

The single most consequential finding for MenFem's thesis is the **cost decomposition**: approximately **103.7% of the price decline is attributable to software and architectural innovation, and approximately −0.9% to GPU hardware**. Read plainly, hardware improvement did essentially none of the work; algorithmic and serving-stack efficiency did all of it (and slightly more than all of it, with hardware acting as a mild drag or noise term). The mechanisms behind that number are visible in this KB directly — see [Speculative Decoding](./speculative-decoding.md) and [Low-Bit Quantization](./low-bit-quantization.md), where the measured gains come from a verification scheduler, a loss-support truncation, and a closed-form Hessian transform, all running on unchanged silicon.

The study also locates a **structural break around May 2024** (F = 5.74, p = 0.005) separating a technology-driven price-decline regime from a competition-driven one. That distinction is now observable in real time: July 2026 produced three simultaneous, *divergent* pricing moves from three labs, which a single "the market price of intelligence" number would have hidden entirely.

## The July 2026 pricing divergence

| Lab | Move | Direction | Reading |
|---|---|---|---|
| **OpenAI** (Jul 9) | GPT-5.6 ships as three named tiers — Sol $5/$30, Terra $2.50/$15, Luna $1/$6 per Mtok — replacing a single-flagship cadence | Explicit tier segmentation | Productizes exactly the tier-differentiated structure the Tiered Super-Moore paper found empirically |
| **Moonshot AI** (Jul 16) | Kimi K3 launches at $3/$15 per Mtok, up from K2.6's $0.95/$4 | **Price increase** vs. own prior generation | An open-weight Chinese lab competing on *capability at Sonnet-tier price*, not on cost — a reversal of the DeepSeek V4 posture |
| **Anthropic** (Jun 30) | Claude Sonnet 5 at $2/$10 introductory through 2026-08-31, $3/$15 standard after — **with a new tokenizer producing ~30% more tokens for the same text** | Quoted price flat/down, effective price up | The tokenizer is the story, not the price sheet |

*Pricing rows are vendor-published figures. OpenAI's are journalism-mediated (`news`, primary page returned 403); Moonshot's are vendor-press-release-via-journalism plus third-party aggregators; Anthropic's are first-party.*

## Key Claims

- **Token prices fell ~600× from 2020 to 2026, and the decline is tiered, not uniform** — Economy-tier price half-life **1.10 years** and mid-tier **1.55 years**, both faster than the 2-year Moore's Law benchmark; flagship/reasoning tiers resist the trend, with reasoning commanding roughly a **31.5× premium** over non-reasoning pricing. *Evidence: strong (arXiv paper, largest known cross-validated pricing dataset, DEA + Chow structural-break methodology — but this KB holds abstract + reported findings only, not the full PDF)* ([Tiered Super-Moore's Law](../../raw/tiered-super-moores-law-token-pricing.md))
- **~103.7% of the cost decline is software/architecture; ~−0.9% is GPU hardware** — The single most thesis-relevant number in this KB. Hardware improvement did not meaningfully drive the price of inference down over this period; the serving stack and model architecture did. *Evidence: strong (same paper; decomposition methodology stated, underlying regression not inspected at this KB's fetch depth)* ([Tiered Super-Moore's Law](../../raw/tiered-super-moores-law-token-pricing.md))
- **A structural break in May 2024 separates a technology-driven pricing regime from a competition-driven one** — F = 5.74, p = 0.005. Before: pricing behaves like a technology curve. After: like a competitive market with entrants, undercutting, and tiered product lines. *Evidence: strong (formal statistical test with reported statistic and p-value)* ([Tiered Super-Moore's Law](../../raw/tiered-super-moores-law-token-pricing.md))
- **A tokenizer change can raise effective per-word price while the quoted per-token price falls** — Anthropic's Claude Sonnet 5 (and the Fable 5 / Mythos 5 generation before it) ship a **new tokenizer producing ~30% more tokens for the same text**. Introductory pricing of $2/$10 per Mtok is nominally below the $3/$15 standard rate, but ~30% more tokens are billed for identical input. This makes per-token prices **non-comparable across tokenizer generations and across vendors**. *Evidence: strong for the fact that it shipped (first-party changelog); the pricing implication is this KB's inference, not Anthropic's claim* ([Anthropic Platform Release Notes](../../raw/anthropic-claude-platform-updates-2026-07.md))
- **Vendors are now competing on tokens-consumed, not just price-per-token** — OpenAI claims GPT-5.6 is **54% more token-efficient on agentic coding** than prior/rival models, with GPT-5.6 Sol reportedly matching Anthropic's "Mythos Preview" on ExploitBench while emitting about **one-third** as many output tokens; Terra is pitched as GPT-5.5-equivalent for "half the cost and 16% fewer tokens." Independently, Artificial Analysis reports Kimi K3 using ~21% fewer output tokens than K2.6 at ~$0.94 cost per task. *Evidence: weak for the OpenAI figures (vendor claim relayed by `news`; primary page 403; underlying eval methodology not reproduced in the sources held here). Weak-to-moderate for the Kimi figure (third-party aggregator, but pre-open-weights so unreproducible).* ([GPT-5.6](../../raw/gpt-5-6-tiering-token-efficiency.md), [Kimi K3](../../raw/kimi-k3-moonshot-open-weight-2026.md))
- **Caching is now a first-class pricing axis** — GPT-5.6 offers a **90% discount on cached reads** and charges **1.25× on cache writes**; Kimi K3 prices cache-hit input at **$0.30** vs. $3.00 on cache miss (a 10× spread). Headline per-Mtok rates increasingly describe a worst case rather than a realized cost. *Evidence: moderate (two independent vendors, consistent structure; individual figures are vendor-reported)* ([GPT-5.6](../../raw/gpt-5-6-tiering-token-efficiency.md), [Kimi K3](../../raw/kimi-k3-moonshot-open-weight-2026.md))
- **Reasoning effort is becoming a non-configurable, vendor-set cost** — Kimi K3 launched with a single reasoning-effort setting ("max"), driving token consumption on trivial prompts (an observed 13,241 reasoning tokens, ~$0.25, on a hello-world-grade request). In the same window Anthropic made **adaptive thinking on by default and removed manual extended-thinking budgets** on Sonnet 5 (a 400 error if attempted). Two labs, opposite mechanisms, same effect: the buyer loses direct control of how much deliberation they pay for. *Evidence: moderate (Anthropic side first-party and unambiguous; Kimi side single-reviewer observation)* ([Anthropic Platform Release Notes](../../raw/anthropic-claude-platform-updates-2026-07.md), [Kimi K3](../../raw/kimi-k3-moonshot-open-weight-2026.md))
- **Capability is a function of inference spend, not a fixed score — which collapses "how good" and "how much" into one curve** — Across up to 12 frontier models × 7 benchmarks (SWE, math, medicine, cybersecurity) under controlled inference-scaling interventions (larger token budgets, context compaction, repeated attempts), fixed single-budget evaluations increasingly **understate** models — and the underestimation *grows for newer models*, which have a higher ceiling that only opens at generous budgets. The economic reading is direct: if a model's measured capability rises with the compute you pour into it at inference time, then "how good is model X" and "how much does X cost to run" are the same question asked at two points on one curve. That is the strongest theoretical support in this KB for pricing shifting from **per-token to per-task** — you are buying an outcome at a budget, not a fixed-quality token. *Evidence: moderate-to-strong for the qualitative claim (preprint, third-party — does not evaluate the authors' own model; methodologically careful); per-benchmark numeric curves not captured at ingest* ([How Inference Compute Shapes Frontier LLM Evaluation](../../raw/how-inference-compute-shapes-frontier-llm-evaluation.md))

## Benchmarks & Data

| Metric | Value | Source |
|---|---|---|
| Token price decline, 2020–2026 | ~600× | [Tiered Super-Moore](../../raw/tiered-super-moores-law-token-pricing.md) |
| Economy-tier price half-life | 1.10 years (~1.8× faster than Moore) | [Tiered Super-Moore](../../raw/tiered-super-moores-law-token-pricing.md) |
| Mid-tier price half-life | 1.55 years (~1.3× faster than Moore) | [Tiered Super-Moore](../../raw/tiered-super-moores-law-token-pricing.md) |
| Reasoning premium over non-reasoning | ~31.5× | [Tiered Super-Moore](../../raw/tiered-super-moores-law-token-pricing.md) |
| Cost-decline attribution: software/architecture | ~103.7% | [Tiered Super-Moore](../../raw/tiered-super-moores-law-token-pricing.md) |
| Cost-decline attribution: GPU hardware | ~−0.9% | [Tiered Super-Moore](../../raw/tiered-super-moores-law-token-pricing.md) |
| Structural break | May 2024 (F = 5.74, p = 0.005) | [Tiered Super-Moore](../../raw/tiered-super-moores-law-token-pricing.md) |
| Dataset size | 3,237 models (Epoch AI) + 318 (OpenRouter) + 62 milestones | [Tiered Super-Moore](../../raw/tiered-super-moores-law-token-pricing.md) |
| GPT-5.6 Sol / Terra / Luna ($ per Mtok in/out) | $5/$30 · $2.50/$15 · $1/$6 | [GPT-5.6](../../raw/gpt-5-6-tiering-token-efficiency.md) |
| Kimi K3 ($ per Mtok in/out; cache hit) | $3/$15; $0.30 cache-hit input | [Kimi K3](../../raw/kimi-k3-moonshot-open-weight-2026.md) |
| Kimi K2.6 → K3 price change | $0.95/$4 → $3/$15 (increase) | [Kimi K3](../../raw/kimi-k3-moonshot-open-weight-2026.md) |
| Claude Sonnet 5 ($ per Mtok in/out) | $2/$10 intro through 2026-08-31; $3/$15 standard | [Anthropic Release Notes](../../raw/anthropic-claude-platform-updates-2026-07.md) |
| Claude Sonnet 5 tokenizer shift | ~+30% tokens for the same text | [Anthropic Release Notes](../../raw/anthropic-claude-platform-updates-2026-07.md) |

## Conflicts & Cautions

- **"Token efficiency" is not a measurable cross-vendor quantity right now.** Anthropic's tokenizer emits ~30% more tokens for identical text than its own prior generation. Any claim of the form "model A uses N% fewer tokens than model B" is partly a statement about tokenizers, not about reasoning economy — unless normalized to characters or words, which none of the sources here do. OpenAI's 54% claim and the ~one-third-output-tokens comparison against Anthropic's "Mythos Preview" are both exposed to this.
- **Vendor efficiency claims are exactly the class of claim [Agent Harness Evolution](./agent-harness-evolution.md) says to discount** — self-reported, on self-selected benchmarks, without matched-budget controls or held-out generalization. Treat 54% as marketing until independently reproduced.
- **Kimi K3's own benchmark story is internally inconsistent.** Moonshot's own evaluation places K3 *behind* Claude Fable 5 and GPT-5.6 Sol overall, while Arena.ai's Frontend Code leaderboard places it **#1, ahead of Claude Fable 5**. Both are reported in the same source. This is a genuine conflict, not a resolvable one — task-specific leaderboards and general eval suites are measuring different things, and the vendor has an incentive to cite whichever favours it.
- **Headline prices increasingly overstate realized cost** (caching discounts up to 90%, 10× cache-hit spreads) while **hidden overhead understates it** (an observed ~85-token implicit system prompt on Kimi K3 meant a 10-word prompt billed as 95 tokens). Neither direction is captured by a price sheet.

## Open Questions

- Do the serving-layer efficiency gains now being published (60–85% production speedups, 6.978× compounded) reach published API prices, or are they retained as provider margin? The Tiered Super-Moore data covers a period ending before these disclosures.
- Does the May-2024 competition-driven regime persist, or does Kimi K3's *price increase* mark the start of capability-led repricing at the open-weight frontier?
- The ~31.5× reasoning premium is a cross-sectional average. Does it hold per-lab and per-generation, and does it survive the shift to non-configurable adaptive reasoning?
- If reasoning effort becomes vendor-set rather than buyer-set, does per-token pricing remain a meaningful contract at all, or does the market move to per-task pricing? (Artificial Analysis already reports Kimi K3 at ~$0.94 **per task** — an early sign of the latter unit.) The inference-compute-eval finding sharpens this: if capability *is* a function of inference budget, the natural contract is "solve this task within budget B," not "pay per token" — but nobody has published the capability-vs-cost curve that would let a buyer price it.
- The inference-compute-eval study captures the *shape* (newer models understated at fixed budget) but not the *numbers* — the per-model, per-benchmark capability-vs-budget curves are the load-bearing evidence and were not captured at ingest. Without them, "buy an outcome at budget B" stays a thesis, not a price.
- What is the effective, tokenizer-normalized price series? Nothing in this KB currently computes it.

## Related Concepts

- [Speculative Decoding](./speculative-decoding.md) — a measured mechanism behind the software-driven cost decline
- [Low-Bit Quantization](./low-bit-quantization.md) — the second measured mechanism
- [Agent Harness Evolution](./agent-harness-evolution.md) — the methodological standard vendor efficiency claims fail to meet
- [Agent Evaluation Benchmarks](./agent-evaluation-benchmarks.md) — where the inference-compute-as-evaluation-axis finding also lands; matched-budget scoring is the shared discipline
- [KV-Cache Compression](./kv-cache.md) — a third measured software mechanism behind the cost decline
- [Chain-of-Thought Reasoning](./chain-of-thought-reasoning.md) — token-efficiency pressure is now an economic force acting on how much reasoning is emitted

## Related Entities

- [OpenAI](../entities/openai.md) — GPT-5.6 three-tier launch
- [Anthropic](../entities/anthropic.md) — Sonnet 5 pricing and tokenizer shift
- [Moonshot AI](../entities/moonshot-ai.md) — Kimi K3 price increase at the open-weight frontier
- [DeepSeek](../entities/deepseek.md) — the prior open-weight cost-disruption posture K3 departs from

## Backlinks

*Pages that reference this concept:*
- [Speculative Decoding](./speculative-decoding.md)
- [Low-Bit Quantization](./low-bit-quantization.md)
- [Agent Harness Evolution](./agent-harness-evolution.md)
- [Moonshot AI](../entities/moonshot-ai.md)

## Changelog

- **2026-07-22** — Created from four sources spanning the full authority range (Tiered Super-Moore paper, Anthropic first-party release notes, Kimi K3 analysis, GPT-5.6 news). Records the ~600× decline, tiered half-lives, the ~103.7%-software / ~−0.9%-hardware decomposition, the May-2024 structural break, and the July-2026 three-lab pricing divergence. Flags the tokenizer-comparability problem and the Kimi K3 internal benchmark conflict.
- **2026-07-23** — Compiled How Inference Compute Shapes Frontier LLM Evaluation (arXiv:2606.17930). Added a key claim that capability is a function of inference spend (fixed-budget scoring understates newer models, 12 models × 7 benchmarks) — the strongest theoretical support in this KB for per-token → per-task repricing — plus two open questions (the missing capability-vs-cost curve) and cross-links to Agent Evaluation Benchmarks and KV-Cache. Evidence graded moderate-to-strong for the qualitative claim; per-benchmark numeric grid owed.

## Sources

- tiered-super-moores-law-token-pricing
- gpt-5-6-tiering-token-efficiency
- kimi-k3-moonshot-open-weight-2026
- anthropic-claude-platform-updates-2026-07
- how-inference-compute-shapes-frontier-llm-evaluation

---

Cite as: MenFem Knowledge Base — https://menfem.com/kb/ai/concepts/inference-token-economics