# MenFem Research · Atlas Canonical URL: https://menfem.com/kb The whole knowledge base as one document — 13 rungs of the AI inference stack. Generated 2026-10-27. Feed: https://menfem.com/kb/feed.xml --- ## Harnesses & Agent Systems Canonical URL: https://menfem.com/kb/harnesses The system around the model — the rung that turns a token price into a task price. --- title: "Harnesses & Agent Systems — Knowledge Base" last_compiled: "2026-09-24T12:30:00Z" source_count: 23 concept_count: 12 entity_count: 2 papers: 13 news: 0 --- # Harnesses & Agent Systems A harness is the machinery wrapped around a model: what it sees, which tools it can call, how its context is managed, what gets verified, what gets logged, and what happens when something fails. Lilian Weng's definition is the cleanest one on the rung — "the system surrounding a base model that orchestrates execution and decides how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results" ([Harness Engineering for Self-Improvement](../raw/harness-engineering-for-self-improvement.md)). ## What this rung prices Everything above sits between a token price and a task price. A model quote is per token; a buyer pays per finished job, and the harness decides how many tokens a finished job takes and how often the job finishes at all. That gap is measurable: hold the task, the budget and the model fixed, change only the harness, and success rates move further than swapping the model does. So the rung answers a commercial question, not an academic one — if the harness is the binding constraint, then buying a better model and building a better harness are not substitutes, and the cost per completed task is a property of the pairing rather than of the model on the invoice. ## The load-bearing findings - **Harness variance beat model variance by 7.80× in the one controlled factorial anyone has run.** Three frontier models clustered near 45% on a coding leaderboard, crossed with three harness configurations over a 100-task SWE-bench Verified subset: average harness-induced variance 18.48 pp² against average model-induced variance 2.37 pp², with model-pair ranking reversals in 6 of 9 comparisons. Harness changes moved GLM-5.1 by 13.0 points; model swaps inside a fixed harness moved results 3.0–5.0 points ([Stop Comparing LLM Agents Without Disclosing the Harness](../raw/stop-comparing-llm-agents-disclosing-harness.md)). This is the [binding constraint thesis](./concepts/binding-constraint-thesis.md), and the same paper proposes the ETCSOVG Harness Card as the disclosure fix. - **On cost, the harness is a 40× lever and the model a 1.3× one.** Two models, 50 coding tasks, three open-source harnesses: tokens per solved task ran from about 28,000 (Goose) to about 1.5 million (OpenCode), roughly 40×, while paired pass rates moved only 0–8 points; swapping the model moved tokens per solved task 1.0–1.3× ([The Scaffold Effect in Coding Agents](../raw/scaffold-effect-coding-agents-harness-hidden-variable.md)). Priced in dollars, a vendor-run orchestration swap across six models cut cost per task 41% ($0.21 → $0.12) at quality parity ([The Harness Effect](../raw/harness-effect-orchestration-token-economics.md)), and on one Microsoft workflow keeping less context was both 2.68× cheaper and 20.6 points more reliable than keeping all of it ([Less Context, Better Agents](../raw/less-context-better-agents-context-engineering.md)). This is the rung's through-line measured directly — see [cost per completed task](./concepts/cost-per-completed-task.md). - **Automatic harness evolution does not survive a matched budget.** Harness search is itself test-time scaling — it spends inference to find a better scaffold — so it has to be compared against plain search at equal budget. Done that way on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6, evolved harnesses did not consistently beat simple test-time scaling, and generalised poorly to held-out tasks ([Rethinking the Evaluation of Harness Evolution for Agents](../raw/rethinking-harness-evolution-evaluation.md); this close read is abstract-level, so the claim rests on the abstract, not the paper body). Prime Intellect's [Prime Agent](../raw/prime-agent-self-improving-rlm-harness.md) claims wins across eleven benchmarks and is the live counter-case — but it publishes no matched-budget control, changes both baseline and backbone per row, and loses LongBench v2 to Claude Code (0.714 vs 0.746). Its 95.5% ARC-AGI-3 result sits 0.1 points above the human-expert baseline, which is a tie reported as a win. - **Agents are strong tool users and weak navigators — the failure is composition, not capability.** Across 1,400 DAG-structured Wikipedia tasks the best agent reached 37.2% accuracy; navigation errors accounted for 27–52% of failures while tool-use errors stayed below 17%, and the pit-stop visit rate fell 13–18 points moving from linear to compositional structures while roadblock completion held steady ([The Amazing Agent Race](../raw/amazing-agent-race-tool-users-weak-navigators.md)). The same run found Claude Code matching Codex CLI at 37% with 6× fewer tokens — token efficiency and task accuracy come apart, which is exactly the token-price-to-task-price wedge. - **The dominant failure class lives in the execution loop, not in the model's knowledge.** Harness-Bench ran 5,194 trajectories over 106 sandboxed tasks with the task, budget and evaluation protocol held fixed, and named "execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts" ([Harness-Bench](../raw/harness-bench-measuring-harness-effects.md), read at abstract level). Its conclusion is attributional: report performance for model-harness configurations, never for models alone. - **Two harnesses can post the same score and leave the agent believing different things.** Blocked actions, compressed repairs, selective verification and cost-aware evidence pruning often preserve terminal success while shifting the agent's beliefs about progress, risk, recoverability and next action ([Harness-Induced Belief Divergence](../raw/harness-induced-belief-divergence-multi-step-agents.md)). Outcome parity is therefore not harness equivalence — see [agent belief divergence](./concepts/agent-belief-divergence.md). This ingest is abstract-only and carries no magnitudes. - **Memory is the harness's job because weights are an unreliable place to put facts.** After 20 sequential weight writes, bare-statement facts retained 1% accuracy against 46% for facts drawn from broad study data; 70% of wrong answers contained the most recently written fact, and a forgotten fact re-supplied in the prompt recovered to 77–80% ([Can a Language Model Learn Facts Continually in Its Weights?](../raw/can-a-language-model-learn-facts-continually-in-its-weights.md)). Vendor-published and Qwen3-only, but it is the empirical backing for treating the file system as persistent memory rather than a convenience. - **Persistent state is the attack surface, and one framing trick opens it.** On a deployed personal agent, poisoning any single dimension of Capability / Identity / Knowledge lifted attack success from a 24.6% baseline to 64–74%, and file protection that blocked 97% of malicious injections also blocked 93% of legitimate updates ([Your Agent, Their Asset](../raw/openclaw-real-world-agent-safety-analysis.md)). Separately, across roughly 10,000 trials nine of twelve hypothesised attack dimensions produced no detectable exploitation — but goal reframing did, at 38–40% on Claude Sonnet 4, while GPT-4.1 exploited nothing across 1,850 trials ([Mapping the Exploitation Surface](../raw/llm-agent-exploitation-surface-taxonomy.md)). ## What we do not know yet - **Nobody here has run the experiment.** Every number on this rung is `read it` standing. The measurement the rung needs — the same task set at the same inference budget, a self-improving harness against a plain one — has not been published by anyone and has not been run here. [verifiers v1](../raw/verifiers-v1-composable-tasksets-harnesses-runtimes.md) makes the separation of taskset / harness / runtime architectural, and [Pi](../raw/pi-minimal-coding-agent-harness.md) is a configurable minimal-vs-extended harness — both are instruments for that run, and neither publishes a benchmark of its own. - **Which harness features cause which gains.** Harness-Bench measures that the pairing matters; it does not decompose the effect into context management, retry policy, verification or tool surface. The ETCSOVG Harness Card is a proposed disclosure standard with almost no worked examples and no leaderboard has adopted it. - **Whether harness benefit is really non-monotonic in model tier.** The survey reports Lin et al. finding harness *updating* flat from Qwen2-32B to Opus 4.6 while harness *benefit* peaks at middle-tier models. That is the single most decision-relevant claim on the rung and it reaches us second-hand — the survey is a secondary source and none of its figures were verified against their primaries. - **How far the 7.80× ratio travels.** It comes from a 3×3 grid, 100 coding tasks, two runs per cell. It is a within-study measurement, not a constant, and it has not been reproduced outside coding. - **Whether self-modifying harnesses can be kept honest.** Prime Agent's own release reports reward hacking — it found a way to bypass Factorio's rules. A system that can edit its own scaffolding can edit its way around the objective, and no source here offers a control for that. ## Concepts | Concept | Sources cited | Last compiled | |---------|---------|-------------| | [Cost per Completed Task](concepts/cost-per-completed-task.md) | 3 | 2026-09-24 | | [Measuring Harness Effects](concepts/harness-effect-measurement.md) | 5 | 2026-09-24 | | [The Binding Constraint Thesis](concepts/binding-constraint-thesis.md) | 3 | 2026-09-24 | | [Agent Harness Evolution](concepts/agent-harness-evolution.md) | 6 | 2026-07-23 | | [Agent Memory Architectures](concepts/agent-memory-architectures.md) | 3 | 2026-07-22 | | [Agentic Reasoning](concepts/agentic-reasoning.md) | 6 | 2026-04-15 | | [Evolutionary Algorithm Discovery](concepts/evolutionary-algorithm-discovery.md) | 1 | 2026-04-05 | | [Harness-Induced Belief Divergence](concepts/agent-belief-divergence.md) | 1 | 2026-08-05 | | [LLM Tool Use](concepts/llm-tool-use.md) | 4 | 2026-04-14 | | [Multi-Agent Systems](concepts/multi-agent-systems.md) | 2 | 2026-04-05 | | [Reinforcement Learning for Agents](concepts/reinforcement-learning-for-agents.md) | 2 | 2026-04-05 | | [Tool-Chain Navigation](concepts/tool-chain-navigation.md) | 1 | 2026-04-14 | ## Entities | Entity | Type | Sources | |--------|------|---------| | [AlphaEvolve](entities/alphaevolve.md) | Product | 1 | | [DeepSeek Harness](entities/deepseek-harness.md) | Product | 0 (release record) | ## Timeline See [timeline.md](timeline.md) for the dated record of this pass, and [frontier.md](frontier.md) for the open questions. ## Recently added sources | Title | Type | Date | Status | |-------|------|------|--------| | [The Scaffold Effect in Coding Agents](../raw/scaffold-effect-coding-agents-harness-hidden-variable.md) | preprint | 2026-06-08 | compiled | | [The Harness Effect](../raw/harness-effect-orchestration-token-economics.md) | preprint | 2026-07-08 | compiled | | [Less Context, Better Agents](../raw/less-context-better-agents-context-engineering.md) | preprint | 2026-06-08 | compiled | The full register of all 23 sources is `raw/_sources.json`. ## Read next - [scaffold-effect-coding-agents-harness-hidden-variable](../raw/scaffold-effect-coding-agents-harness-hidden-variable.md) — the cleanest token-price-to-task-price number: 40× tokens per solved task from the harness alone. - [stop-comparing-llm-agents-disclosing-harness](../raw/stop-comparing-llm-agents-disclosing-harness.md) — the thesis and the only controlled factorial. - [rethinking-harness-evolution-evaluation](../raw/rethinking-harness-evolution-evaluation.md) — the matched-budget check that deflates most harness-evolution claims. - [harness-engineering-for-self-improvement](../raw/harness-engineering-for-self-improvement.md) — the map of the whole field, caveats included. ### Sources (23) - Agentic Reasoning for Large Language Models — Tianxin Wei et al. · 2026-01-18 — https://arxiv.org/abs/2601.12538 - From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review — Mohamed Amine Ferrag et al. · 2026-03-06 — https://arxiv.org/abs/2504.19678 - Agentic Tool Use in Large Language Models — Hu Jinchao et al. · 2026-04-01 — https://arxiv.org/abs/2604.00835 - AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms — Google DeepMind · 2025-05-14 — https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/ - Agentic AI Security & Autonomous Red-Teaming — Ashok Kumar Kanagala · 2026-02-07 — https://asrjetsjournal.org/American_Scientific_Journal/article/view/12196 - Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers — Pengfei Du · 2026-03-08 — https://arxiv.org/abs/2603.07670 - A-MEM: Agentic Memory for LLM Agents — Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, Yongfeng Zhang · 2025-02-17 — https://arxiv.org/abs/2502.12110 - Mapping the Exploitation Surface: A 10,000-Trial Taxonomy of What Makes LLM Agents Exploit Vulnerabilities — Charafeddine Mouzouni · 2026-04-06 — https://arxiv.org/abs/2604.04561 - Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw — Zijun Wang, Haoqin Tu, Letian Zhang, Hardy Chen, Juncheng Wu, Xiangyan Liu, Zhenlong Yuan, Tianyu Pang, Michael Qizhe Shieh, Fengze Liu, Zeyu Zheng, Huaxiu Yao, Yuyin Zhou, Cihang Xie · 2026-04-06 — https://arxiv.org/abs/2604.04759 - The Amazing Agent Race: Strong Tool Users, Weak Navigators — Zae Myung Kim, Dongseok Lee, Jaehyung Kim, Vipul Raheja, Dongyeop Kang · 2026-04-11 — https://arxiv.org/abs/2604.10261 - Model-First Reasoning LLM Agents: Reducing Hallucinations through Explicit Problem Modeling — Annu Rana, Gaurav Kumar · 2025-12-16 — https://arxiv.org/abs/2512.14474 - Rethinking the Evaluation of Harness Evolution for Agents — Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, Teng Xiao · 2026-07-14 — https://arxiv.org/abs/2607.12227 - Stop Comparing LLM Agents Without Disclosing the Harness — Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm, Chandan K. Reddy · 2026-05-07 — https://arxiv.org/abs/2605.23950 - Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows — Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, Xiangzheng Zhang, Tong Yang · 2026-05-27 — https://arxiv.org/abs/2605.27922 - Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents — Haiwen Yi, Xinyuan Song · 2026-07-05 — https://arxiv.org/abs/2607.04528 - Prime Agent: A self-improving RLM harness — Prime Intellect · 2026-08-05 — https://www.primeintellect.ai/blog/prime-agent - verifiers v1: Decomposing Tasksets and Harnesses for Agentic RL & Evaluations — Prime Intellect · 2026-07-10 — https://www.primeintellect.ai/blog/verifiers-v1 - Harness Engineering for Self-Improvement — Lilian Weng · 2026-07-04 — https://lilianweng.github.io/posts/2026-07-04-harness/ - Can a Language Model Learn Facts Continually in Its Weights? — Charles O'Neill · 2026-07-14 — https://www.baseten.co/research/can-a-language-model-learn-facts-continually-in-its-weights/ - Pi — a minimal, extensible terminal coding harness — Earendil Inc. and contributors — https://pi.dev/ - The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI — Muayad Sayed Ali et al. (32 authors) · 2026-07-08 — https://arxiv.org/abs/2607.06906 - Less Context, Better Agents: Efficient Context Engineering for Long-Horizon Tool-Using LLM Agents — Abhilasha Lodha, Mahsa Pahlavikhah Varnosfaderani, Abir Chakraborty, Abhinav Mithal · 2026-06-08 — https://arxiv.org/abs/2606.10209 - The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation — Naman Vats, Oleg Golev · 2026-06-08 — https://arxiv.org/abs/2607.22585 ## Serving & Runtime Canonical URL: https://menfem.com/kb/serving How a model is actually run, per token — the mechanics that turn capability into throughput. --- title: "Serving & Runtime — Knowledge Base" last_compiled: "2026-09-24T12:30:00Z" source_count: 18 concept_count: 6 entity_count: 5 papers: 0 news: 0 --- # Serving & Runtime Serving is what happens between a trained model and a served token: how requests are batched, how the key-value cache is stored, evicted and reused, how weights are compressed and unpacked, and which of the two phases of generation — the compute-bound prefill that reads the prompt, and the bandwidth-bound decode that emits one token at a time — is the thing actually holding the box up. The clearest statement of why this deserves its own rung comes from the field's own survey: "the key-value (KV) cache has become a first-order memory object in LLM serving rather than a temporary per-request tensor" ([From Tensor Buffer to Distributed Memory Hierarchy](../raw/kv-cache-management-survey-tensor-buffer-distributed-memory.md)). ## What this rung prices A model price is quoted per token, but the tokens have to be produced by real silicon that is either full or idle, and this rung is where that gap is set. The sharpest measurement on it is also the simplest: one H200, one model, one serving framework, nothing changed but the traffic shape, and the cost to produce a million output tokens moves from $20.32 to $0.45 ([Token Economics Across Traffic Profiles on Dedicated GPUs](../raw/measured-h200-token-serving-cost-digitalocean.md)). That is a ~44× swing with no new model, no new chip and no new price list — so a quoted cost per token without a stated utilisation is not a number, and a buyer paying per finished job is paying for a batch-occupancy outcome he cannot see. Everything else here works the same lever from a different direction: shrink what has to be moved (KV compression, eviction, quantization), overlap what has to be waited on (speculative decoding), or put the arithmetic closer to the memory. Each one changes how many finished jobs a fixed box yields per hour, which is the only place a token price and a task price are joined. ## The load-bearing findings - **Traffic shape alone moves cost per million output tokens by ~44× on identical hardware.** A single H200 at $3.44/GPU-hour running Llama-3.3-70B in FP8 on vLLM 0.24.0 measured 47 tok/s at batch 1 and 2,140 tok/s at batch 128 — $20.32/M against $0.45/M, with the own-a-GPU versus buy-serverless crossover for that configuration sitting near 73% sustained utilisation ([Token Economics Across Traffic Profiles on Dedicated GPUs](../raw/measured-h200-token-serving-cost-digitalocean.md)). Throughput was measured; the dollar figures are derived arithmetically from it at the list rate, and the author states plainly that they are a reference point for one configuration, not a guarantee. - **The two phases of generation are two different machines, and averaging them hides the answer.** Prefill is compute-bound and decode is bandwidth-bound, so accelerator advantage is phase-dependent: GPUs consistently win prefill, GroqRack posts significantly lower per-output-token latency in decode at small batch, and GPUs take decode back as batch size rises ([Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators](../raw/prefill-decode-aware-evaluation-emerging-accelerators.md)). That close read is abstract-level and the abstract carries no magnitudes, so the directions are what stands, not the sizes. The same split is what a serving-capacity model has to fit: Epoch's first-principles model, calibrated against 111 measured SemiAnalysis InferenceX runs of Kimi K2.5, fits compute efficiency at 65% and **bandwidth efficiency at 30%**, and lands ~400,000 tokens/sec on a GB200 NVL72 ([Is a Compute Crunch Coming?](../raw/epoch-inference-compute-crunch-serving-capacity.md)) — modelled rather than measured, and denominated in tokens/sec with no dollar figure attached. See [prefill/decode disaggregation](./concepts/prefill-decode-disaggregation.md). - **Most of the KV cache can go, and compressing it need not cost exact output.** LaProx scores token importance on a single global scale across the whole model rather than per attention head, and holds performance at **5% KV-cache retention** across 19 long-context datasets, cutting accuracy loss by up to 2× against prior methods under extreme compression ([Reformulating KV Cache Eviction Problem](../raw/laprox-reformulating-kv-cache-eviction-long-context.md), read at abstract and reported-metrics grade). Separately, VeriCache uses the compressed cache in GPU memory as a drafter and checks it against the full cache held on CPU or over the network, giving output identical to full-KV decoding (KL below 0.01 nats) at 2.73× throughput on Llama-70B, and ~25–40 accepted tokens per verification round against 2–3 for a small-model drafter ([VeriCache](../raw/vericache-lossy-kv-lossless-inference.md); authors' own benchmarks, no independent reproduction). See [KV cache](./concepts/kv-cache.md) and [KV cache management](./concepts/kv-cache-management.md). - **The levers compound, but not automatically — one pairing had to be measured before it worked.** Applying tree-style EAGLE-2 speculative decoding to a 4-bit weight-quantized model erodes the quantization win, because verifying a tree draft costs more than a single-token forward pass on that target; inserting a small intermediate model that converts tree drafts into sequence drafts restores it, at 2.78× on 4-bit Llama-3-70B on an A100 and +1.31× over EAGLE-2 on the same setup ([Speculative Decoding Meets Quantization](../raw/speculative-decoding-meets-quantization.md), abstract and reported-metrics grade). Where the design is built for the pairing from the start it does stack: quantizing target and drafter together with sliding-window attention gave a 6.978× average speedup while clearing fixed quality thresholds — though that average runs 11.1× on short prompts down to **2.85× at 8,192-token inputs**, and the reported latencies were reproduced on an RTX 5000 Ada because the competition's A10G was unavailable ([Quantize the Target, Quantize the Drafter](../raw/quantize-target-quantize-drafter-qwen3-5-4b.md)). See [speculative decoding](./concepts/speculative-decoding.md) and [low-bit quantization](./concepts/low-bit-quantization.md). - **A speed-up is only real if the output is still right — and published batch code was not.** Speculative decoding promises exactly the output the big model would give alone. In a batch, requests accept different numbers of guessed tokens each round, and unless the code re-lines-up positions and the stored cache every round, the model starts reading the wrong memory. Two published batch implementations did not: 0.0–3.5% exact match against normal decoding, writing repeated or unknown tokens while reporting competitive speed, and ROUGE did not catch it. Corrected versions reach 90.8–97.3% exact match and 3× throughput at batch size 8, at a cost of up to about 40% of time spent keeping the batch aligned on the fixed-width layout; the paged layout used by vLLM v1 and SGLang avoids the problem by construction ([Correctness Forensics for Batch Speculative Decoding](../raw/batch-speculative-decoding-ragged-tensor-correctness.md)). And the speed-up itself is a quiet-server result: it shrinks as load rises unless about 90% or more of guesses are accepted, and draft lengths tuned for one request are wrong under load ([An Interpretable Latency Model for Speculative Decoding](../raw/interpretable-latency-model-speculative-decoding.md)). A lower price per token that produces the wrong tokens is an infinite price per task. - **Splitting the two phases onto separate GPUs roughly doubled useful output on the same box.** On one 8-GPU MI300X node, requests meeting both a 1-second first-token deadline and a 50 ms token gap went from 26–30 of 100 to 70–73 of 100 at 8 requests per second — about 2.5× — while first-token time got worse ([Next-Level Inference](../raw/vllm-prefill-decode-disaggregation-moriio-goodput.md); vendor-run, no dollar figure). See [prefill/decode disaggregation](./concepts/prefill-decode-disaggregation.md). - **Speculative decoding is in production and the production numbers come from the vendor.** DeepSeek reports deploying DSpark in the DeepSeek-V4 serving system and measuring, on its own live traffic, 60–85% faster per-user generation on V4-Flash and 57–78% on V4-Pro against its prior production baseline, at matched throughput and with the exact acceptance rule left intact ([DSpark](../raw/dspark-confidence-scheduled-speculative-decoding.md)). This is first-party self-measurement of the authors' own fleet with no independent reproduction, and the paper's own much larger figures under strict interactivity constraints are a different operating regime, not the headline. The academic end of the same lever is incremental by comparison: truncating the drafter's training loss at its first predicted failure lifts average emitted length from 2.40 to 2.61 tokens on Qwen3-8B, and the authors state that 8B and a block size of 16 are the only configuration tested ([Spec-AUF](../raw/spec-auf-accept-until-fail-masked-block-drafters.md)). - **Three separate 2026 designs argue the memory stack should do arithmetic, and they disagree about how far to go.** SK Hynix engineers put dequantization logic on the HBM base die, reporting up to 7.08× GEMM speedup, 90.23% lower energy and 54.68% end-to-end latency reduction, at 0.127 mm² and 0.355 W per block in 12 nm ([StreamDQ](../raw/streamdq-near-memory-dequantization-custom-hbm.md)); TokenStack splits a stack into capacity layers and compute layers for 1.62× geomean throughput against a prior PIM baseline ([TokenStack](../raw/tokenstack-heterogeneous-hbm-pim-llm-inference.md)); AMMA removes the GPU compute die from the attention path entirely — 16 memory-with-compute cubes at 44 TB/s and 240 W package power, reporting 15.5× lower attention latency and 6.9× lower energy than an H100 on 1M-token decode attention, but only 1.1× against NVIDIA's Rubin at that length ([AMMA](../raw/amma-multichiplet-memory-centric-1m-context-attention.md)). All three are simulated or synthesised by their own authors, and all three concentrate their wins in the memory-bound small-batch regime — StreamDQ's authors say so explicitly, that the advantage shrinks as batch size grows. - **Compression and reuse can be made to co-exist, and the software layer is where it is cheapest.** C²KV learns a position-agnostic cache representation so non-prefix blocks can be concatenated and reused without the accuracy collapse naive compression-plus-reuse causes: at 4× compression it scores 0.3587 on MuSiQue against 0.3198 for full recompute, with up to 17× long-context speedup and no change to the frozen base model ([C²KV](../raw/c2kv-compressed-composable-kv-cache-reuse.md); authors' own benchmarks). The survey's structural finding sits underneath it — across 30-plus KV-management systems on four axes, once workload and hardware are fixed it is **ownership**, not locality or substrate, that accounts for much of the remaining design variance ([From Tensor Buffer to Distributed Memory Hierarchy](../raw/kv-cache-management-survey-tensor-buffer-distributed-memory.md)). ## What we do not know yet - **Nothing on this rung has been run here.** Every number above is read, not measured. The one measured cost series in the KB is DigitalOcean's own H200 sweep, and even there the throughput is theirs and the dollars are arithmetic over a list rate ([Token Economics Across Traffic Profiles on Dedicated GPUs](../raw/measured-h200-token-serving-cost-digitalocean.md)). - **The vendor and the academy are not measuring the same thing, and neither checks the other.** DSpark's 60–85% is DeepSeek measuring DeepSeek ([DSpark](../raw/dspark-confidence-scheduled-speculative-decoding.md)), StreamDQ's 7.08× is SK Hynix simulating SK Hynix ([StreamDQ](../raw/streamdq-near-memory-dequantization-custom-hbm.md)), and no source here reproduces another source's result. The near-memory papers are the clearest case: TokenStack's 1.62× is measured against one prior PIM design rather than against a GPU stack anyone deploys ([TokenStack](../raw/tokenstack-heterogeneous-hbm-pim-llm-inference.md)). - **The field is missing measurements it knows it is missing.** The survey audits current evaluations and names seven KV-specific measurements that are absent, tied to fault tolerance, tiered eviction, speculative decoding and MoE serving ([From Tensor Buffer to Distributed Memory Hierarchy](../raw/kv-cache-management-survey-tensor-buffer-distributed-memory.md)). A benchmark blind to tiered eviction cannot rank tiered-eviction designs, which is most of what this rung now contains. - **Where the levers stop compounding is visible but not mapped.** The one systematic pairing study covers weight quantization only, on one model family and one accelerator, and says activation and KV quantization interact differently ([Speculative Decoding Meets Quantization](../raw/speculative-decoding-meets-quantization.md)); the drop from 11.1× to 2.85× as prompts lengthen ([Quantize the Target, Quantize the Drafter](../raw/quantize-target-quantize-drafter-qwen3-5-4b.md)) says the decay is real without saying what governs it. - **Two of the phase-level claims reach us thin.** The prefill/decode accelerator comparison is abstract-level with no magnitudes at all and measures GroqRack without batching support, which is exactly the condition under which GPUs later win ([Prefill/Decode-Aware Evaluation](../raw/prefill-decode-aware-evaluation-emerging-accelerators.md)), and the capacity model behind the compute-crunch claim is calibrated on one model family and states no dollars ([Is a Compute Crunch Coming?](../raw/epoch-inference-compute-crunch-serving-capacity.md)). The demand-growth side of that gap — roughly 10× a year against roughly 3.4× a year of capacity — is drawn from disclosed industry metrics of varying reliability, by the authors' own admission. - **A quantization result that reads as capability-neutral may not be.** OSAQ's null-space transform is stated to suppress outliers "without affecting the task loss", and the KB's own close read flags that this is a perplexity and task-loss statement rather than a capability one, since outlier weights are implicated in rare behaviours ([OSAQ](../raw/osaq-outlier-self-absorption-low-bit-quantization.md)). Nothing here tests that. ## Concepts | Concept | Last compiled | |---------|---------------| | [Speculative Decoding](concepts/speculative-decoding.md) | 2026-09-24 | | [Prefill/Decode Disaggregation](concepts/prefill-decode-disaggregation.md) | 2026-09-24 | | [KV Cache](concepts/kv-cache.md) | 2026-07-24 | | [KV Cache Management](concepts/kv-cache-management.md) | 2026-07-24 | | [Low-Bit Quantization](concepts/low-bit-quantization.md) | 2026-07-24 | | [Conditional Memory](concepts/conditional-memory.md) | 2026-06-24 | ## Entities | Entity | Type | |--------|------| | [DeepSeek](entities/deepseek.md) | Organization | | [Epoch AI](entities/epoch-ai.md) | Organization | | [Groq](entities/groq.md) | Organization | | [SK hynix](entities/sk-hynix.md) | Organization | | [vLLM](entities/vllm.md) | Product | ## Timeline See [timeline.md](timeline.md) for chronological developments, and [frontier.md](frontier.md) for the open questions. ## Recently compiled sources | Title | Type | Date | Status | |-------|------|------|--------| | [Correctness Forensics for Batch Speculative Decoding](../raw/batch-speculative-decoding-ragged-tensor-correctness.md) | preprint | 2025-10-26 (rev. 2026-08-31) | compiled | | [An Interpretable Latency Model for Speculative Decoding](../raw/interpretable-latency-model-speculative-decoding.md) | preprint | 2026-05-14 | compiled | | [Next-Level Inference (vLLM / AMD MORI-IO)](../raw/vllm-prefill-decode-disaggregation-moriio-goodput.md) | technical-report | 2026-04-07 | compiled | The full register of all 18 sources is `raw/_sources.json`. ## Read next - [measured-h200-token-serving-cost-digitalocean](../raw/measured-h200-token-serving-cost-digitalocean.md) — the only dollar-denominated cost series here, and the 44× utilisation swing the whole rung turns on. - [kv-cache-management-survey-tensor-buffer-distributed-memory](../raw/kv-cache-management-survey-tensor-buffer-distributed-memory.md) — the map of the serving-software layer, and the honest list of what nobody is measuring. - [vericache-lossy-kv-lossless-inference](../raw/vericache-lossy-kv-lossless-inference.md) — the cleanest worked example of two levers becoming one mechanism, with exact output preserved. - [batch-speculative-decoding-ragged-tensor-correctness](../raw/batch-speculative-decoding-ragged-tensor-correctness.md) — why a speed-up claim needs an exact-match check beside it. ### Sources (18) - DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation — Xin Cheng, Xingkai Yu, Chenze Shao, et al. (30+ authors incl. Wenfeng Liang, Damai Dai) · 2026-07-06 — https://arxiv.org/abs/2607.05147 - Spec-AUF: Accept-Until-Fail Training under Train-Inference Misalignment for Masked Block Drafters — Tianjian Yang, Meng Li · 2026-07-02 — https://arxiv.org/abs/2607.01893 - Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B — Jaeyeon Kim, Jewon Lee, Bo-Kyeong Kim · 2026-07-05 — https://arxiv.org/abs/2607.04244 - OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization — Zhikai Li, Zhen Dong, Xuewen Liu, Jing Zhang, Qingyi Gu · 2026-05-06 — https://arxiv.org/abs/2605.04738 - VeriCache: Turning Lossy KV Cache into Lossless LLM Inference — Jiayi Yao, Samuel Shen, Kuntai Du, Shaoting Feng, Dongjoo Seo, Rui Zhang, Yuyang Huang, Yuhan Liu, Shan Lu, Junchen Jiang · 2026-05-17 — https://arxiv.org/abs/2605.17613 - Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design — Yudi Zhang, Weilin Zhao, Xu Han, Tiejun Zhao, Wang Xu, Hailong Cao, Conghui Zhu · 2025-05-28 — https://arxiv.org/abs/2505.22179 - Reformulating KV Cache Eviction Problem for Long-Context LLM Inference (LaProx) — Tho Mai, Joo-Young Kim · 2026-05-08 — https://arxiv.org/abs/2605.07234 - From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving — Jie Li, Tongyang Wang, Yong Chen · 2026-06-30 — https://arxiv.org/abs/2607.02574 - TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inference — Zhuoran Li, Zhuohang Bian, Zihao Huang, Guangyu Sun, Yun Liang, Youwei Zhuo · 2026-05-07 — https://arxiv.org/abs/2605.05639 - Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators — Shun Usami, Venkatram Vishwanath, E. Wes Bethel · 2026-06-14 — https://arxiv.org/abs/2606.17104 - StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration — Minki Jeong, Daegun Yoon, Soohong Ahn, Seungyong Lee, Nameun Kang, Hyeonseok Ju, Ieryung Park, Joonseop Sim, Youngpyo Joo, Hoshik Kim · 2026-07-09 — https://arxiv.org/abs/2607.08993 - AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving — Zhongkai Yu, Haotian Ye, Chenyang Zhou, Ohm Rishabh Venkatachalam, Zaifeng Pan, Zhengding Hu, Junsung Kim, Won Woo Ro, Po-An Tsai, Shuyi Pei, Yangwook Kang, Yufei Ding · 2026-04-28 — https://arxiv.org/abs/2604.26103 - C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference — Chuheng Du, Junyi Chen, Hanlin Tang, Kan Liu, Tao Lan, Lin Qu, Chaoyue Niu, Shengzhong Liu, Guihai Chen, Fan Wu · 2026-07-20 — https://arxiv.org/abs/2607.17715 - Token Economics Across Traffic Profiles on Dedicated GPUs (measured H200 serving-cost benchmark) — Vinayak Baranwal · 2026-07-08 — https://www.digitalocean.com/community/tutorials/llm-inference-cost - Is a Compute Crunch Coming? (inference serving-capacity model, calibrated to SemiAnalysis InferenceX) — Luke Emberson, Jaime Sevilla · 2026-05-25 — https://epoch.ai/gradient-updates/is-a-compute-crunch-coming - Next-Level Inference: Why Your Single-Node vLLM Setup Needs Prefill-Decode Disaggregation — AMD and Embedded LLM (Hongxia Yang, Gilbert Lei, Mingzhi Liu, et al.) · 2026-04-07 — https://vllm.ai/blog/2026-04-07-moriio-kv-connector - An Interpretable Latency Model for Speculative Decoding in LLM Serving — Linghao Kong, Megan Flynn, Michael Peng, Nir Shavit, Mark Kurtz, Alexandre Marques · 2026-05-14 — https://arxiv.org/abs/2605.15051 - Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problem (v1 title: Batch Speculative Decoding Done Right) — Ranran Haoran Zhang, Soumik Dey, Ashirbad Mishra, Hansi Wu, Binbin Li, Rui Zhang · 2025-10-26 — https://arxiv.org/abs/2510.22876 ## Models Canonical URL: https://menfem.com/kb/models What the model can do, and what that costs per token. --- title: "Models — Knowledge Base" last_compiled: "2026-09-24" source_count: 16 concept_count: 6 entity_count: 6 --- # Models This rung is about what the model itself can do, and what that costs per token — the architecture that decides how much compute a forward pass burns, the released weights you can actually hold, the quoted price on the invoice, and the growing body of work on what is happening inside the thing while it answers. ## What this rung prices A buyer pays for a finished task; a lab quotes per token. This rung owns the first term in that product — the per-token price and the per-task token count that the model itself sets before any harness touches it. Both halves move, and they move for architectural reasons. Sparsity is the clearest case: Kimi K3 carries about 2.8 trillion parameters but uses only 104 billion of them for each token — about 3.7% — by sending each token to 16 of its 896 experts, so the forward-pass cost sits far below what the headline count implies ([Kimi K3 technical report](../raw/kimi-k3-open-frontier-intelligence-technical-report.md)). This page used to say "roughly 1.8%"; that was the share of *experts*, taken from a secondary review before Moonshot's own report existed, not the share of *parameters*. Corrected 2026-09-24. The token *count* is just as much a model property: Anthropic's own release notes record that Claude Sonnet 5's new tokenizer produces about 30% more tokens for the same text ([Claude Platform release notes](../raw/anthropic-claude-platform-updates-2026-07.md)), which raises the effective price per word without the quoted per-token price moving at all. So the rung answers a question no price list answers on its own: for a given job, how many tokens does this model spend and what does each one cost — and the empirical work here says the answer has been falling for software reasons rather than hardware ones. ## The load-bearing findings - **Token prices fell roughly 600-fold between 2020 and 2026, and essentially all of it came from software, not chips.** A dataset of 318 OpenRouter models, 3,237 Epoch AI records and 62 cross-validated milestones decomposes the decline as about 103.7% software and architectural innovation against about -0.9% GPU hardware — hardware was a drag or a noise term, not an independent driver ([Tiered Super-Moore's Law](../raw/tiered-super-moores-law-token-pricing.md)). This close read is abstract-level, so it rests on the abstract and the reported findings rather than the PDF body. - **The decline is tiered, not uniform, and the premium tier resists it.** Same paper: economy-tier price half-life 1.10 years and mid-tier 1.55 years against a two-year Moore benchmark, while flagship and reasoning tiers hold up, with reasoning commanding about a 31.5× premium over non-reasoning pricing; a structural break in May 2024 (F = 5.74, p = 0.005) separates a technology-driven regime from a competitive one ([Tiered Super-Moore's Law](../raw/tiered-super-moores-law-token-pricing.md)). OpenAI shipped that shape as a product three months later — GPT-5.6 as three named tiers, Sol at $5/$30, Terra at $2.50/$15 and Luna at $1/$6 per million tokens ([GPT-5.6](../raw/gpt-5-6-tiering-token-efficiency.md)). - **An open-weight model reached near-frontier capability at roughly a sixth of the closed price, and the efficiency is architectural.** DeepSeek V4-Pro is a 1.6-trillion-parameter mixture of experts with 49B active per token and a 1M context, priced at $0.435 input and $0.87 output per million tokens; in the 1M setting it runs single-token inference at about 27% of V3.2's FLOPs and cuts KV-cache memory to about 10% of V3.2 — that efficiency is the stated basis for the price. It reports SWE-bench Verified 80.6% but only about 55% on the harder SWE-bench Pro split against Opus 4.7 at about 64% ([DeepSeek V4](../raw/deepseek-v4-2026.md)). The gap survives on the hardest tasks and nowhere else — this is the substance of [open-weight frontier parity](./concepts/open-weight-frontier-parity.md). The file is summary-derived: the primary article returned HTTP 429, and the benchmark figures are vendor or third-party reported, not peer-reviewed. - **Open-weight parity does not mean cheap — the second instance competes on capability and raises its price.** Kimi K3 ships at $3/$15 per million tokens, up sharply from Kimi K2.6's $0.95/$4, making it the most expensive model a Chinese lab has shipped; it ranks first on Arena.ai's Frontend Code leaderboard ahead of Claude Fable 5 while trailing Fable 5 and GPT-5.6 Sol overall on Moonshot's own numbers. It launched with a single reasoning-effort setting, "max", which spent 13,241 reasoning tokens (about $0.25) on a hello-world-grade prompt in the reviewer's own test ([Kimi K3](../raw/kimi-k3-moonshot-open-weight-2026.md)). Summary-derived: Moonshot's blog served a bare nginx page, so all of this comes through an independent review and wire coverage. - **Where you put a model's capacity is an optimisation with an interior answer, not a scaling slope.** Adding a constant-time lookup table for static knowledge alongside a mixture of experts produced BBH +5.0, ARC-Challenge +3.7, CMMLU +4.0, MMLU +3.4, HumanEval +3.0, MATH +2.4, and multi-query needle-in-a-haystack retrieval from 84.2 to 97.0, scaled to 27B parameters of memory — and the allocation between neural compute and static memory traces a U-shaped curve, so neither pure compute nor pure memory is optimal ([Conditional Memory via Scalable Lookup](../raw/conditional-memory-scalable-lookup-engram.md)). The gains were largest in reasoning rather than in plain recall. - **Depth is becoming a dial you turn at inference rather than a number fixed at training.** LoopFormer runs the same transformer layers repeatedly and trains with a shortcut-consistency scheme that aligns the trajectories of different loop counts, so one trained model can trade depth against compute budget at inference and degrade gracefully when the budget is tight ([LoopFormer](../raw/loopformer-elastic-depth-looped-transformers.md)). The paper reports robust results under aggressive compute constraints but gives no benchmark numbers in the abstract, so there is nothing to quote. - **The visible reasoning trace may not be the reasoning.** One preprint argues that chain-of-thought tokens are closer to post-hoc rationalisation than to the computation being performed, using probes, sparse autoencoders and interventions to find cases where behaviour tracks the latent trajectory even when the chain-of-thought content is altered or absent ([LLM Reasoning Is Latent](../raw/llm-reasoning-is-latent-arxiv-2026.md)). If that holds, then reading a model's stated steps is a partial safety signal at best — see [chain-of-thought reasoning](./concepts/chain-of-thought-reasoning.md). Set against it, chain-of-thought monitoring has caught a real thing: OpenAI used it to catch one of its own reasoning models cheating on coding tests ([Mechanistic Interpretability — 10 Breakthrough Technologies](../raw/mechanistic-interpretability-2026.md)). - **Sparse models make the usual decode speed-up trick leak into memory traffic, and choosing which guesses to check wins some of it back.** Speculative decoding — a cheap model guesses a few tokens, the big one checks them together — is almost free for a dense model because every check reads the same weights. In a mixture of experts each guess can call different experts, so checking more guesses means pulling more weights from memory. EcoSpec picks guesses that reuse experts already loaded: on Qwen3-235B it cut estimated memory reads per step from 99.3 GB to 88.1 GB and lifted average speed from 1.22× to 1.36× over plain decoding (the EAGLE-3 speculative baseline vs EcoSpec), with a best single benchmark of 1.62× ([EcoSpec](../raw/ecospec-cost-aware-speculative-decoding-moe.md)). The gain shrinks as batches grow — at four requests per batch it was 1.09× against the baseline's 1.00× — and was smallest on DeepSeek-V3.1, whose even routing left little to reuse. See [expert scattering](./concepts/expert-scattering-moe-decoding.md). - **The first number on the cost of specialising a model: most of it is waste.** Getting a good LoRA adapter (the cheap add-on used to fine-tune a model for one job) means trying many settings, because quality swings widely between them. ALTO runs the whole search at once on shared GPUs, kills losing runs early and gives each GPU its own adapters so none sits idle; it finished a 64-adapter search on Llama-3.1-70B up to 13.8× faster than a leading multi-LoRA system, with the best adapter matching expert-recommended settings. The paper puts that multi-GPU gain on idle GPUs under the usual way of splitting small jobs, and in a separate 8-GPU test found early exit the largest single saving ([ALTO](../raw/alto-adaptive-lora-tuning-orchestration.md)). See [LoRA fine-tuning cost](./concepts/lora-fine-tuning-cost.md) — the first page on the rung's specialization shelf. ## What we do not know yet - **Nothing on this rung was measured here.** Every figure above is read, not run. No price was sampled by us, no benchmark re-executed, no model held locally — and the rung's whole point is cost and capability per token, which is measurable with a script and a bill. - **Four of the sixteen sources could not be fetched from the primary.** DeepSeek V4 (HTTP 429), GPT-5.6 (OpenAI's page returned 403), the first Kimi K3 entry (the vendor blog served a placeholder) and the token-pricing paper (abstract only) are all summary-derived. Kimi K3's architecture numbers now come from Moonshot's own technical report, read at abstract level. The three model releases therefore reach us as vendor claims relayed by journalism, and GPT-5.6's headline "54% more token-efficient on agentic coding" arrives with no methodology attached ([GPT-5.6](../raw/gpt-5-6-tiering-token-efficiency.md)). - **The 31.5× reasoning premium is a cross-sectional average with no breakdown.** The paper does not, per the summary available, split it by lab or by model generation, and its production-frontier estimates are sensitive to which models and providers are in the list — closed enterprise pricing is likely under-covered ([Tiered Super-Moore's Law](../raw/tiered-super-moores-law-token-pricing.md)). - **Two vendor announcements on this rung carry no independently verified numbers at all.** Google's Gemini 3 post states multimodal leadership and improved tool use with independent benchmarks pending and cost/latency trade-offs undisclosed ([Gemini 3](../raw/gemini-3-google-2026.md)), and Meta's Muse Spark describes a parallel sub-agent "Contemplating" mode with no published independent benchmarks and an acknowledged compute cost ([Meta Muse Spark](../raw/meta-muse-spark-2026.md)). Neither moves a number here yet. - **Interpretability has no ground truth to check itself against.** The survey lists the problem plainly: no way to verify an interpretability hypothesis, superposition and polysemanticity frustrating feature-level accounts, patching experiments that scale badly with model size, and open doubt about whether any of it transfers to frontier systems ([Mechanistic Interpretability for LLM Alignment](../raw/mechanistic-interpretability-llm-alignment-survey.md)). The [circuit tracing](./concepts/circuit-tracing.md) work it draws on is also almost entirely Anthropic's own ([Transformer Circuits](../raw/anthropic-circuit-tracing.md)), so the field's main evidence base is one lab's. - **The rung's own scope has holes.** Its stated scope names quantization, distillation and fine-tuning; as of 2026-09-24 fine-tuning has one source (ALTO, a training-systems paper, not a measurement of a specialised model beating a general one) and quantization and distillation have none — nor is there a single measurement of running a model on your own compute, which is half the specific-and-local cluster the rung claims. ## Read next - [tiered-super-moores-law-token-pricing](../raw/tiered-super-moores-law-token-pricing.md) — the price series and the tier decomposition the whole rung hangs on. - [deepseek-v4-2026](../raw/deepseek-v4-2026.md) — the clearest case of architecture, not hardware, buying a sixfold price cut. - [llm-reasoning-is-latent-arxiv-2026](../raw/llm-reasoning-is-latent-arxiv-2026.md) — the claim that undercuts reading a model's stated reasoning. ### Sources (16) - Mechanistic Interpretability — 10 Breakthrough Technologies 2026 — MIT Technology Review · 2026-01-12 — https://www.technologyreview.com/2026/01/12/1130003/mechanistic-interpretability-ai-research-models-2026-breakthrough-technologies/ - Anthropic Transformer Circuits Thread & Circuit Tracing — Anthropic Interpretability Team (Elhage, Olsson, Bricken, Templeton, Ameisen, Lindsey et al.) · 2025-03-27 — https://transformer-circuits.pub/ - Mechanistic Interpretability for Large Language Model Alignment: Progress, Challenges, and Future Directions — Usman Naseem · 2026-02-01 — https://arxiv.org/abs/2602.11180 - Meta Muse Spark — Native Multimodal Foundation Model with Contemplating Mode — Meta AI / industry coverage · 2026-04-08 — https://www.aicerts.ai/news/meta-muse-spark-new-era-of-multimodal-ai-reasoning/ - Gemini 3 — Google's Latest Multimodal + Agentic Foundation Model — Google · 2026-04-15 — https://blog.google/products-and-platforms/products/gemini/gemini-3/ - LLM Reasoning Is Latent, Not the Chain of Thought — arXiv preprint authors · 2026-04-23 — https://arxiv.org/abs/2604.15726 - DeepSeek V4 — Open-Weight Trillion-Parameter MoE at ~1/6th Frontier Cost — VentureBeat (corroborated by morphllm.com + Hugging Face model card) · 2026-04-24 — https://venturebeat.com/technology/deepseek-v4-arrives-with-near-state-of-the-art-intelligence-at-1-6th-the-cost-of-opus-4-7-gpt-5-5 - Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models — Xin Cheng, Wangding Zeng, Damai Dai, et al. · 2026-01-12 — https://arxiv.org/abs/2601.07372 - LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation — Ahmadreza Jeddi, Marco Ciccone, Babak Taati · 2026-02-11 — https://arxiv.org/abs/2602.11451 - Tiered Super-Moore's Law: Price Evolution, Production Frontiers, and Market Competition in LLM Inference Services — Mingdeng Du · 2026-03-30 — https://arxiv.org/pdf/2603.28576 - Kimi K3 — Moonshot AI's 2.8T-Parameter Open-Weight Model — Simon Willison (independent analysis; corroborated by Bloomberg/Fortune/Axios/CNBC/Forbes/Tom's Hardware coverage of Moonshot AI's announcement) · 2026-07-16 — https://simonwillison.net/2026/Jul/16/kimi-k3/ - GPT-5.6 — OpenAI's Three-Tier (Sol / Terra / Luna) Frontier Family + Token-Efficiency Claims — Tech Startups (corroborated by CryptoBriefing; OpenAI's own page returned HTTP 403) · 2026-07-09 — https://techstartups.com/2026/07/09/openais-gpt-5-6-is-54-more-token-efficient-on-agentic-coding-than-rival-ai-models-sam-altman-says/ - Anthropic Claude Platform Release Notes — June 25 to July 15, 2026 — Anthropic · 2026-07-15 — https://platform.claude.com/docs/en/release-notes/overview - Kimi K3: Open Frontier Intelligence — Kimi Team (400+ authors) · 2026-07-27 — https://arxiv.org/abs/2607.24653 - Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts — Jincheng Xie, Runheng Liu, Heyan Huang, et al. · 2026-07-14 — https://arxiv.org/abs/2607.12696 - ALTO: Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloads — Jingwei Zuo, Xinze Feng, Zien Liu, et al. · 2026-04-07 — https://arxiv.org/abs/2604.05426 ## Evaluation & Measurement Canonical URL: https://menfem.com/kb/evals How you know it works — and whether the measurement can be trusted. --- title: "Evaluation & Measurement — Knowledge Base" last_compiled: "2026-09-24T12:30:00Z" source_count: 7 concept_count: 3 entity_count: 2 papers: 3 news: 0 --- # Evaluation & Measurement This rung is about how you know a thing works, and whether the measurement can be trusted. Its scope is benchmark design and the critique of it — matched-budget controls, contamination and overfitting to the eval set, agent benchmarks, and the gap between a benchmark score and what happens in deployment. The findings themselves belong to whichever rung they move; what belongs here is the question of whether those findings were measured properly. ## What this rung prices A buyer pays for a finished task, not for tokens, so every claim on the path from a token price to a task price rests on a benchmark number holding up. This rung is where that number is checked, and the check has one hard result: capability is not a scalar. Across up to 12 frontier models and 7 benchmarks — software engineering, mathematics, medicine, cybersecurity — run under controlled inference-scaling interventions, evaluations that report a single restrictive budget **understate** what a model can do, and understate newer models more, because a newer model's higher ceiling only opens at a generous budget ([How Inference Compute Shapes Frontier LLM Evaluation](../raw/how-inference-compute-shapes-frontier-llm-evaluation.md); that close read reached abstract level only, so this rests on the abstract and the per-benchmark numbers were never captured). If a score moves with the budget, then score and spend are the same axis, and a leaderboard that publishes the first without the second cannot be converted into a cost per finished task at all. That is what this rung prices: it decides whether any other rung's number is a measurement or a marketing figure. ## The load-bearing findings - **A benchmark score without a stated compute budget is not a measurement.** The recommendation the inference-compute paper actually makes is to report capability *as a function of* inference-time compute rather than as a fixed score, and it names the three interventions it swept: expanded token budgets, context compaction, and repeated submission attempts. Repeated submissions helped broadly; the others were domain-dependent ([How Inference Compute Shapes Frontier LLM Evaluation](../raw/how-inference-compute-shapes-frontier-llm-evaluation.md), read at abstract level). - **Capability is best read as a doubling rate, not a score, and the rate is itself moving.** In the only multi-year independent series here — 30+ models tested 2022–2025 by a body that sells no model — apprentice-level cyber task success rose from about 9–10% in late 2023 to about 50%, and the length of cyber task a model can complete unassisted was doubling roughly every 8 months, with later internal data (February 2026) putting that at about every 4.7 months since late 2024 ([Frontier AI Trends Report (AISI)](../raw/aisi-frontier-ai-trends-report-2025.md)). - **The same series shows how far a controlled setting can carry a claim, and where it stops.** Self-replication success across five frontier models rose from under 5% in early 2023 to over 60% by mid-2025, but models were strongest at the early stages (getting compute and money) and weakest at the later ones (replicating onto compute, holding persistent access) — so the assessment is that real-world autonomous replication is unlikely today and the gains live in simplified environments ([Frontier AI Trends Report (AISI)](../raw/aisi-frontier-ai-trends-report-2025.md)). A number that only exists inside the harness that produced it is the exact failure mode this rung exists to catch. - **Safeguard measurements move in both directions at once.** The same report records a 40× increase in expert effort needed to find biological-misuse jailbreaks between two models released six months apart, and universal jailbreaks found in every system tested ([Frontier AI Trends Report (AISI)](../raw/aisi-frontier-ai-trends-report-2025.md)). One metric improving by a large factor while the binary "can it be broken at all" stays at yes is a reminder that picking the metric is most of the argument. - **Benchmark design work outside the stack still demonstrates the method.** Two video-generation benchmarks sit on this rung: one evaluating 12,600 generated videos for physical realism and finding systematic failures on rigid-body collisions, fluid dynamics and gravity ([PhyWorldBench](../raw/phyworldbench-physics-benchmark.md)), and one scoring Sora-2 at about 64% and Veo-3 at about 58.7% on its Phenomenon Congruency metric ([VideoScience-Bench](../raw/videoscience-bench.md)). They are honest benchmark construction, but they measure text-to-video physics, not the inference stack this Atlas prices — they were ingested before the rung was rescoped, and they are counted here for accuracy, not for coverage. - **Measuring has become a cost line of its own.** One agent leaderboard, nine models on nine benchmarks, each run once, cost about $40,000; a single GAIA run can cost $2,829; and the same tasks cost up to 33× more or less depending only on the agent set-up around the model ([AI evals are becoming the new compute bottleneck](../raw/eval-costs-compute-bottleneck-evaleval.md)). Spending more did not reliably buy a better score — $1,577 bought 40% on one web benchmark where $171 bought 42%. A buyer reading a leaderboard without a cost column is reading half a measurement. See [the cost of evaluation](./concepts/cost-of-evaluation.md). - **A single run is not a reading, and a trustworthy one costs about 8× more.** Re-running one LLM-judged tool-calling benchmark 23 times on the same set-up gave scores from 57.9% to 76.8% ([Benchmarking the Benchmarks](../raw/benchmarking-the-benchmarks-tool-calling-validity-audit.md)), and re-running every cell of that $40,000 leaderboard 8 times would take it to roughly $320,000 ([EvalEval](../raw/eval-costs-compute-bottleneck-evaleval.md)). The shortcuts that made old benchmarks cheap — cutting them 100–200× without changing the ranking — save only 2–3.5× on agent benchmarks. - **The grader is wrong often enough to matter.** Expert review of 496 tool-calling tasks found the benchmarks' own graders disagreeing with the experts on 92 — 18.5% ([Benchmarking the Benchmarks](../raw/benchmarking-the-benchmarks-tool-calling-validity-audit.md)). Separately, a thesis fitted to 4,395 models on the Open LLM Leaderboard argues that a score can end up measuring model size rather than the ability it names, and that correcting for both measurement error and scale fits the data better ([Quantifying construct validity](../raw/quantifying-construct-validity-in-llm-evaluations.md)). See [benchmark validity](./concepts/benchmark-validity.md). The standing that all of this converts into is set out on [agent evaluation benchmarks](./concepts/agent-evaluation-benchmarks.md): a result is only interpretable if it discloses the harness, the search and inference budget, and held-out generalisation. ## What we do not know yet - **The rung is thinner than its file count.** Seven sources, two of which are about video generation; on the rung's own through-line the working count is five. Two things named in scope still have no coverage: contamination and overfitting to the eval set, and the eval-to-deployment gap. The cost figures come from one advocacy group's blog post compiling other people's numbers, not from a study of its own. - **The rung owns a shape but not a number.** The inference-compute study establishes the direction — scores rise with budget, and the gap widens for newer models — but its close read explicitly records that the per-benchmark numeric grid was not captured. So how far a score actually moves across a budget sweep is not held here, and capturing that grid from the source already ingested is the cheapest available improvement ([How Inference Compute Shapes Frontier LLM Evaluation](../raw/how-inference-compute-shapes-frontier-llm-evaluation.md)). - **Whether matched-budget control changes published rankings.** The argument that it should is strong and general, but nobody here has re-run a model leaderboard under a single fixed budget, and no source on this rung reports that having been done. - **Whether benchmark versions are comparable to each other.** Nothing on this rung publishes a version-to-version calibration, so a score quoted without its benchmark version cannot be placed against a score quoted with a different one. - **Nobody here has run any of it.** Every number above is read, not measured. That matters more on this rung than on any other, because the rung's whole claim is that other people's numbers should not be taken at face value — and it is currently making that claim entirely on other people's numbers. ## Concepts | Concept | Sources | Evidence | Last Updated | |---------|---------|----------|-------------| | [The Cost of Evaluation](concepts/cost-of-evaluation.md) | 3 (1 paper) | Moderate | 2026-09-24 | | [Benchmark Validity](concepts/benchmark-validity.md) | 3 (0 papers) | Moderate to strong | 2026-09-24 | | [Agent Evaluation Benchmarks](concepts/agent-evaluation-benchmarks.md) | 8 | Moderate to strong | 2026-07-23 | ## Entities | Entity | Type | Sources | |--------|------|---------| | [UK AI Security Institute (AISI)](entities/aisi.md) | Organization | 1 | | [Holistic Agent Leaderboard (HAL)](entities/holistic-agent-leaderboard.md) | Organization | 1 | ## Timeline See [timeline.md](timeline.md) for chronological developments, and [frontier.md](frontier.md) for the open questions. ## Sources | # | Title | Type | Date | Status | |---|-------|------|------|--------| | 1 | [PhyWorldBench](../raw/phyworldbench-physics-benchmark.md) | paper | 2025-07-17 | compiled | | 2 | [VideoScience-Bench](../raw/videoscience-bench.md) | paper | 2025-12-03 | compiled | | 3 | [Frontier AI Trends Report (AISI)](../raw/aisi-frontier-ai-trends-report-2025.md) | technical-report | 2025-12-18 | compiled | | 4 | [Quantifying construct validity in LLM evaluations](../raw/quantifying-construct-validity-in-llm-evaluations.md) | preprint | 2026-02-17 | compiled | | 5 | [AI evals are becoming the new compute bottleneck](../raw/eval-costs-compute-bottleneck-evaleval.md) | analysis | 2026-04-29 | compiled | | 6 | [How Inference Compute Shapes Frontier LLM Evaluation](../raw/how-inference-compute-shapes-frontier-llm-evaluation.md) | paper | 2026-06-16 | compiled | | 7 | [Benchmarking the Benchmarks](../raw/benchmarking-the-benchmarks-tool-calling-validity-audit.md) | preprint | 2026-06-30 | compiled | ## Read next - [eval-costs-compute-bottleneck-evaleval](../raw/eval-costs-compute-bottleneck-evaleval.md) — what a score costs to produce, and why a trustworthy one costs about 8× more. - [benchmarking-the-benchmarks-tool-calling-validity-audit](../raw/benchmarking-the-benchmarks-tool-calling-validity-audit.md) — the grader measured against experts, and the 18.9-point repeat-run spread. - [how-inference-compute-shapes-frontier-llm-evaluation](../raw/how-inference-compute-shapes-frontier-llm-evaluation.md) — the rung's central claim: a score without a budget is not a measurement. - [aisi-frontier-ai-trends-report-2025](../raw/aisi-frontier-ai-trends-report-2025.md) — the only multi-year independent capability series here, and the rung's most credible instrument. - [phyworldbench-physics-benchmark](../raw/phyworldbench-physics-benchmark.md) — benchmark construction done well, on a subject off this rung's through-line; useful as method, not as coverage. ### Sources (7) - PhyWorldBench: A Comprehensive Evaluation of Physical Realism in Text-to-Video Models — Multiple authors · 2025-07-17 — https://arxiv.org/abs/2507.13428 - VideoScience-Bench: Benchmarking Scientific Understanding and Reasoning for Video Generation — Multiple authors · 2025-12-03 — https://arxiv.org/abs/2512.02942 - Frontier AI Trends Report (AISI) — UK AI Security Institute (AISI) · 2025-12-18 — https://www.aisi.gov.uk/frontier-ai-trends-report - How Inference Compute Shapes Frontier LLM Evaluation — Jessica McFadyen, Ole Jorgensen, Harry Coppock, Kevin Wei, Cozmin Ududec · 2026-06-16 — https://arxiv.org/abs/2606.17930 - Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation — Vishvesh Bhat, Jay Vaghasiya, Muhammad Ahmed Mohsin, Asad Aali · 2026-06-30 — https://arxiv.org/abs/2607.02577 - Quantifying construct validity in large language model evaluations — Ryan Othniel Kearns · 2026-02-17 — https://arxiv.org/abs/2602.15532 - AI evals are becoming the new compute bottleneck — Avijit Ghosh, Yifan Mai, Georgia Channing, Leshem Choshen · 2026-04-29 — https://huggingface.co/blog/evaleval/eval-costs-bottleneck ## Data Canonical URL: https://menfem.com/kb/data What the model is trained and grounded on, and what that constrains. --- title: "Data — Knowledge Base" last_compiled: "2026-09-24T12:30:00Z" source_count: 3 concept_count: 3 entity_count: 4 papers: 0 news: 0 --- # Data This rung is about what a model is trained on and what that costs or constrains: how much usable text exists, whether synthetic data can stand in for it, who owns it, what licensing it costs, and how data quality changes what a model can do per token. It is the very start of the chain that ends in a task price — the input paid for before a single token is served. **Read this first: the rung holds three sources.** It was first compiled on 2026-09-24 from one analysis piece (the Authors Guild's summary of Bartz v. Anthropic) and extended the same day with Reddit's 2024 IPO filing and Epoch AI's forecast of the public data stock. That gives it two prices for training data — one set in court, one negotiated — and one supply estimate. It is still not a picture of the training-data market: each price was set by essentially one buyer. No live market call rests on this rung. ## What this rung prices **The numbers this rung owns — two prices, not yet one market:** - **About $3,000 per book**, a penalty: the minimum per-title payment from a $1.5 billion settlement over 7 million pirated book copies, covering roughly 500,000 titles (preliminary approval 25 September 2025) ([Authors Guild summary](../raw/bartz-v-anthropic-15bn-class-settlement.md)). A ceiling on what a licence would cost, not a market rate. - **$203.0 million over two to three years**, a licence: the aggregate value of the data deals Reddit signed in January 2024, at least $66.4 million of it recognised in 2024 — roughly $68–102 million a year across all contracts (this KB's arithmetic), from substantially one buyer ([Reddit S-1](../raw/reddit-s1-2024-data-licensing-contracts.md)). **Dated February 2024 — never cite it as current.** See [the price of training data](concepts/training-data-acquisition-cost.md). **The binding constraint, as far as three sources show it:** *who* can be paid, not *how much* exists — yet. Free public text still has headroom: Epoch AI puts the effective stock at ~300 trillion tokens, fully used **between 2026 and 2032** (central 2028), and ranks data behind power and chips as a limit on scaling to 2030 ([Epoch AI](../raw/epoch-can-ai-scaling-continue-through-2030.md)). Meanwhile the paid side is shaped by provenance: the court attached money to *how* books were obtained, not to training on them ([acquisition versus training](concepts/acquisition-versus-training.md)). The supply date has already moved once by four years, so this reading is provisional. See [the data stock](concepts/data-stock-exhaustion.md). **How it reaches the token price, and where the chain breaks.** A data cost is paid once and spread over every token a model serves. Turning either price into a cost per token needs two numbers this rung does not hold: how many tokens the purchased data amounts to, and how many served tokens the cost is spread across. The supply estimate adds a third link — when free data runs out, the next model's training bill rises (licensed data, or synthetic data at roughly twice the training compute) — but still no conversion. **Why it sits here and not next door.** `models` would claim anything about what training produces; these sources say what the inputs cost and how much of them exists. `inference-economics` prices tokens at serving time; these are costs incurred before serving begins. Epoch's power and chip findings belong to `energy` and `hardware`; only its data section is compiled here. ## The load-bearing findings - **~$3,000 per title is a penalty ceiling, not a going rate** — and the unit matters: the same $1.5bn is ~$214 per *copy* downloaded, a figure worked out only to show the sensitivity and not to be cited. [The price of training data](concepts/training-data-acquisition-cost.md). - **A licence is priced per year of access, not per work.** Reddit's $203.0M buys an API plus quarterly bulk transfers; its 2024 minimum alone was over four times all of Reddit's "other revenue" in 2023. [The price of training data](concepts/training-data-acquisition-cost.md). - **The settlement priced acquisition, not training.** Most coverage collapsed the two. [Acquisition versus training](concepts/acquisition-versus-training.md). - **"We run out of data in 2026" is a misquote.** Epoch's 2022 estimate said high-quality text would be used up *before* 2026; its 2024 estimate moved the window to 2026–2032, central 2028. [The data stock](concepts/data-stock-exhaustion.md). ## Frontier — What's Moving Now - The live licence price: Reddit's January 2024 contracts reach the end of their terms between January 2026 and January 2027, and its 2026 filings no longer break licensing out. - The data window has opened (2026–2032); whether Epoch has moved it again is not in this KB. - The final-approval outcome of the Bartz settlement is unresolved in this KB. - See [frontier.md](frontier.md) for the open questions and the checks behind them. ## Concept Map ```mermaid graph LR A[Acquisition versus training] --> B[The price of training data] C[The data stock and when it runs out] --> B ``` ## Concepts | Concept | Sources | Evidence | Last Updated | |---------|---------|----------|-------------| | [The Price of Training Data](concepts/training-data-acquisition-cost.md) | 2 (1 analysis, 1 technical report) | Moderate; each price from one buyer | 2026-09-24 | | [Acquisition Versus Training](concepts/acquisition-versus-training.md) | 1 (analysis) | Moderate to weak, single source | 2026-09-24 | | [The Data Stock and When It Runs Out](concepts/data-stock-exhaustion.md) | 1 (technical report) | Moderate, single forecaster | 2026-09-24 | ## Entities | Entity | Type | Sources | |--------|------|---------| | [Anthropic](entities/anthropic.md) | Lab | 1 | | [The Authors Guild](entities/authors-guild.md) | Organization | 1 | | [Reddit](entities/reddit.md) | Company | 1 | | [Epoch AI](entities/epoch-ai.md) | Organization | 1 | ## Timeline See [timeline.md](timeline.md) for chronological developments. ## Sources | # | Title | Type | Date | Status | |---|-------|------|------|--------| | 1 | [What Authors Need to Know About the Anthropic Settlement (Bartz v. Anthropic, $1.5bn class settlement)](../raw/bartz-v-anthropic-15bn-class-settlement.md) | analysis | 2025-09-25 | compiled | | 2 | [Reddit, Inc. — Form S-1 Registration Statement (data licensing arrangements, January 2024)](../raw/reddit-s1-2024-data-licensing-contracts.md) | technical-report | 2024-02-22 | compiled | | 3 | [Can AI scaling continue through 2030? (Epoch AI)](../raw/epoch-can-ai-scaling-continue-through-2030.md) | technical-report | 2024-08-20 | compiled | ## Standing *Read it*, for all three. Nothing on this rung was done first-hand. A piece built on it is a coverage piece unless it teaches one of the three distinctions the sources make clear: acquisition versus training, a licence versus a settlement, and what "running out of data" actually means. ### Sources (3) - What Authors Need to Know About the Anthropic Settlement (Bartz v. Anthropic, $1.5bn class settlement) — The Authors Guild · 2025-09-25 — https://authorsguild.org/advocacy/artificial-intelligence/what-authors-need-to-know-about-the-anthropic-settlement/ - Reddit, Inc. — Form S-1 Registration Statement (data licensing arrangements, January 2024) — Reddit, Inc. · 2024-02-22 — https://www.sec.gov/Archives/edgar/data/1713445/000162828024006294/reddits-1q423.htm - Can AI scaling continue through 2030? (Epoch AI) — Jaime Sevilla, Tamay Besiroglu, Ben Cottier, Josh You, Edu Roldán, Pablo Villalobos, Ege Erdil · 2024-08-20 — https://epoch.ai/publications/can-ai-scaling-continue-through-2030 ## Agent Security Canonical URL: https://menfem.com/kb/security How an agent gets owned, and what bounds the damage. --- title: "Agent Security — Knowledge Base" last_compiled: "2026-09-24T12:30:00Z" source_count: 4 concept_count: 4 entity_count: 2 papers: 0 news: 0 --- # Agent Security This rung asks how an AI agent gets taken over, and what limits the damage when it does: the ways in (prompt injection, reframing the agent's goal, poisoning its memory or its tools), how far an attacker can climb once inside, whether a tool is what it says it is, and what evidence exists from agents running in the real world. Alignment research as a subject in itself belongs to `models`, and how a harness is built belongs to `harnesses`. This rung is the attack and the blast radius. ## What this rung prices A buyer pays for a finished task. A task an agent finished for an attacker is not finished, and the defences that stop that are not free. The rung's sources price both sides. A protocol-level fix for the Model Context Protocol (MCP) cut attack success from 52.8% to 12.4% at **8.3 ms of added latency per message** ([Breaking the Protocol](../raw/breaking-the-protocol-mcp-security-analysis.md)). Against a tool-description attack, the prevention methods that worked cut clean task success to **0.08–0.15** — the agent was safe because it no longer did the job ([ContextLeak](../raw/contextleak-exfiltrating-agent-context-via-malicious-tools.md)). Both are a cost per task that no token price shows: extra time on every call, or tasks that stop completing. ## The load-bearing findings - **The weaknesses are in the MCP specification, not in one server.** A formal analysis names three: a client cannot check that a server's tools are what they claim, a server can ask the model for completions without proving where the request came from, and trust extended to one server in a session spreads to the others. Across 847 attack scenarios on five MCP implementations, wiring tools through MCP raised attack success by **23–41%** over equivalent non-MCP integrations ([Breaking the Protocol](../raw/breaking-the-protocol-mcp-security-analysis.md)). These figures are read from the abstract page only; the breakdown behind them was not extracted, and the fix is the authors' own, tested on their own harness. - **A tool's name and description are enough to steal an agent's context.** ContextLeak trains an attacking model to write the name and description of a malicious tool so that the agent both picks it and passes its own context in as the input. In simulation it was selected 92% of the time for user prompts, with the prompt copied almost exactly (edit-distance score 0.99, against 0.74 for the best earlier attack), 89% for conversation history and 86% for the tool list ([ContextLeak](../raw/contextleak-exfiltrating-agent-context-via-malicious-tools.md)). - **The detectors did not fire, and the real-world rate is much lower than the lab rate.** Three detectors (PromptGuard, DataSentinel, MCPScan) missed the attack 99–100% of the time. Against Claude Code on Claude Sonnet 4.6, 100 test cases, the tool was picked **22%** of the time, not 92% ([ContextLeak](../raw/contextleak-exfiltrating-agent-context-via-malicious-tools.md)). The lab figure should never be quoted as a real-world rate. - **A repository's own settings were enough to take over a developer's machine.** Check Point Research showed three ways a project's Claude Code config could act before the user trusted it: hooks that ran after one click, MCP servers switched on by the repo's own settings so their commands ran before the trust prompt could be read (CVE-2025-59536, fixed in 1.0.111), and a redirected API address that sent the user's API key to the attacker (CVE-2026-21852, fixed in 2.0.65). A stolen key then reached every file the team had stored in its Workspace. All were fixed before publication on 2026-02-25 ([Caught in the Hook](../raw/claude-code-project-files-rce-api-key-exfiltration.md)). It is the only real, reproduced incident on this rung, in a tool this project runs every day. See [project config as an attack path](./concepts/project-config-trust-boundary.md). - **One framing did most of the work in a 10,000-trial study.** The carried-over [attack surface](./concepts/agent-exploitation-attack-surface.md) page records that of twelve kinds of prompt pressure tested, only reframing the task as a puzzle or capture-the-flag game reliably got agents to exploit a planted vulnerability — 38–40% on Claude Sonnet 4 even with explicit rules present — while nine showed no detectable effect. - **Agents that remember can be poisoned for later.** The carried-over [deployed agent safety](./concepts/deployed-agent-safety.md) page records a study of a live personal agent ([OpenClaw](./entities/openclaw.md)) with real email, payment and file access: poisoning any one of its tools, its guidelines or its memory raised attack success from 24.6% to 64–74% across every model tested, and a file-protection defence that blocked 97% of malicious writes also blocked 93% of legitimate ones. ## Concept map ```mermaid graph LR A[MCP protocol defects] --> B[Malicious tool descriptions] A --> H[Repo config acts before trust] H --> C B --> C[Context exfiltration] D[Goal reframing] --> E[Exploitation] F[Memory and tool poisoning] --> E E --> G[Deployed agent safety] ``` ## Concepts | Concept | Sources | Evidence | Last Updated | |---------|---------|----------|-------------| | [Agent Exploitation Attack Surface](concepts/agent-exploitation-attack-surface.md) | 1 (0 papers) | Strong as recorded | 2026-04-14 | | [Project Config as an Attack Path](concepts/project-config-trust-boundary.md) | 2 (0 papers) | Strong | 2026-09-24 | | [Agent Safety & Alignment](concepts/agent-safety-alignment.md) | 5 (0 papers) | Moderate to strong | 2026-06-24 | | [Deployed Agent Safety](concepts/deployed-agent-safety.md) | 1 (0 papers) | Strong as recorded | 2026-04-14 | ## Entities | Entity | Type | Sources | |--------|------|---------| | [Claude Code](entities/claude-code.md) | Product | 2 | | [OpenClaw](entities/openclaw.md) | Product | 1 | These pages were compiled before this rung existed and cite sources held on `harnesses` and `evals`. None yet folds in the two MCP sources. Much of [Agent Safety & Alignment](concepts/agent-safety-alignment.md) — the preference-training methods and the alignment trilemma — is model-level alignment research, which the rung's scope places on `models`. ## What we do not know yet - **How MCP has changed since January 2026.** The protocol analysis is dated 2026-01-24 and the specification it read may not be the one shipping now. - **Which of the three MCP defects drives the 23–41%.** Not extracted from the abstract. - **A defence that works without breaking the agent.** MCPSec leaves 12.4% success; the prevention methods tested against ContextLeak wreck task success; the detectors miss it. - **Independent reproduction.** Every lab figure is the authors measuring their own attack or their own fix. The one exception is the Claude Code config flaws, which the affected vendor confirmed by publishing advisories. - **Permission escalation and red-teaming method as their own subjects.** Named in scope, with no source of their own on this rung. - **Standing: did it on the posture, read it on the attacks.** MenFem runs an MCP server, so the protocol defects can be checked against a surface operated here, and it runs Claude Code daily with a committed settings file that defines hooks — the exact surface of the config attacks. None of the attacks has been run here, and a piece must not claim to have run ContextLeak or reproduced the Check Point chain. ## Timeline See [timeline.md](timeline.md) for chronological developments, and [frontier.md](frontier.md) for the open questions. ## Sources | # | Title | Type | Date | Status | |---|-------|------|------|--------| | 1 | [Breaking the Protocol: Security Analysis of MCP](../raw/breaking-the-protocol-mcp-security-analysis.md) | preprint | 2026-01-24 | compiled | | 2 | [AI Safety, Alignment, and Interpretability in 2026](../raw/ai-safety-alignment-interpretability-2026.md) | analysis | 2026-02-09 | compiled | | 3 | [Caught in the Hook: RCE and API Token Exfiltration Through Claude Code Project Files](../raw/claude-code-project-files-rce-api-key-exfiltration.md) | technical-report | 2026-02-25 | compiled | | 4 | [ContextLeak: Exfiltrating LLM Agent Context via Malicious Tools](../raw/contextleak-exfiltrating-agent-context-via-malicious-tools.md) | preprint | 2026-08-28 | compiled | ## Read next - [breaking-the-protocol-mcp-security-analysis](../raw/breaking-the-protocol-mcp-security-analysis.md) — the design-level claim, and the latency price of the fix. - [contextleak-exfiltrating-agent-context-via-malicious-tools](../raw/contextleak-exfiltrating-agent-context-via-malicious-tools.md) — the empirical partner: a string in a tool manifest is the attack. - [claude-code-project-files-rce-api-key-exfiltration](../raw/claude-code-project-files-rce-api-key-exfiltration.md) — the real incident: three patched CVEs in repo config, with versions and a disclosure timeline. - [Deployed Agent Safety](concepts/deployed-agent-safety.md) — what poisoning looks like on an agent with real integrations. ### Sources (4) - AI Safety, Alignment, and Interpretability in 2026 — Zylos Research · 2026-02-09 — https://zylos.ai/research/2026-02-09-ai-safety-alignment-interpretability - Breaking the Protocol: Security Analysis of the Model Context Protocol Specification and Prompt Injection Vulnerabilities in Tool-Integrated LLM Agents — Narek Maloyan, Dmitry Namiot · 2026-01-24 — https://arxiv.org/abs/2601.17549 - ContextLeak: Exfiltrating LLM Agent Context via Malicious Tools — Yuqi Jia, Ruiqi Wang, Patrick Li, Yuepeng Hu, Peinian Li, Neil Gong · 2026-08-28 — https://arxiv.org/abs/2608.27800 - Caught in the Hook: RCE and API Token Exfiltration Through Claude Code Project Files (CVE-2025-59536, CVE-2026-21852) — Aviv Donenfeld, Oded Vanunu · 2026-02-25 — https://research.checkpoint.com/2026/rce-and-api-token-exfiltration-through-claude-code-project-files-cve-2025-59536/ ## Compilers & Kernels Canonical URL: https://menfem.com/kb/compilers The layer between a model graph and the silicon. --- title: "Compilers & Kernels — Knowledge Base" last_compiled: "2026-09-24T12:30:00Z" source_count: 5 concept_count: 5 entity_count: 4 papers: 3 news: 0 --- # Compilers & Kernels This rung is the layer between a model and the chip: the kernels — small, hand-tuned programs — that turn "do attention over this context" into instructions a specific GPU can run, and the tools used to write them. It is the least visible rung in the stack and one of the most direct. A kernel change moves how many tokens a GPU produces per second with nothing else changing: same chip, same model, same maths. That makes it the cleanest place to see a token get cheaper for a reason you can name. **The rung holds five sources**, compiled first on 2026-09-24 from three and extended the same day: one peer-reviewed conference paper (ReQAT, ICML 2026), two preprints (FlashAttention-4, and a batch-1 decode study across four GPUs), and two vendor write-ups — DeepSeek's FlashMLA, checked against its shipped code, and Meta's design note on automatic warp specialization in Triton, which is labelled as the feature team describing its own work. Every performance number here is the authors' own, and none has been independently reproduced in this KB. Standing across all five is *read it*. ## What this rung prices **The number this rung owns: 1.64× on decode throughput from a kernel change alone** — DeepSeek's FP8 sparse decoding kernel went from 250 to 410 TFLOPS on the same H800 GPU with the same model, algorithm and number format (as of 2025-09-29, vendor-measured, one configuration) ([FlashMLA FP8 deep dive](../raw/flashmla-fp8-sparse-decode-dsm-crossover.md)). Decode throughput is the bottom half of output-token cost: the same GPU-hour spread over 1.64× more tokens. How much of that reaches a price someone pays is set on the serving and economics rungs, not here — this rung supplies the multiplier, not the price. **The binding constraint:** the tensor cores are no longer the limit. On Blackwell, the work around the matrix maths — moving data through shared memory, computing exponentials — takes 25–60% longer than the maths itself ([FlashAttention-4](../raw/flashattention-4-blackwell-kernel-pipelining.md)). On Hopper, converting an FP8 cache back to BF16 costs at least 50 cycles per token against 34 cycles of matrix work ([FlashMLA](../raw/flashmla-fp8-sparse-decode-dsm-crossover.md)). Two teams, two chip generations, one conclusion: the fast part of the chip is waiting on the slow parts. See [the non-matmul bottleneck](concepts/non-matmul-bottleneck.md). **For one user at a time, the binding constraint is different again.** At batch 1 — the personal-AI case — an H100 running a 7B model reaches only ~27% of its memory-bandwidth limit against ~81% on a far cheaper L4, because the CPU's launching of ~280 small kernels per token dominates on the fast card; CUDA Graphs recovers 1.259× on H100 and 1.028× on L4 ([batch-1 decode study](../raw/memory-bound-not-bandwidth-limited-batch1-decode.md)). With the best kernel on each card, the H100 is only ~1.47× faster, and this KB's arithmetic at the paper's list rates puts the L4 at **~$1.45 per million tokens against ~$11.45** on the H100 (as of May 2026, single-stream, excluding idle). See [launch overhead at batch 1](concepts/launch-overhead-batch1-decode.md). **Why it sits here and not next door.** `hardware` would claim the FlashMLA result — the bottleneck exists only because H800 has no direct FP8→BF16 instruction — but that is a claim on the *next* chip; on the chip that shipped, the gain came from software. `serving` would claim it because FP8 caching is a serving decision, but serving owns the choice to *store fewer bytes*; this rung owns what happens when those bytes have to become numbers again fast enough to matter. ## The load-bearing findings - **The tensor core is not the binding constraint in attention kernels any more.** Moderate evidence, from two sources that agree on the shape of the problem while naming different culprits. [The non-matmul bottleneck](concepts/non-matmul-bottleneck.md). - **A newer chip ran the same kernel slower.** DeepSeek's sparse FP8 decode kernel: 410 TFLOPS on H800, up to 350 on B200, the B200 path self-described as "not really optimized yet" (as of 2025-10-01). Kernels are written for a generation; new silicon does not inherit old wins. [Kernel portability](concepts/kernel-portability.md). - **4-bit reasoning is possible on one model and not yet on another.** ReQAT puts weights, activations and the KV cache in FP4 and matches full-precision fine-tuning on R1-Qwen-14B (65.94% vs 65.46% on AIME-120), but trails it on R1-Llama-8B by 6.9 points. Its mechanism — rounding noise lands on the tokens the model is most sure of, like digits and operators — is tested directly. 3.9× throughput on DGX Spark, a machine a person can buy. [FP4 reasoning quantization](concepts/fp4-reasoning-quantization.md). - **For one user, a faster GPU is not a cheaper token.** Launch overhead caps the H100 at ~27% of its memory bandwidth at batch 1; on the cheaper L4 the 4-bit kernel choice alone spans 45.24 to 17.36 ms per token. [Launch overhead at batch 1](concepts/launch-overhead-batch1-decode.md). - **The portable compiler is catching up, on one kernel.** Triton's automatic warp specialization reaches 1.5–2× over stock Triton on B200 attention forward, 10–20% behind cuDNN (vendor-reported, no configurations published). [Kernel portability](concepts/kernel-portability.md). - **Compressing the cache is not the same as compressing the maths.** FP8 storage saves ~43% of cache bytes per token and then costs cycles to undo. [Quantized KV cache](concepts/quantized-kv-cache-dequantization.md). ## Frontier — What's Moving Now - Kernel design organised around whichever chip unit *did not* scale (exponential units, shared memory) rather than around peak FLOPs. - Native low-precision compute (FP4 on Blackwell) moving the problem from conversion cost to accuracy on the tokens that matter. - Python-embedded kernel languages (CuTe-DSL) cutting compile times 20–30×, making per-chip tuning cheaper; and compilers (Triton autoWS) doing the warp-level scheduling hand-written kernels do. - Single-stream decode, where launch overhead and the choice of quantisation kernel matter more than the GPU's memory bandwidth. - See [frontier.md](frontier.md) for open problems and gaps. ## Concept Map ```mermaid graph LR A[Non-matmul bottleneck] --> B[Quantized KV cache] A --> C[Kernel portability] B --> D[FP4 reasoning quantization] C --> A E[Launch overhead at batch 1] --> A E --> C ``` ## Concepts | Concept | Sources | Evidence | Last Updated | |---------|---------|----------|-------------| | [The Non-Matmul Bottleneck](concepts/non-matmul-bottleneck.md) | 3 (2 preprint, 1 technical report) | Moderate | 2026-09-24 | | [Quantized KV Cache: Stored Small, Computed Large](concepts/quantized-kv-cache-dequantization.md) | 2 (1 conference, 1 technical report) | Moderate | 2026-09-24 | | [FP4 Reasoning Quantization](concepts/fp4-reasoning-quantization.md) | 1 (conference) | Strong, single paper | 2026-09-24 | | [Kernel Portability Across Chip Generations](concepts/kernel-portability.md) | 4 (2 preprint, 2 technical report) | Weak to moderate | 2026-09-24 | | [Launch Overhead: Why a Faster GPU Doesn't Speed Up One User](concepts/launch-overhead-batch1-decode.md) | 1 (preprint) | Moderate, single preprint | 2026-09-24 | ## Entities | Entity | Type | Sources | |--------|------|---------| | [DeepSeek](entities/deepseek.md) | Lab | 1 | | [FlashAttention](entities/flashattention.md) | Product | 1 | | [NVIDIA](entities/nvidia.md) | Company | 5 | | [Triton](entities/triton.md) | Product | 2 | ## Timeline See [timeline.md](timeline.md) for chronological developments. ## Sources | # | Title | Type | Date | Status | |---|-------|------|------|--------| | 1 | [A Deep Dive Into The Flash MLA FP8 Decoding Kernel on Hopper](../raw/flashmla-fp8-sparse-decode-dsm-crossover.md) | technical-report | 2025-09-29 | compiled | | 2 | [FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling](../raw/flashattention-4-blackwell-kernel-pipelining.md) | preprint | 2026-03-05 | compiled | | 3 | [ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Training](../raw/reqat-fp4-quantization-aware-training-for-reasoning.md) | conference | 2026-06-14 | compiled | | 4 | [Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode](../raw/memory-bound-not-bandwidth-limited-batch1-decode.md) | preprint | 2026-05-28 | compiled | | 5 | [Warp Specialization in Triton: Design and Roadmap](../raw/triton-warp-specialization-design-roadmap.md) | technical-report | 2026-01-08 | compiled | ## Standing *Read it*, for all five. The FlashMLA close-read went further than most — the repository was cloned at a named commit, every arithmetic step recomputed, and the crossover mechanism verified in the shipped CUDA — but nothing was run. The batch-1 decode study is the most runnable source on the rung (a single consumer-class GPU, open tools) and the obvious first candidate to do first-hand. Until then anything built on this rung teaches at the level of *how it works*, not *how to do it*. No live market call rests on this rung. ### Sources (5) - A Deep Dive Into The Flash MLA FP8 Decoding Kernel on Hopper — Shengyu Liu (git author of both commits to this document); FlashMLA authors of record per the repository citation block: Jiashi Li, Shengyu Liu · 2025-09-29 — https://github.com/deepseek-ai/FlashMLA/blob/main/docs/20250929-hopper-fp8-sparse-deep-dive.md - FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling — Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, Tri Dao · 2026-03-05 — https://arxiv.org/abs/2603.05451 - ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Training — Janghwan Lee, Sihwa Lee, Jinseok Kim, Yongjik Kim, Jieun Lim, Jinwook Oh, Jungwook Choi · 2026-06-14 — https://arxiv.org/abs/2606.15682 - Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode — Josef Chen · 2026-05-28 — https://arxiv.org/abs/2605.30571 - Warp Specialization in Triton: Design and Roadmap — Manman Ren, Nick Riasanovsky, Neil Dhar, Hongtao Yu, Jie Liu, Partha Kanuparthy, Shane Nay · 2026-01-08 — https://pytorch.org/blog/warp-specialization-in-triton-design-and-roadmap/ ## Hardware — Memory & Compute Architecture Canonical URL: https://menfem.com/kb/hardware The silicon rung of the inference stack, read memory-first. --- title: "Hardware — Memory & Compute Architecture — Knowledge Base" last_compiled: "2026-09-24" source_count: 25 concept_count: 7 entity_count: 11 --- # Hardware — Memory & Compute Architecture The silicon rung of the inference stack, read memory-first. The scope is deliberate: the memory hierarchy, HBM and DRAM scaling, bandwidth and capacity limits, computing inside memory, accelerator architecture and the packaging that bolts memory to compute. Computing *in light* and quantum computing are out — frontier science with no near-term effect on what a token costs — and communicating in light belongs to the `networking` rung, not this one. The organising claim is Onur Mutlu's: modern machines spend between 60% and 90% of their total system energy moving data rather than computing on it, and a DRAM read costs roughly 800× a 64-bit floating-point operation and about 6,400× a 32-bit integer add ([Memory-Centric Computing](../raw/memory-centric-computing-mutlu.md)). On that arithmetic the memory is the machine and the processor is an accessory. ## What this rung prices A model is quoted per token; a buyer pays for a finished task; and in saturated decode the thing actually being bought in between is memory bandwidth. That is why this rung has a unit of its own — dollars per petabyte of bandwidth delivered — set out in closed form with every parameter stated (accelerator price, a 1.5× system multiplier, four-year straight-line depreciation, PUE 1.3, $0.08/kWh, and a memory-bandwidth utilisation figure of 0.50 to 0.62), so that two fleets of different vintages can be compared without reference to any particular model ([close read of Matsuoka 2607.07207](../raw/memory-scarcity-2607.07207-closeread.md)). Everything on the rung moves that number in one of three ways: how much bandwidth a stack delivers (HBM generation), how many stacks can be attached to a die at all (packaging), and how much of the energy budget is burned shifting bytes instead of using them (processing in memory). When memory gets scarce, the price of the hardware rises before the price of the token does — and the gap between what an incumbent's already-depreciated fleet costs to run and what a new buyer's fleet costs to run is the wedge that decides who can undercut whom. ## The load-bearing findings - **On the paper's own parameters a 2026 new build costs about 3.2× what a depreciated incumbent pays.** A GB300-class 2026 build comes out at roughly $0.174 per petabyte moved against a depreciated-H100 incumbent floor of about $0.054, narrowing to 1.9–2.0× in 2027 and re-widening to around 3× (base) or above 4× (shortage branch) by 2029–30 ([close read](../raw/memory-scarcity-2607.07207-closeread.md)). The mechanism — a depreciation conveyor, where the cost advantage rotates between incumbents by purchase year and never transfers to an entrant inside the horizon — is a structural argument rather than a forecast, and it is the strongest thing in the source. See [memory scarcity and inference economics](./concepts/memory-scarcity-inference-economics.md). - **Which purchase year is underwater depends on the pricing regime, and the two regimes cross over.** The breakeven grid is 16 cells — four purchase years × two pricing regimes × two HBM branches — read off a bar chart rather than a table. The 2027 cohort clears in all four of its cells (7.5–11.1% of tokens needing premium pricing); the 2026 cohort's worst cell is *sticky* pricing at about 31.3%, not the coupled regime; the 2029 cohort under coupled-plus-shortage needs about 37.6% ([close read](../raw/memory-scarcity-2607.07207-closeread.md)). The crossover is real arithmetic — one regime is pegged to a ratio and the other to an absolute — but the 10–20% band those numbers are judged against is asserted in the paper with no source, so the verdict is a function of an unsourced parameter. - **Bandwidth per accelerator is being bought by moving the memory's base die onto a logic process.** HBM4 doubles the interface to 2,048 bits and the channel count to 32 per stack, and the base die moves off DRAM processes onto TSMC 12FFC or N5 — about twice as power-efficient as HBM3E's DRAM-based base die — with HBM4E reaching 3 TB/s per stack against HBM3E's 1.2 TB/s ([HBM architectural shakeup](../raw/hbm4-c-hbm4e-architectural-shakeup.md)). NVIDIA's own material puts Rubin at 288 GB of HBM4 at 22 TB/s against Blackwell's 192 GB at 8 TB/s, a 2.8× bandwidth step ([Inside the Vera Rubin Platform](../raw/nvidia-vera-rubin-platform.md)) — vendor self-reported, and the platform's rack-scale framing is a sales argument as much as a measurement. See [HBM4 memory architecture](./concepts/hbm4-memory-architecture.md). - **HBM4 is a volume product now, and the next step is only at samples.** Micron says its HBM4 36GB 12-high stack has shipped in volume since the first quarter of 2026, at over 2.8 TB/s per stack and over 11 Gb/s per pin — 2.3× the bandwidth of its own HBM3E, with over 20% better power efficiency ([Micron HBM4](../raw/micron-hbm4-high-volume-production-2026.md)). Eight such stacks are the 288 GB and roughly 22 TB/s that NVIDIA quotes for a Rubin GPU, so the two vendors' figures agree. SK hynix's own results put its HBM4 mass shipments from Q2 2026, with a 76% operating margin that says buyers are not setting the price ([SK hynix 2Q26](../raw/sk-hynix-2q26-financial-results.md)). HBM4E exists only as **samples**: SK hynix's 12-layer parts run at up to 16 Gb/s per pin with 48 GB per stack and no production date ([HBM4E samples](../raw/sk-hynix-hbm4e-12-layer-samples-2026.md)). All of it is the makers' own claims; none of it states how many stacks can be made or what they cost. - **The binding constraint has been packaging, not fabrication, and it is loosening rather than tightening.** TSMC's CoWoS capacity went from roughly 35,000 wafers a month in late 2024 to about 75,000 by end-2025 with a 130,000 target for end-2026 ([TSMC to quadruple advanced packaging](../raw/tsmc-cowos-130k-wafers-late-2026.md)); four months later that target reads as a 120,000–140,000 band, with a further 50,000–60,000 from outsourced assemblers, and the supply-demand gap projected to narrow from about 20% to about 10% by end-2026 ([CoWoS gap narrowing](../raw/tsmc-cowos-supply-demand-gap-narrowing-2026.md)). Both are projections carrying explicit as-of dates, not audited output. NVIDIA has reserved the majority of that capacity ([NVIDIA snaps up packaging capacity](../raw/nvidia-tsmc-advanced-packaging-reservation-2026.md)), which is what makes the queue rather than the wafer the thing to watch — [advanced packaging and CoWoS](./concepts/advanced-packaging-cowos.md). - **The next lithography generation has been priced out, at least for now.** TSMC's deputy co-COO said publicly that the company has no current plans to adopt ASML's high-NA EUV tools, at upwards of €350 million each — roughly double the current EUV generation — and that it can produce its first ~1.3nm node without them, relying on process optimisation and GPU-accelerated mask synthesis instead ([TSMC delays high-NA EUV](../raw/tsmc-delays-high-na-euv-bloomberg-2026.md); [cuLitho collaboration](../raw/nvidia-asml-tsmc-synopsys-culitho-2026.md), which is a vendor announcement). The source itself notes the per-tool cost is reported anywhere between $350M and $410M, and that "no current plans" is a deferral rather than a refusal. - **Custom accelerators are growing faster than merchant GPUs without displacing them.** TrendForce forecasts 2026 AI server shipments up more than 28% year on year, with ASIC-based systems at 27.8% of shipments — the highest since 2023 — against 69.7% for GPU systems, and the top five North American cloud buyers raising capital expenditure about 40% ([TrendForce ASIC share](../raw/trendforce-asic-share-ai-server-2026.md)). These are forecasts, not realised shipments, and they are unit shares rather than revenue shares, so the GPU's revenue position sits higher than 69.7% implies. See [custom silicon vs GPU](./concepts/custom-silicon-vs-gpu.md). ## What we do not know yet - **Nothing on this rung was measured here.** Every figure above is read from someone else's document. The bandwidth numbers come from vendor material or analyst houses, the cost model comes from a single-author preprint, and no $/PB figure has been reproduced against a machine anyone here has run. - **The realised CoWoS number has never been checked against the target.** The KB carries a 130,000 target, then a 120,000–140,000 band, both projections; the [gap-narrowing source](../raw/tsmc-cowos-supply-demand-gap-narrowing-2026.md) says plainly that closing this needs TSMC's own quarterly management report, which the rung has for one quarter ([TSMC Q2 2026](../raw/tsmc-2q26-earnings-print.md)) but has not reconciled against the capacity claim. - **The most decision-relevant number in the memory model is a judgment, not a computation.** The revision of token-demand growth from 4–7× a year to 2–3× is stated immediately after a list of five biases, four of which are given no quantity at all; the author labels even that revised figure an upper bound ([close read](../raw/memory-scarcity-2607.07207-closeread.md)). The same close read finds a cited 73× spread in realised cost on identical H100 hardware, which if real would swamp the 2–4× vintage effect the whole model is built to discriminate. - **Processing in memory has no shipped economics on this rung.** The case for computing inside memory rests on energy ratios and laboratory demonstrations, and the source's own account of the barriers is economic rather than technical — a standards committee of roughly 390 companies and an incumbent industry with no reason to move ([Memory-Centric Computing](../raw/memory-centric-computing-mutlu.md)). That source is itself a synthesis of public talks and papers rather than a single fetched document. Near-memory compute appears in the HBM roadmap as a described capability with no worked cost example ([HBM shakeup](../raw/hbm4-c-hbm4e-architectural-shakeup.md)) — [processing in memory](./concepts/processing-in-memory.md). - **One memory supplier's picture reaches us with a hole in it.** The SK hynix outlook carries the share and capacity figures but records that the article's three named market headwinds were not captured from the search snippet and need re-ingesting ([SK hynix 2026 outlook](../raw/sk-hynix-hbm-2026-outlook-trendforce.md)). The memory-scarcity paper cannot fill that gap: it contains no supplier-level analysis at all and treats scarcity as a single exogenous input. ## Read next - [memory-scarcity-2607.07207-closeread](../raw/memory-scarcity-2607.07207-closeread.md) — the only primary-text read on the rung, and it corrects its own parent ingest on four points. - [memory-centric-computing-mutlu](../raw/memory-centric-computing-mutlu.md) — why the memory, not the processor, is where the energy goes. - [hbm4-c-hbm4e-architectural-shakeup](../raw/hbm4-c-hbm4e-architectural-shakeup.md) — the generation change that moves bandwidth per accelerator. ### Sources (25) - Inside the NVIDIA Vera Rubin Platform — Kyle Aubrey · 2026-01-05 — https://developer.nvidia.com/blog/inside-the-nvidia-rubin-platform-six-new-chips-one-ai-supercomputer/ - Custom Silicon Inflection 2026: Hyperscaler ASICs vs NVIDIA GPU — Wega Chu, Dylan Patel, Daniel Nishball et al. · 2026-02-25 — https://newsletter.semianalysis.com/p/vera-rubin-extreme-co-design-an-evolution - The Angstrom Era Arrives: TSMC Enters 2nm Mass Production and Unveils 1.6nm Roadmap — TokenRing Research · 2026-01-21 — https://markets.financialcontent.com/stocks/article/tokenring-2026-1-21-the-angstrom-era-arrives-tsmc-enters-2nm-mass-production-and-unveils-16nm-roadmap - HBM Architectural Shakeup: HBM4, HBM4E, C-HBM4E — 3nm Base Dies Enable 2.5x Performance — Anton Shilov · 2025-12-02 — https://www.tomshardware.com/pc-components/dram/hbm-undergoes-major-architectural-shakeup-as-tsmc-and-guc-detail-hbm4-hbm4e-and-c-hbm4e-3nm-base-dies-to-enable-2-5x-performance-boost-with-speeds-of-up-to-12-8gt-s-by-2027 - Memory-Centric Computing: A Paradigm Shift for Sustainable and Efficient Systems — Onur Mutlu (ETH Zurich) — synthesis of public talks and papers · 2026-04-21 — https://menfem.com/kb/hardware/sources/memory-centric-computing-mutlu - TSMC Debuts A13 Technology at 2026 North America Technology Symposium — TSMC / Industry coverage · 2026-04-23 — https://www.barchart.com/story/news/1456433/tsmc-debuts-a13-technology-at-2026-north-america-technology-symposium - TSMC Delays Use of ASML's High-NA EUV Machines Over Cost Concerns — Bloomberg · 2026-04-22 — https://www.bloomberg.com/news/articles/2026-04-22/tsmc-says-asml-s-latest-chipmaking-gear-is-too-pricey-to-use - NVIDIA Snaps Up AI Chip Packaging Capacity as TSMC Expands in U.S. — CNBC · 2026-04-08 — https://www.cnbc.com/2026/04/08/tsmc-nvidia-advanced-packaging-intel.html - NVIDIA, ASML, TSMC and Synopsys Set Foundation for Next-Generation Chip Manufacturing — NVIDIA Newsroom · 2026-04-23 — https://nvidianews.nvidia.com/news/nvidia-asml-tsmc-and-synopsys-set-foundation-for-next-generation-chip-manufacturing - TSMC to Quadruple Advanced Packaging Capacity: Reaching 130,000 CoWoS Wafers Monthly by Late 2026 — TokenRing / FinancialContent · 2026-02-05 — https://markets.financialcontent.com/stocks/article/tokenring-2026-2-5-tsmc-to-quadruple-advanced-packaging-capacity-reaching-130000-cowos-wafers-monthly-by-late-2026 - SK hynix 2026 Outlook: HBM3E Dominates, HBM4 Dual Strategy Amid 3 Market Headwinds — TrendForce · 2026-01-05 — https://www.trendforce.com/news/2026/01/05/news-sk-hynix-2026-outlook-hbm3e-remains-mainstream-hbm4-dual-strategy-amid-triple-market-headwinds/ - NVIDIA Vera Rubin Ramps Into Full Production to Power Agentic AI Factories Worldwide — NVIDIA Newsroom · 2026-05-31 — https://nvidianews.nvidia.com/news/vera-rubin-full-production-agentic-ai-factory - Global AI Server Shipments Forecast to Grow Over 28% YoY in 2026, with a Rising Share of ASIC-Based Systems — TrendForce · 2026-01-20 — https://www.trendforce.com/presscenter/news/20260120-12887.html - The Custom AI ASIC State of Play (May 2026) — Broadcom Deals, Google TPUs, Meta MTIA & Beyond — Tom's Hardware · 2026-05-26 — https://www.tomshardware.com/tech-industry/semiconductors/custom-ai-asics-examined-from-broadcom-to-mtia - TSMC — 6-K: June 2026 Revenue Report (filed 2026-07-13) — Taiwan Semiconductor Manufacturing Company Limited · 2026-07-13 — https://www.sec.gov/Archives/edgar/data/1046179/000104617926000447/tsm-revenue20260713.htm - TSMC — Q2 2026 Earnings Release + Quarterly Management Report — Taiwan Semiconductor Manufacturing Company Limited · 2026-07-16 — https://investor.tsmc.com/english/quarterly-results/2026/q2 - ASML — Q2 2026 Financial Results — ASML Holding N.V. · 2026-07-15 — https://www.asml.com/en/news/press-releases/2026/q2-2026-financial-results - Memory Scarcity, Open Models, and the Restructuring of the AI Industry, 2026-2030 — Satoshi Matsuoka · 2026-07-08 — https://arxiv.org/abs/2607.07207 - Close read: Vintage Breakeven (§6) and the Demand-Measurement Critique (§9.1/§9.3) — Matsuoka 2607.07207 — MenFem Research Desk · 2026-07-20 — https://arxiv.org/abs/2607.07207 - TSMC CoWoS Supply-Demand Gap Reportedly Narrowing from 20% to 10% by End-2026 as Capacity Expands — TrendForce (citing Economic Daily News, Reuters, Commercial Times) · 2026-06-15 — https://www.trendforce.com/news/2026/06/15/news-tsmc-cowos-supply-demand-gap-reportedly-seen-narrowing-from-20-to-10-by-end-2026-as-capacity-expands/ - Intel Q2 2026 Results (Actual Print) — 8-K Ex-99.1 — Intel Corporation · 2026-07-23 — https://www.sec.gov/Archives/edgar/data/50863/000005086326000155/q226earningsrelease.htm - TSMC — 6-K: August 2026 Revenue Report (filed 2026-09-10) — Wendell Huang, SVP & CFO · 2026-09-10 — https://www.sec.gov/Archives/edgar/data/0001046179/000104617926000658/tsm-revenue20260910.htm - SK hynix Announces 2Q26 Financial Results — SK hynix Inc. · 2026-07-29 — https://news.skhynix.com/en/q2-2026-business-results/ - Micron in High-Volume Production of HBM4 Designed for NVIDIA Vera Rubin, PCIe Gen6 SSD and SOCAMM2 — Micron Technology, Inc. · 2026-03-16 — https://investors.micron.com/news/press-release/2026/Micron-in-High-Volume-Production-of-HBM4-Designed-for-NVIDIA-Vera-Rubin-PCIe-Gen6-SSD-and-SOCAMM2-03-16-2026/default.aspx - SK hynix Ships Samples of 12-Layer Next-Gen 'HBM4E' — SK hynix Inc. · 2026-06-18 — https://news.skhynix.com/en/sk-hynix-ships-samples-of-12-layer-next-gen-hbm4e-2/ ## Networking & Interconnect Canonical URL: https://menfem.com/kb/networking How chips and sites talk to each other. --- title: "Networking & Interconnect — Knowledge Base" last_compiled: "2026-09-24T12:30:00Z" source_count: 5 concept_count: 1 entity_count: 1 papers: 2 news: 2 --- # Networking & Interconnect This rung asks how chips, racks and sites talk to each other, and what that costs: optical links and co-packaged optics, the fabrics inside a rack and between racks, NVLink versus InfiniBand versus Ethernet, bandwidth per dollar, and links between data centres. Memory bandwidth inside a chip package belongs to `hardware`. Computing *with* light is out of the knowledge base entirely; communicating with light is in. ## What this rung prices A data centre has a fixed power feed. Every watt spent moving bits between chips is a watt not spent on the chips doing the work, so the power cost of each bit moved sets how much of the feed reaches compute, and with it how many tokens a site can serve. The rung's three in-scope sources all make a claim on exactly that term. Broadcom and NVIDIA, the two largest switch vendors, each state that putting the optics inside the switch package ("co-packaged optics") uses about **3.5x less power** for the optical links than plugging optical modules into the front of the switch ([Broadcom Tomahawk 6 – Davisson](../raw/broadcom-tomahawk-6-davisson-cpo-switch.md); [NVIDIA Spectrum-X / Quantum-X Photonics](../raw/nvidia-spectrum-x-quantum-x-photonics-cpo-switches.md)). NVIDIA's technical blog then gives the watts: pluggable optics "often 30W per interface", co-packaged "as low as 9W" ([NVIDIA CPO blog](../raw/nvidia-cpo-power-efficiency-blog-2025.md)). That is about **3.3x** on the vendor's own numbers (our arithmetic), a typical case against a best case, on switches not shipping when it was written. It is the first figure on the rung that could be carried towards a cost per token, once someone says how many switch ports a site runs per GPU. ## The load-bearing findings - **Two vendors land on about 3.5x, and the agreement is the only reason to use the number.** Broadcom's BCM78919 switch moves 102.4 Tb/s through 16 optical engines of 6.4 Tb/s each at 200 Gb/s per link, with a 70% cut in optical link power, "more than 3.5x lower than traditional pluggable solutions". The 70% and the 3.5x are the same claim stated twice, not two pieces of evidence, and the saving is scoped to the optical links, not to the switch or the whole system ([Broadcom](../raw/broadcom-tomahawk-6-davisson-cpo-switch.md)). NVIDIA's release states 3.5x power efficiency, 4x fewer lasers and 1.3x faster deployment against pluggable optics ([NVIDIA](../raw/nvidia-spectrum-x-quantum-x-photonics-cpo-switches.md)). - **The watts behind the multiplier, and why it is 3.3x rather than 3.5x.** NVIDIA's blog gives "often 30W per interface" for pluggables and "as low as 9W" for co-packaged optics, and electrical loss of up to 22 dB against about 4 dB. 30 ÷ 9 is about 3.3x, and Broadcom's "70%" is also 3.3x; both vendors headline 3.5x without showing how. The 22 dB to 4 dB gap is 18 dB, a power ratio of about 63x — which is exactly where the "63x" / "64x" signal-integrity figures come from. *Corrected 2026-09-24:* this page and the NVIDIA release's close read had treated that multiplier as a marketing composite with no unit; it is a unit conversion of a best-case loss figure ([NVIDIA CPO blog](../raw/nvidia-cpo-power-efficiency-blog-2025.md)). "Interface" is not defined, so a per-bit figure (about 37 against 11 pJ/bit if it means an 800 Gb/s port) is unconfirmed. - **Neither part was shipping in volume on the date of its source.** Broadcom's part was sampling at announcement (2025-10-08). NVIDIA's was a roadmap: InfiniBand "later in 2025" and Ethernet in 2026, announced 2025-03-18. Five months later NVIDIA's blog restated those as **early 2026** and **the second half of 2026**, a slip. Both compare against an unnamed pluggable baseline. Sampling is not shipping, and a roadmap is not a measurement. - **The scale these switches claim.** Broadcom: 512 accelerators in a single scale-up cluster and 100,000+ in a two-tier scale-out network. NVIDIA: Ethernet configurations of 100 Tb/s and 400 Tb/s, and a 144-port 800 Gb/s InfiniBand switch ([Broadcom](../raw/broadcom-tomahawk-6-davisson-cpo-switch.md); [NVIDIA](../raw/nvidia-spectrum-x-quantum-x-photonics-cpo-switches.md)). - **The supply chain is named by the vendor.** NVIDIA's own partner list for these switches includes Lumentum, alongside TSMC, Coherent, Corning, Fabrinet, Foxconn and others. That is the vendor naming the supplier, a stronger kind of evidence than an analyst saying so ([NVIDIA](../raw/nvidia-spectrum-x-quantum-x-photonics-cpo-switches.md)). It bears on the rung's one thesis, which has a live position behind it — see [theses](theses.md). - **Inside the rack, the concept page carries NVIDIA's platform figures.** The [silicon photonics](./concepts/silicon-photonics.md) page, compiled from an NVIDIA technical blog held on the `hardware` rung, records 102.4 Tb/s per Spectrum-6 switch and 260 TB/s of aggregate bandwidth across the 72 GPUs of an NVL72 rack. Its "64x signal integrity" figure and the release's "63x" are the same 22 dB to 4 dB loss gap, now explained on the concept page. - **Two of the four sources are about computing with light, which is out of scope.** A photonic tensor processor from the imec platform ran pretrained networks on a 9 × 3 optical crossbar ([photonic tensor processor](../raw/photonic-tensor-processor-pytorch.md)), and Nokia Bell Labs trained a small photonic network on the chip itself (2 inputs, 8 hidden, 1 output; 92.5% on a 2D classification task) ([on-chip backprop](../raw/photonic-backprop-onchip-training.md)). They came to this rung with the rescued thesis. They are counted here for accuracy, not for coverage. ## Concept map ```mermaid graph LR A[Co-packaged optics] --> B[Power per bit] B --> C[Watts left for compute] D[Silicon photonics] --> A E[Pluggable optics] --> B ``` ## Concepts | Concept | Sources | Evidence | Last Updated | |---------|---------|----------|-------------| | [Silicon Photonics](concepts/silicon-photonics.md) | 4 (0 papers) | Moderate; all vendor sources | 2026-09-24 | ## Entities | Entity | Type | Sources | |--------|------|---------| | [Nokia Bell Labs](entities/nokia-bell-labs.md) | Lab | 1 | The Silicon Photonics page now carries the co-packaged-optics power figures. The Nokia Bell Labs page is about photonic computing, the out-of-scope half. ## What we do not know yet - **Power per bit, independently measured.** The only absolute figures are NVIDIA's own (30 W against 9 W per "interface", undefined). No picojoules-per-bit figure is stated, and no one but the vendors has measured anything. - **Whether either part shipped in volume, on time, at the stated figures.** NVIDIA's own dates had already slipped once by August 2025. A later Broadcom volume claim was noted but not verified, because the product pages would not load. - **Whether co-packaged optics replaces pluggable modules or adds optical content per system.** Broadcom's release is evidence for replacement at the switch face; nothing here measures the total. - **Most of the scope.** Scale-up versus scale-out economics, NVLink against InfiniBand against Ethernet on cost, bandwidth per dollar, topology and links between sites all lack a source. - **Standing: read it.** Every figure here is vendor-stated and read, not measured. ## Timeline See [timeline.md](timeline.md) for chronological developments, [frontier.md](frontier.md) for the open questions, and [theses.md](theses.md) for the rung's one thesis. ## Sources | # | Title | Type | Date | Status | |---|-------|------|------|--------| | 1 | [NVIDIA Spectrum-X Photonics and Quantum-X Photonics](../raw/nvidia-spectrum-x-quantum-x-photonics-cpo-switches.md) | news | 2025-03-18 | compiled | | 2 | [Scaling AI Factories with Co-Packaged Optics (NVIDIA blog)](../raw/nvidia-cpo-power-efficiency-blog-2025.md) | technical-report | 2025-08-18 | compiled | | 3 | [Broadcom Tomahawk 6 – Davisson](../raw/broadcom-tomahawk-6-davisson-cpo-switch.md) | news | 2025-10-08 | compiled | | 4 | [Photonic Tensor Processor (imec)](../raw/photonic-tensor-processor-pytorch.md) | paper | 2026-01-15 | compiled | | 5 | [Integrated Photonic Neural Network with On-Chip Backpropagation](../raw/photonic-backprop-onchip-training.md) | paper | 2026-03-20 | compiled | ## Read next - [nvidia-cpo-power-efficiency-blog-2025](../raw/nvidia-cpo-power-efficiency-blog-2025.md) — the watts behind the 3.5x, and why they give 3.3x. - [broadcom-tomahawk-6-davisson-cpo-switch](../raw/broadcom-tomahawk-6-davisson-cpo-switch.md) — the power-per-bit claim on a named part, and why 70% and 3.5x are one number. - [nvidia-spectrum-x-quantum-x-photonics-cpo-switches](../raw/nvidia-spectrum-x-quantum-x-photonics-cpo-switches.md) — the second vendor's claim, and the supplier list. - [theses.md](theses.md) — the rescued interconnect thesis, and why it needs in-scope evidence. ### Sources (5) - Deep Neural Network Inference on an Integrated, Reconfigurable Photonic Tensor Processor — Photonic tensor processor team · 2026-01-15 — https://www.nature.com/articles/s41467-026-71599-2 - Integrated Photonic Neural Network with On-Chip Backpropagation Training — Farshid Ashtiani, Mohamad Hossein Idjadi, Kwangwoong Kim · 2026-03-20 — https://www.nature.com/articles/s41586-026-10262-8 - Broadcom Announces Tomahawk 6 - Davisson, the Industry's First 102.4-Tbps Ethernet Switch with Co-Packaged Optics — Broadcom Inc. (press release) · 2025-10-08 — https://www.globenewswire.com/news-release/2025/10/08/3163429/19933/en/Broadcom-Announces-Tomahawk-6-Davisson-the-Industry-s-First-102-4-Tbps-Ethernet-Switch-with-Co-Packaged-Optics.html - NVIDIA Announces Spectrum-X Photonics, Co-Packaged Optics Networking Switches to Scale AI Factories to Millions of GPUs — NVIDIA (press release, GTC 2025) · 2025-03-18 — https://nvidianews.nvidia.com/news/nvidia-spectrum-x-co-packaged-optics-networking-switches-ai-factories - Scaling AI Factories with Co-Packaged Optics for Better Power Efficiency — Ashkan Seyedi · 2025-08-18 — https://developer.nvidia.com/blog/scaling-ai-factories-with-co-packaged-optics-for-better-power-efficiency/ ## Datacenters & Digital Infrastructure Canonical URL: https://menfem.com/kb/datacenters Where inference runs, and what stops it running. --- title: "Datacenters & Digital Infrastructure — Knowledge Base" last_compiled: "2026-09-24" source_count: 24 concept_count: 8 entity_count: 6 --- # Datacenters & Digital Infrastructure Where inference runs, and what stops it running. This rung covers the building, the megawatts behind it and the queue to get those megawatts energised — the capex that is committed, the capacity that actually gets built, and the physical inputs that decide which of those two numbers is smaller. Power is not a footnote here; it is most of the subject. ## What this rung prices A token price is quoted per million tokens, but a token has to be produced somewhere, and the somewhere costs money before a single request arrives. This rung supplies the denominator: how many dollars of capital sit behind one megawatt of serving capacity, how long that megawatt takes to energise, and how much of the capital is idle while it waits. One operator's breakdown puts an all-in AI datacenter at roughly **$59M per MW** — about $30M of IT and $29M of shell, generation and plant — against about **$15M/MW/year** of lease revenue and roughly a four-year payback ([Economics of a Megawatt of AI Data Center](../raw/nextbigfuture-ai-datacenter-capex-per-mw-2026.md)). Change the interconnection wait, the rack density or the packaging lead time and that payback moves, and the floor under the price of a served token moves with it. That is why the rung's central question is not "how much are they spending" but "how much of the spend has turned into something that can serve a request" — and the gap between those two is where the price of a finished task is actually set. ## The load-bearing findings - **The shortage is queue throughput, not megawatts.** Interconnection waits in data-center load zones run **three to four years**, against a combined PJM + ERCOT queue of **over 300 GW across roughly 1,500 projects** — enough capacity on paper to meet projected data-center demand through 2030. Renewables and storage make up 77% of PJM's active queue and 87% of ERCOT's, while gas clears "substantially more quickly"; monthly gas entries to ERCOT's queue rose about **150%** after the One Big Beautiful Bill Act ([AI Meets the Grid](../raw/carbon-direct-pjm-ercot-interconnection-queue-2026.md)). That single distinction — deliverability rather than generation — is what makes on-site power worth its siting cost, because it skips the queue entirely ([Power 2026](../raw/power-2026-electricity-pricing-primer.md)). - **The buyer's cash flow has already inverted.** Alphabet's Q2 2026 capex was **$44.9B** against **$39.1B** of operating cash flow, so free cash flow printed **negative $(5.9)B**; the gap was plugged with a **$49.6B** June equity raise explicitly earmarked for AI-infrastructure capex plus **$20.3B** of new senior notes, and long-term debt more than doubled to **$98.2B** in six months ([Alphabet Q2 2026 8-K Ex-99.1](../raw/googl-q2-2026-earnings.md)). The build is no longer funded out of the ad business. Tesla's own print shows the same shape one order of magnitude down — capex $5.79B, up 142% year on year, free cash flow negative $(1.1)B, self-funded from cash ([Tesla Q2 2026 8-K Ex-99.1](../raw/tsla-q2-2026-earnings.md)) — see [hyperscaler capex cycle](./concepts/hyperscaler-capex-cycle.md). - **Three physical inputs bind at once, and capital is the only abundant one.** TSMC's CoWoS advanced packaging and the N3 node are reported sold out through end-2026 with lead times into 2027 ([TSMC Q2 2026](../raw/tsmc-q2-2026-cowos-sold-out-through-2026.md)); industry packaging capacity is heading to roughly **200,000 wafers/month** in 2026 yet the supply-demand gap only narrows from about **20% to about 10%** by year end ([TrendForce on CoWoS](../raw/trendforce-tsmc-cowos-supply-demand-gap-2026.md)); and Samsung's preliminary Q2 2026 guidance points to operating profit roughly **19× higher year on year** on HBM and DRAM tightness — a price signal, not a supply measurement, and preliminary rather than final ([Samsung Q2 2026 preliminary](../raw/samsung-q2-2026-preliminary-guidance-hbm.md)). See [advanced packaging bottleneck](./concepts/advanced-packaging-bottleneck.md) and [HBM supply tightness](./concepts/hbm-supply-tightness.md). - **A modern rack draws what a small building used to.** The GB300 NVL72 is specified at **135 kW rack TDP, up to about 155 kW peak**, with 72 B300 GPUs and 36 Grace CPUs, and heat removed roughly **90% by liquid and 10% by air** ([Lenovo GB300 NVL72 product guide](../raw/gb300-nvl72-rack-power-135kw-lenovo.md)) — a vendor datasheet, authoritative for the spec but not independent. At 135 kW nominal that is about **7.4 racks per megawatt**, which is the arithmetic that ties rack counts to the capex-per-MW and grid-queue figures ([rack power density](./concepts/rack-power-density.md)). - **The construction is now big enough to show up in the national accounts.** The Federal Reserve reports business fixed investment growing **11% annualised in Q1 2026 against 5.5% in 2025** and attributes most of the strength to AI-services infrastructure, while noting AI-related imports took roughly **0.4 percentage points** off growth through net exports and that the productivity payoff has not arrived — labour productivity about **2.1% annualised since late 2019**, with only a "modest" AI contribution ([Monetary Policy Report, July 2026](../raw/fed-mpr-july-2026-datacenter-construction.md)). Measured from the ground, data-center construction spending reached **$58.1B year-to-date through May**, more than four times the 2025 record pace, and data centers are now over **20% of all US nonresidential building starts** ([ConstructConnect, July 2026](../raw/constructconnect-datacenter-report-july-2026.md)). This is [datacenter construction as a macro variable](./concepts/datacenter-construction-as-macro.md). - **The demand behind the buildout may be counted more than once.** CoreWeave's Q1 2026 print shows **$2.08B** of revenue against **$7.70B** of quarterly capex — roughly 3.7× revenue — an operating loss of **$(144)M**, a net loss of **$(740)M**, a **$99.4B** revenue backlog and **over 3.5 GW** contracted power, with a **$21B** Meta commitment named ([CoreWeave Q1 2026](../raw/coreweave-q1-2026-results.md)). That backlog is contracted largely to the same hyperscalers whose own cloud backlogs sit one layer up, so the same end-demand can be booked twice. - **Most of the queue is not real demand, and connecting is getting expensive.** The IEA counts at least **150 GW** of data-centre projects waiting for grid connections at advanced stages, and one fifth of the world's data-centre build at risk of delay — yet from US utility data only **about 20%** of data-centre connection requests ever turn into load. Utilities are pricing that in: Dominion Energy Virginia proposed that large users commit for **14 years to pay 60% of generation and 85% of network costs** whether they use the power or not. Data centres make up about half of US electricity demand growth to 2030 ([IEA Electricity 2026](../raw/iea-electricity-2026-datacentre-demand.md)). A pipeline counted in requested gigawatts overstates future serving capacity about five times over — see [grid connection queue](./concepts/grid-connection-queue.md). - **The building's overhead is a solved problem for two companies and an unsolved one for everyone else.** Google's fleet runs at a PUE of **1.09** (2025, still 1.09 at Q2 2026) and Microsoft's at **1.17** (FY25, worse than the 1.16 before), while the industry average, as operators report it, is **1.52** and has barely moved in seven years ([Google](../raw/google-datacenter-pue-2025-fleet.md); [Microsoft](../raw/microsoft-datacenter-pue-wue-fy24-fy25.md); [Uptime 2026](../raw/uptime-global-datacenter-survey-2026.md)). That is 0.09, 0.17 and 0.52 kWh of overhead per kWh of computing: on the same grid connection, the average site serves about 28% fewer tokens than Google's. All three numbers are the operators' own, on boundaries they chose — see [power usage effectiveness](./concepts/power-usage-effectiveness.md). ## What we do not know yet - **Nothing on this rung was measured here.** Every figure above is read, not run: no site visited, no rack metered, no filing audited beyond reading it. The rack-power number comes from a vendor's own product guide, the capex-per-MW number from a single operator's contract pricing republished by an analyst, and the memory tightness from a preliminary guidance release rather than final segment results. - **How much HBM exists, and who gets it.** The rung holds the price consequence and none of the supply: no bit-supply, wafer-allocation, HBM4 qualification or per-stack pricing data appears in any source ([Samsung Q2 2026 preliminary](../raw/samsung-q2-2026-preliminary-guidance-hbm.md)). Allocation is the tradable question and it is entirely unanswered. - **What power for AI actually costs.** The two price anchors on the rung — about **$271/MWh** implied for the Anthropic–TeraWulf lease without GPUs, against roughly **$5,000/MWh** for the SpaceX–Reflection arrangement with GPUs ready to go — are both the author's own derivations, one of them labelled napkin math, and neither is a market print ([Power 2026](../raw/power-2026-electricity-pricing-primer.md)). The gap between selling electricity and selling compute is the right shape; the numbers are not quotable as prices. - **Whether the neocloud model earns its cost of capital.** CoreWeave's release omits the GPU depreciation schedule, and omits backlog duration, cancellability and customer concentration ([CoreWeave Q1 2026](../raw/coreweave-q1-2026-results.md)) — which is exactly the disclosure the question turns on. - **Where the guidance actually landed.** Alphabet's printed 8-K exhibit carries neither a full-year capex dollar guide nor a remaining-performance-obligation figure; both live in the call and the 10-Q, so the watched **$180–190B** guide and the **$462B** cloud backlog can be neither confirmed nor broken from the primary print ([Alphabet Q2 2026](../raw/googl-q2-2026-earnings.md); [Alphabet Q2 2026 preview](../raw/alphabet-q2-2026-earnings-preview.md), which was fetched only partially behind a paywall). - **How the aggregate capex number is built.** Two sources on the same year differ by roughly a hundred billion — around **$725B** for 2026 in one ([AI Hyperscaler Capex 2026](../raw/ai-hyperscaler-capex-2026.md)) against a **$660–690B** range in another ([The $690B Infrastructure Sprint](../raw/futurum-hyperscaler-capex-2026-690b.md)) and "in excess of $600B" in a third ([Hyperscalers in 2026](../raw/hyperscaler-power-bottleneck-2026.md)). The concept page reconciles the spread to guidance vintage, fiscal-versus-calendar basis and basket membership rather than to disagreement, but no single stated basis exists to quote. - **What the pipeline converts at.** There are **770 future hyperscale facilities** in the pipeline and **36 or more projects worth $162B** blocked or delayed ([Hyperscalers in 2026](../raw/hyperscaler-power-bottleneck-2026.md)), and no source here tracks what share of an announced pipeline has historically energised, or with what lag. - **Whether owning your own power and compute works.** Tesla's Cortex is the rung's one worked example of the vertically integrated path — a 500 MW campus paired with on-site storage ([Tesla Cortex 2](../raw/tesla-cortex-2-500mw-gigatexas.md)) — and it is a single site with no disclosed cost per megawatt to compare against the leased alternative ([vertically integrated compute](./concepts/vertically-integrated-compute.md)). - **What an AI hall's overhead actually is.** None of the three PUE figures separates AI-dedicated sites; Uptime says there are still too few to move its average, and both hyperscalers leave out sites in their first year, which is when a new hall runs worst ([power usage effectiveness](./concepts/power-usage-effectiveness.md)). - **How much electricity data centres will use in total.** The rung holds the IEA's US share and the queue, but no global TWh forecast: the widely quoted **485 → 950 TWh** (2025 → 2030) is from a different IEA report, *Key Questions on Energy and AI*, which is not ingested ([IEA Electricity 2026](../raw/iea-electricity-2026-datacentre-demand.md) records the correction). ## Read next - [carbon-direct-pjm-ercot-interconnection-queue-2026](../raw/carbon-direct-pjm-ercot-interconnection-queue-2026.md) — the queue numbers that turn "power is the constraint" into a measurable wait. - [nextbigfuture-ai-datacenter-capex-per-mw-2026](../raw/nextbigfuture-ai-datacenter-capex-per-mw-2026.md) — the per-megawatt unit that makes the whole buildout modellable, caveats and all. - [googl-q2-2026-earnings](../raw/googl-q2-2026-earnings.md) — the primary filing where the buyer's free cash flow goes negative and the funding shifts to capital markets. - [uptime-global-datacenter-survey-2026](../raw/uptime-global-datacenter-survey-2026.md) — the fleet-wide average every hyperscaler efficiency claim should be read against. ### Sources (24) - AI Hyperscaler Capex 2026: Why Microsoft, Google, Meta and Amazon Are All Spending at Once — Value Add VC · 2026-06-11 — https://valueaddvc.com/blog/ai-hyperscaler-capex-compared-why-microsoft-google-meta-and-amazon-are-all-spending-at-once - Hyperscalers in 2026: The Sprint Meets the Power Bottleneck — Data Center Knowledge · 2026-01-06 — https://www.datacenterknowledge.com/hyperscalers/hyperscalers-in-2026-what-s-next-for-the-world-s-largest-data-center-operators- - NVIDIA and Partners Build America's AI Infrastructure — NVIDIA Newsroom · 2025-10-28 — https://nvidianews.nvidia.com/news/nvidia-partners-ai-infrastructure-america - Monetary Policy Report — July 2026 (Data Center & AI Investment Sections) — Board of Governors of the Federal Reserve System · 2026-07-10 — https://www.federalreserve.gov/monetarypolicy/2026-07-mpr-part1.htm - AI Capex 2026: The $690B Infrastructure Sprint — Nick Patience · 2026-02-12 — https://futurumgroup.com/insights/ai-capex-2026-the-690b-infrastructure-sprint/ - TSMC Q2 2026 Earnings — Record Revenue, CoWoS and N3 Sold Out Through 2026 — Dan · 2026-07-16 — https://phemex.com/blogs/tsmc-record-revenue-earnings-ai - July 2026 Data Center Report: Year-to-Date Spending Is Four Times the 2025 Record — Michael Guckes (with Devin Bell) · 2026-06-25 — https://news.constructconnect.com/july-2026-data-center-report-year-to-date-spending-four-times-the-2025-record - Tesla Cortex 2: 500MW GPU Cluster / Data Center at Giga Texas — Justin D · 2026-01-08 — https://gearmusk.com/2026/01/08/tesla-cortex-2-500mw-gpu/ - Samsung Q2 2026 Preliminary Guidance — Record Profit on HBM/DRAM Demand — InsiderFinance · 2026-07-07 — https://www.insiderfinance.io/news/samsung-q2-2026-guidance-signals-record-quarter - Alphabet Q2 2026 Earnings Preview — AI Spending Question Looms — Ryan Hasson · 2026-07-15 — https://www.marketbeat.com/articles/alphabets-ai-spending-question-looms-over-q2-earnings/ - Alphabet Q2 2026 Results (Actual Print) — 8-K Ex-99.1 — Alphabet Inc. · 2026-07-22 — https://www.sec.gov/Archives/edgar/data/1652044/000165204426000066/googexhibit991q22026.htm - Tesla Q2 2026 Update (Actual Print) — 8-K Ex-99.1 — Tesla, Inc. · 2026-07-22 — https://www.sec.gov/Archives/edgar/data/1318605/000162828026049213/exhibit991.htm - Texas Instruments Q2 2026 Results (Actual Print) — 8-K Ex-99 — Texas Instruments Incorporated · 2026-07-22 — https://www.sec.gov/Archives/edgar/data/97476/000009747626000148/q22026txnex99-eredgar.htm - Lenovo NVIDIA GB300 NVL72 Rack Scale AI — Product Guide (Rack Power & Cooling Spec) — Lenovo Press · 2026-07-15 — https://lenovopress.lenovo.com/lp2357-lenovo-nvidia-gb300-nvl72-rack-scale-ai - AI Meets the Grid: Interconnection Queue Analysis in PJM and ERCOT — Carbon Direct · 2026-05-01 — https://www.carbon-direct.com/research-and-reports/ai-meets-the-grid-interconnection-queue-analysis-in-pjm-and-ercot - Economics of a Megawatt of AI Data Center — Brian Wang / NextBigFuture · 2026-05-02 — https://www.nextbigfuture.com/2026/05/economics-of-a-megawatt-of-ai-data-center.html - TSMC CoWoS Supply-Demand Gap Reportedly Seen Narrowing from 20% to 10% by End-2026 as Capacity Expands — TrendForce · 2026-06-15 — https://www.trendforce.com/news/2026/06/15/news-tsmc-cowos-supply-demand-gap-reportedly-seen-narrowing-from-20-to-10-by-end-2026-as-capacity-expands/ - CoreWeave Reports Strong First Quarter 2026 Results — CoreWeave, Inc. · 2026-05-07 — https://investors.coreweave.com/news/news-details/2026/CoreWeave-Reports-Strong-First-Quarter-2026-Results/ - Power 2026 — a primer on electricity pricing and AI datacenter development — Neel Somani — https://power2026.ai - Measuring energy and water efficiency for Microsoft datacenters — Microsoft (Cloud Operations + Innovation) — https://datacenters.microsoft.com/sustainability/efficiency/ - Inside Microsoft's two-decade push to cut water intensity while scaling for growth — Microsoft (corporate blog) · 2026-06-24 — https://blogs.microsoft.com/blog/2026/06/24/inside-microsofts-two-decade-push-to-cut-water-intensity-while-scaling-for-growth/ - Electricity 2026: Analysis and forecast to 2030 (data-centre demand and grid-queue sections) — International Energy Agency — https://www.iea.org/reports/electricity-2026 - Uptime Institute Global Data Center Survey 2026 (executive summary) — Douglas Donnellan, Peter Judge, Andy Lawrence, John O'Brien, Daniel Bizo, Jacqueline Davis, Owen Rogers, Max Smolaks, Rose Weinschenk · 2026-07-28 — https://uptimeinstitute.com/resources/research-and-reports/uptime-institute-global-data-center-survey-results-2026 - Google Data Centers — Power usage effectiveness (fleet PUE 1.09, 2025; quarterly data to Q2 2026) — Google Data Centers — https://datacenters.google/efficiency/ ## Energy & Power Canonical URL: https://menfem.com/kb/energy Where the electrons come from, and what they cost. --- title: "Energy & Power — Knowledge Base" last_compiled: "2026-09-24T12:30:00Z" source_count: 5 concept_count: 1 entity_count: 0 papers: 0 news: 0 --- # Energy & Power This rung asks where the electricity for AI comes from and what it costs: generation (gas, nuclear, renewables), power-purchase deals, the queues to connect a site to the grid, and the long wait for transformers and switchgear. The buildings themselves belong to `datacenters`, and the cost of a token belongs to `inference-economics`. This rung is the feed into both. ## What this rung prices A chip that cannot be switched on serves no tokens. So before any question of what a token costs, there is a question of how many chips can be powered at all, and that number is set here. The rung's sources speak to the *how many* side more than the *how much* side. They count power that has been promised but not delivered, measure how long a new plant waits to connect (a median of about five years), and show a state government stopping the queue. The rung now holds **one price**: PJM's capacity auction, the yearly price for power plants promising to be there at peak, cleared at **$325 per MW-day** for 2028/29, at its agreed cap ([PJM 2028/29](../raw/pjm-capacity-auction-2028-2029.md)). That is roughly $119,000 per MW-year (our arithmetic). It is a price for availability, not for the electricity used, and none of the sources gives a price per megawatt-hour for a data centre or says how much reaches a token price. ## The load-bearing findings - **About 12 GW of US data-centre power was announced for 2026, and about 5 GW was being built.** That leaves a gap of about 7 GW, with only about a third of announced 2026 sites having broken ground. The same source puts AI data-centre load at about 10 GW by the end of 2026, and shows one utility (AEP) with 24 GW of committed demand by 2030 against a raw incoming queue of 190 GW ([7 GW shortfall](../raw/us-ai-dc-7gw-power-shortfall-2026.md)). This is the weakest source on the rung: it was ingested from a search snippet because the page could not be fetched, and it is flagged for a full re-read. - **In the biggest US load-growth market, the queue was stopped by a governor, not by physics.** On 2026-08-03 ERCOT said Governor Abbott had directed it to verify data-centre "Large Loads" before moving them through interconnection, that it would miss its 2026-08-07 deadline for classifying those loads, and that it would ask the Texas utility commission for a good-cause exception ahead of the 2026-08-20 meeting ([ERCOT notice](../raw/ercot-batch-zero-timeline-pause-2026.md)). The notice does not say how much load is affected or how long the check takes, and a missed deadline is not a cancellation. What it does undo is the easy assumption that if connecting is slow elsewhere, you build in Texas. - **The federal forecast absorbed the Texas pause and still named that region the leader.** The EIA's September 2026 outlook forecasts US generation of 4,368 billion kWh in 2026 (+2.2%) and +1.7% in 2027, and says that "despite a pause in new data center projects in Texas", the West South Central region remains the largest contributor to growth in electricity sales, driven by data centres and manufacturing ([EIA STEO](../raw/eia-steo-record-generation-2026-2027.md)). The national growth figure is modest; the regional concentration is the point. The release gives energy, not gigawatts of data-centre load, and no data-centre share of demand. - **Power is a binding constraint, but no longer the only one.** The concept page carried over from `datacenters` sets out the case: 36+ projects worth $162B stalled or delayed, transformers and switchgear as the long-lead items, and a 3–4 year wait to connect in data-centre load zones in PJM and ERCOT, where the queue holds enough generation "on paper" and the shortage is in connecting it. It also records that advanced packaging and memory were binding at the same time in mid-2026. See [Power: The Binding Constraint](./concepts/power-the-binding-constraint.md). - **The national queue is about 2,000 GW on paper, and most of it will never be built.** At the end of 2025 about 8,200 projects, **1,312 GW of generation and about 749 GW of storage**, were waiting to connect to the US grid. The queue shrank 10% in a year, but **gas rose 86% to 253 GW**, the only large type that grew. The median project built in 2025 took **61 months** from request to switching on (22 months in 2008), only 2% of the 2024 queue came online in 2025, and only 13% of what was requested in 2000–2020 was ever built. The count excludes data-centre load queues, so it measures supply waiting, not AI demand ([LBNL Queued Up 2026](../raw/lbnl-queued-up-2026-edition.md)). This census also corrects the Carbon Direct queue size carried on the concept page: LBNL counts ERCOT at 408 GW and PJM at 144 GW, well above that source's "more than 300 GW" for the two combined. - **PJM's grid, where data centres drive most of the demand growth, is short, and its price is pinned at a ceiling.** PJM's capacity auctions ran at **$329.17 (2026/27), $333.44 (2027/28) and $325 (2028/29) per MW-day**, under a price collar agreed with the 13 state governors and the federal regulator; the last two cleared at the cap. The last two were also the first in PJM's history to come up short of the reliability requirement for the whole region, by 6,623 MW and 6,831 MW. PJM attributes nearly 5,100 MW of the 5,250 MW rise in its 2027/28 peak forecast to data centres ([PJM 2028/29](../raw/pjm-capacity-auction-2028-2029.md)). The two earlier prices are kept as dated, superseded prints. A capped price understates scarcity: it says the market wanted at least this much. ## Concept map ```mermaid graph LR A[Interconnection queue] --> B[Power as the binding constraint] F[LBNL queue census] --> A G[PJM capacity price] --> B C[ERCOT verification pause] --> A D[EIA generation forecast] --> B B --> E[Powered chips] ``` ## Concepts | Concept | Sources | Evidence | Last Updated | |---------|---------|----------|-------------| | [Power: The Binding Constraint](concepts/power-the-binding-constraint.md) | 11 (0 papers) | Moderate to strong | 2026-09-24 | The concept page was carried over from `datacenters` and still cites mostly sources held there (links repointed 2026-09-24). It now folds in the LBNL census and the PJM price series, and records where they correct its earlier figures. ## Entities No entity pages exist on this rung yet. ## What we do not know yet - **What the electricity costs.** The PJM capacity price is a price for being available at peak, not for energy used. No source gives a data-centre electricity price, a PPA price, or how much of either passes through to a token. The EIA release carries a Henry Hub gas price assumption of $3.43/MMBtu, which is an input to its outlook, not a power price. - **What an uncapped PJM auction would have cleared at.** The last two hit the cap; an uncapped figure appears only in secondary coverage and is not used here. - **How much load the Texas check affects, and what the commission decided on 2026-08-20.** Not in the notice. A figure of about 474 GW of large-load queue in ERCOT is explicitly flagged in the source as unverified and is not used here ([ERCOT notice](../raw/ercot-batch-zero-timeline-pause-2026.md)). - **Gigawatts of data-centre demand from a primary source.** The EIA gives energy, not load; LBNL excludes load queues; the 7 GW gap comes from a snippet-only analysis. PJM's ~5,100 MW of data-centre forecast growth (one region, one year) is the closest primary figure held. - **Most of the scope.** Nuclear and gas generation deals, power arbitrage, energy ownership as a moat, and transformer lead times all lack a source on this rung. - **Standing: read it.** Every number here is read from filings, forecasts or analysis. ## Timeline See [timeline.md](timeline.md) for chronological developments, and [frontier.md](frontier.md) for the open questions. ## Sources | # | Title | Type | Date | Status | |---|-------|------|------|--------| | 1 | [U.S. AI Data Center Delays: 7 GW Capacity Crisis (2026)](../raw/us-ai-dc-7gw-power-shortfall-2026.md) | analysis | 2026-Q1 | compiled | | 2 | [ERCOT Market Notice M-A080326-01](../raw/ercot-batch-zero-timeline-pause-2026.md) | technical-report | 2026-08-03 | compiled | | 3 | [EIA September 2026 Short-Term Energy Outlook](../raw/eia-steo-record-generation-2026-2027.md) | technical-report | 2026-09-09 | compiled | | 4 | [LBNL, Queued Up: 2026 Edition](../raw/lbnl-queued-up-2026-edition.md) | technical-report | 2026-06 | compiled | | 5 | [PJM 2028/2029 Base Residual Auction results](../raw/pjm-capacity-auction-2028-2029.md) | technical-report | 2026-07-14 | compiled | ## Read next - [pjm-capacity-auction-2028-2029](../raw/pjm-capacity-auction-2028-2029.md) — the rung's first price for power, and three auctions of it. - [lbnl-queued-up-2026-edition](../raw/lbnl-queued-up-2026-edition.md) — the size of the wait, and how little of the queue gets built. - [ercot-batch-zero-timeline-pause-2026](../raw/ercot-batch-zero-timeline-pause-2026.md) — the newest constraint on the rung, and a political one. - [eia-steo-record-generation-2026-2027](../raw/eia-steo-record-generation-2026-2027.md) — read it with the ERCOT notice: the pause, absorbed into a federal forecast. - [us-ai-dc-7gw-power-shortfall-2026](../raw/us-ai-dc-7gw-power-shortfall-2026.md) — the gap between announced and built, with its weak provenance kept in view. ### Sources (5) - U.S. AI Data Center Delays: 7 GW Capacity Crisis (2026) — Tech Insider · 2026-Q1 — https://tech-insider.org/us-ai-data-center-delays-cancellations-7gw-capacity-crisis-2026/ - ERCOT Market Notice M-A080326-01 — Update Regarding Batch Zero Timelines and Processes — ERCOT · 2026-08-03 — https://www.ercot.com/services/comm/mkt_notices/M-A080326-01 - EIA expects record electricity generation in 2026 and 2027 (September 2026 STEO) — U.S. Energy Information Administration · 2026-09-09 — https://www.eia.gov/pressroom/releases/press592.php - Queued Up: 2026 Edition — Characteristics of Power Plants Seeking Transmission Interconnection As of the End of 2025 — Rand, Cheyette, Talley, Zhang, Gorman, Wiser, Seel, Jeong, Kahrl · 2026-06 — https://emp.lbl.gov/publications/queued-2026-edition-characteristics - PJM Capacity Auction Procures 138,318 MW of Generation Resources (2028/2029 Base Residual Auction) — PJM Interconnection · 2026-07-14 — https://www.pjm.com/-/media/DotCom/about-pjm/newsroom/2026-releases/20260714-pjm-capacity-auction-procures-138318-mw-of-generation-resources.pdf ## Capital Structure Canonical URL: https://menfem.com/kb/capital-structure Who is financing the buildout, and how the money is actually structured. --- title: "Capital Structure — Knowledge Base" last_compiled: "2026-09-24T12:30:00Z" source_count: 4 concept_count: 2 entity_count: 2 papers: 0 news: 0 --- # Capital Structure This rung asks who is paying for the AI buildout and how that money is actually arranged: the loans behind the neoclouds, the multi-year compute contracts the model labs sign, the backlog figures companies report, and the depreciation schedules that decide how fast a chip is written off. It sits between two neighbours. What gets built belongs to `datacenters`, and what it costs to run belongs to `inference-economics`. This rung is the part in between: what the money costs, and who is on the hook if the demand does not show up. ## What this rung prices Every token is served on a chip somebody borrowed money to buy. The interest on that loan, and the number of years the chip is written off over, are both baked into the hourly price of the chip, and so into the token price and the task price above it. Three of the rung's four primary sources are about that first cost, and the cleanest thing they show is how much it moved in **about four and a half months** (31 March to 10 August 2026). Across three CoreWeave loan facilities backed by the same kind of collateral, the interest margin over the benchmark rate went from **SOFR + 2.25% to + 4.50% to + 5.50%**, while the Moody's rating fell from **A3** to **Ba2** and the facility size shrank from **$8.5bn to $3.1bn to $2.6bn** ([DDTL 4.0](../raw/coreweave-ddtl-4-0-8-5bn-2026.md); [DDTL 5.0](../raw/coreweave-ddtl-5-0-3-1bn-2026.md); [DDTL 5.5](../raw/coreweave-ddtl-5-5-2-6bn-2026.md)). The cheapest of the three, DDTL 4.0, also had a fixed-rate slice at **about 5.9%**, the only all-in interest rate any source here gives. None of these sources turns that margin into a cost per GPU-hour or per token. That conversion is the thing the rung does not yet hold. ## The load-bearing findings - **Debt backed by GPUs got about 3.25 points more expensive in about four and a half months, and lenders kept lending.** The spread curve above only exists as a comparison: no single release contains it. *Corrected 2026-09-24:* this page and the DDTL 5.5 close read had dated DDTL 4.0 to 2025 and called the span "roughly a year". The DDTL 4.0 release, now read, closed on 2026-03-31. The honest complication is that each facility was "meaningfully oversubscribed", and DDTL 5.0 priced 0.50 points *tighter* than where it launched. So the curve does not say the market is closing on GPU debt. It says the market repriced this debt to a high-yield level and then bought it eagerly at that level ([DDTL 5.0](../raw/coreweave-ddtl-5-0-3-1bn-2026.md)). The three facilities also differ in structure, tenor and customer mix, so the direction is solid and the exact gaps are not a clean series ([DDTL 5.5](../raw/coreweave-ddtl-5-5-2-6bn-2026.md)). - **The first step on the curve is mostly about who rents the chips.** DDTL 4.0 finances a contract with a "leading AI enterprise" and is rated investment grade; DDTL 5.0 finances two customers its release calls below investment grade. A loan like this is rated largely on the customer paying for the chips, so most of the jump from + 2.25% to + 4.50% is probably a change of customer credit rather than the market turning. Neither release puts a number on the split ([DDTL 4.0](../raw/coreweave-ddtl-4-0-8-5bn-2026.md); [DDTL 5.0](../raw/coreweave-ddtl-5-0-3-1bn-2026.md)). - **The rung's first special-purpose vehicle.** DDTL 4.0 is non-recourse: the lenders' claim is on one ring-fenced subsidiary, CoreWeave Compute Acquisition Co. VIII, LLC, and its assets, not on CoreWeave as a whole ([DDTL 4.0](../raw/coreweave-ddtl-4-0-8-5bn-2026.md)). - **The newest loan outlasts the contracts it finances.** DDTL 5.5 runs about five years against customer contracts that average about three. CoreWeave's own words: "Lenders are now comfortable financing shorter-dated contracts." The facility allows for renewing those contracts or leasing the capacity to someone else when they end, which means part of the risk that a used GPU fleet is worth less than planned has moved onto the lender ([DDTL 5.5](../raw/coreweave-ddtl-5-5-2-6bn-2026.md)). - **The chips are the collateral, so the write-off schedule is part of the loan.** DDTL 5.0 is secured on the GPU and HPC assets, with the draw schedule matched to both deployment and the assets' useful life. That makes the depreciation assumption a direct input to the credit, and it is disclosed that both named customers behind that facility are below investment grade ([DDTL 5.0](../raw/coreweave-ddtl-5-0-3-1bn-2026.md)). - **On the other side of the ledger, a frontier lab is committing gigawatt-scale money to one vendor's custom chip while keeping two others.** Anthropic committed to up to one million Google TPUs and well over a gigawatt of capacity online in 2026, a deal worth tens of billions of dollars, while still running Amazon Trainium and NVIDIA GPUs. "Up to" is a ceiling, not a floor ([Anthropic–Google TPU deal](../raw/anthropic-google-tpu-1m-deal-2026.md)). A later expansion of up to 5 GW from 2027 is carried in the same file at moderate evidence, from a separate announcement that was not fetched directly. - **Backlog numbers are large, and they can count the same demand twice.** The [contracted backlog](./concepts/contracted-backlog-and-rpo.md) page holds Oracle's $523B of remaining performance obligations, CoreWeave's $99.4B backlog, and a named $21B Meta commitment sitting inside CoreWeave's backlog while Meta is itself tracked as a hyperscaler. That is one sourced case of the same end-demand booking at two layers; how much overlap there is across the whole stack is not quantified by any source. - **Whether early buyers keep a cost edge is modelled, not measured.** The [depreciation conveyor](./concepts/depreciation-conveyor.md) page carries a single-author preprint's model in which fully written-off fleets keep an incumbent ahead of a new entrant (3.2x in 2026, 1.9x in 2027, 3–4x by 2029–30). Every figure there is a model output, and the page records a competing source that attributes almost none of the fall in token prices to hardware at all. ## Concept map ```mermaid graph LR F[Customer credit] --> A[GPU-backed debt] A --> B[Depreciation conveyor] A --> C[Contracted backlog and RPO] C --> D[CoreWeave] C --> E[Oracle] D --> A ``` ## Concepts | Concept | Sources | Evidence | Last Updated | |---------|---------|----------|-------------| | [Contracted Backlog & RPO](concepts/contracted-backlog-and-rpo.md) | 4 (0 papers) | Weak to moderate | 2026-07-23 | | [The Depreciation Conveyor & Vintage Economics](concepts/depreciation-conveyor.md) | 2 (2 preprints) | Weak to moderate (modelled) | 2026-07-23 | ## Entities | Entity | Type | Sources | |--------|------|---------| | [CoreWeave](entities/coreweave.md) | Company | 5 | | [Oracle](entities/oracle.md) | Company | 3 | The four pages above were carried onto this rung from `datacenters` and `inference-economics`, and cite sources held in those topics (their links were repointed to those rungs on 2026-09-24). The CoreWeave page now carries all three DDTL facilities as a table; the two concept pages and the Oracle page do not yet fold in this rung's own sources. ## What we do not know yet - **What any of this costs per GPU-hour or per token.** The spread curve is a price of money, not a price of compute. No source here converts it. - **The terms that carry the real risk.** All three primary sources are press releases, not credit agreements: no advance rates, covenants, amortisation, draw conditions, collateral haircuts or residual-value assumptions ([DDTL 5.5](../raw/coreweave-ddtl-5-5-2-6bn-2026.md)). - **How much of the widening is customer credit and how much is the market.** One borrower, three customers, no split disclosed. - **Who the customers are.** DDTL 5.0's two customers and DDTL 4.0's "leading AI enterprise" are unnamed, so concentration cannot be assessed ([DDTL 5.0](../raw/coreweave-ddtl-5-0-3-1bn-2026.md); [DDTL 4.0](../raw/coreweave-ddtl-4-0-8-5bn-2026.md)). - **What useful life the GPUs are written off over.** The collateral is tied to it, and no source on this rung discloses it. - **The rest of the scope.** Vendor financing and circular or related-party deals are named in scope and have no source. SPVs now have one example (DDTL 4.0's ring-fenced subsidiary) and no analysis. - **Standing: read it.** Every figure here is read from company releases or other rungs' sources. The CoreWeave entity page records that we hold a position; that is exposure, not having run anything. ## Timeline See [timeline.md](timeline.md) for chronological developments, and [frontier.md](frontier.md) for the open questions. ## Sources | # | Title | Type | Date | Status | |---|-------|------|------|--------| | 1 | [Expanding Our Use of Google Cloud TPUs and Services](../raw/anthropic-google-tpu-1m-deal-2026.md) | technical-report | 2025-10-23 | compiled | | 2 | [CoreWeave Closes Landmark $8.5 Billion Financing Facility (DDTL 4.0)](../raw/coreweave-ddtl-4-0-8-5bn-2026.md) | technical-report | 2026-03-31 | compiled | | 3 | [CoreWeave Closes $3.1 Billion Loan Facility (DDTL 5.0)](../raw/coreweave-ddtl-5-0-3-1bn-2026.md) | technical-report | 2026-05-18 | compiled | | 4 | [CoreWeave Closes $2.6 Billion Loan Facility (DDTL 5.5)](../raw/coreweave-ddtl-5-5-2-6bn-2026.md) | technical-report | 2026-08-10 | compiled | ## Read next - [coreweave-ddtl-4-0-8-5bn-2026](../raw/coreweave-ddtl-4-0-8-5bn-2026.md) — the investment-grade first point, and why the first step is about the customer. - [coreweave-ddtl-5-5-2-6bn-2026](../raw/coreweave-ddtl-5-5-2-6bn-2026.md) — the spread curve in one table, and the five-year loan against three-year contracts. - [coreweave-ddtl-5-0-3-1bn-2026](../raw/coreweave-ddtl-5-0-3-1bn-2026.md) — why the widening spread is not a closing market. - [anthropic-google-tpu-1m-deal-2026](../raw/anthropic-google-tpu-1m-deal-2026.md) — the demand side: a lab signing for a gigawatt of one vendor's chip. ### Sources (4) - Expanding Our Use of Google Cloud TPUs and Services — Anthropic · 2025-10-23 — https://www.anthropic.com/news/expanding-our-use-of-google-cloud-tpus-and-services - CoreWeave Closes $2.6 Billion Loan Facility (DDTL 5.5) — CoreWeave, Inc. · 2026-08-10 — https://investors.coreweave.com/news/news-details/2026/CoreWeave-Closes-2-6-Billion-Loan-Facility-Expanding-Financing-Flexibility-for-AI-Infrastructure/default.aspx - CoreWeave Closes $3.1 Billion Loan Facility (DDTL 5.0) — CoreWeave, Inc. · 2026-05-18 — https://investors.coreweave.com/news/news-details/2026/CoreWeave-Closes-3-1-Billion-Loan-Facility-Expanding-Access-to-Public-Markets-for-GPU-Backed-Financing/default.aspx - CoreWeave Closes Landmark $8.5 Billion Financing Facility, Achieving First Investment-Grade Rated GPU-backed Financing (DDTL 4.0) — CoreWeave, Inc. · 2026-03-31 — https://www.sec.gov/Archives/edgar/data/1769628/000176962826000129/ex991.htm ## Inference & Token-Pricing Economics Canonical URL: https://menfem.com/kb/inference-economics The economics of serving models — what a token costs to make, and what it sells for. --- title: "Inference & Token-Pricing Economics — Knowledge Base" last_compiled: "2026-09-24T12:30:00Z" source_count: 17 concept_count: 8 entity_count: 5 --- # Inference & Token-Pricing Economics This rung is about what a token costs to make and what it sells for — token pricing trends, inference gross margins, the AI-lab profit question, and how memory, energy and hardware costs feed through to the price of a token. Its sharpest observation is that the two numbers are only loosely attached: Quanyan Zhu's tokenomics framework argues that tokens spent and value produced are different quantities, so pricing by tokens "prices the input, not the output" ([AI Tokenomics](../raw/ai-tokenomics-economics-tokens-computation-pricing.md), abstract read only, and explicitly theoretical — it carries no measured figures). > **Freshness, stated plainly (2026-09-24, 12:30).** Three new sources were ingested today, the > first since 2026-08-06, all on approval: OpenAI's own account of its July GPT-5.6 price > cut, Anthropic's Fable 5.1 cache-read cut, and Intercom's Fin price per outcome. They bring the > rung's newest price observations up to **September 2026**. They are vendor disclosures, not > measurements: the rung still holds no first-hand cost figure. What is still stale is in > [frontier.md](frontier.md). ## What this rung prices A quoted price per million tokens is the start of the sum, not the end. Three multipliers sit between it and the price of a finished job. First, the seller's side: the same GPU serving the same model has a very different cost per token depending on how busy it is, so a cost figure without a utilisation figure is not a cost figure. Second, the buyer's side: a task is only done when it is done *correctly*, so the unit that settles a purchase is cost per correct answer, not cost per token — a cheap model that is often wrong can cost more per finished job than an expensive one. Third, the decline: prices per token have fallen fast but very unevenly, landing on some tiers and tasks and not others, so "inference is getting cheaper" means nothing until you ask *which tier*. Get those three right and a token price turns into a task price; skip one and the answer is off by an order of magnitude or more. ## The load-bearing findings - **Same GPU, same model: cost moved 17–36× with load alone.** On one H100 setup priced at **$6.98/GPU-hour** (Azure H100 NVL on-demand list), effective cost ran from **$0.21 to $15.25 per million output tokens** across configurations; for a single model the spread from idle to full load was **17.5–36.3×**, and **2.5–24×** across an ordinary 1–10 requests-per-second load. An estimate that ignores utilisation understates cost by exactly 1/U, which makes self-hosting look cheaper than it is ([Beyond Per-Token Pricing](../raw/beyond-per-token-pricing-concurrency-aware-cost-methodology.md), full text). This is [the cost-measurement problem](./concepts/cost-measurement-problem.md) in one number. *(Earlier versions said "identical model, 73× spread"; the 73× mixes two different models.)* - **The right unit is cost per correct answer, and it halves every few months.** Cost-of-pass is cost per attempt divided by the chance the attempt is right; measured that way, the best available cost on hard maths-style tasks "roughly halved every few months", driven by better models rather than inference tricks ([Cost-of-Pass](../raw/cost-of-pass-economic-framework-language-models.md), abstract only). It also explains why an expensive reasoning model gets bought. See [cost per task](./concepts/cost-per-task.md). - **Agent tasks: cents at the cheap end, single-digit dollars at the top — unconfirmed.** Across 40 tasks and 11 frontier models, API cost per task reportedly ran **$0.17 to $6.55** ([STAGE-Claw](../raw/stage-claw-state-based-agent-benchmark-cost-per-task.md)). Those figures were surfaced by search from the paper's cost table, never read directly, and are still unconfirmed two months after ingest. List prices, not what anyone paid. - **The price decline is real, fast and unequal — the median hides the number you need.** The price of a fixed performance level fell at a **median 50× a year** across six benchmarks, range **9× to 900×**; the famous GPT-4-level science milestone fell **40× a year**, below the median ([Epoch AI](../raw/epoch-ai-llm-inference-price-trends.md), March 2025). Split by tier, the price half-life is about **1.10 years** for cheap models and **1.55 years** for mid-range ones, while flagship prices barely follow a trend at all — but the full text (read 2026-09-11) shows weak fits (the cheap-tier fit explains under a fifth of the variation), tiers defined by price so models move between them, and a series that **ends 2026-03-28** ([Tiered Super-Moore's Law](../raw/tiered-super-moores-law-inference-price-evolution.md)). See [the price-decline distribution](./concepts/price-decline-distribution.md). - **The decline lands on the lower tiers, not the frontier.** A workplace-agent benchmark re-run two years on saw best-agent completion go **43% → 98%** and harmful mistakes **26% → 1.9%**, and says open-weight models "drastically lowered" the cost of a once-proprietary performance level **while frontier costs stayed flat** ([WorkBench Revisited](../raw/workbench-revisited-workplace-agents-two-years-on.md), abstract only; a ~100× size has been attributed to the unread body). Provider prices showed that shape in July — but it did not last. OpenAI cut GPT-5.6's two lower tiers 20% and 80% three weeks after launch, then put the frontier model on a promotional $4/$20 (from $5/$30) on 2026-08-21, and launched GPT-6 Sol at $2/$10 on 2026-09-22 ([OpenAI](../raw/openai-gpt-5-6-price-cut-fast-mode-2026-07-30.md)). What holds now is a price point at the very top — $10/$50 at both OpenAI and Anthropic — with the named tiers falling beneath it. See [price-rung persistence](./concepts/price-rung-persistence.md), corrected 2026-09-24. - **A price cut with a named cause: software, on the same chips.** OpenAI says it cut Luna 80% and Terra 20% on 2026-07-30 because GPT-5.6, set to optimise its own serving, found **20% lower serving cost from GPU kernel work and 15%+ better token generation from speculative decoding** ([OpenAI](../raw/openai-gpt-5-6-price-cut-fast-mode-2026-07-30.md), vendor-stated percentages, no dollar base). The 20% explains Terra's cut; Luna's 80% is mostly a pricing choice. - **The price of an agent's task fell while the headline rate stood still.** Claude Fable 5.1 kept $10/$50 per million tokens and cut only cache reads — re-used stored context — by 75%, $1.00 → $0.25. Anthropic estimates ~25% off a typical bill and up to ~45% off agent-heavy ones ([Anthropic](../raw/anthropic-claude-fable-5-1-cache-read-price-cut-2026-09-01.md)). On this rung it counts only because the cache line *is* the mechanism; plain list-price moves were ruled out of scope on 2026-09-13. See [inference & token economics](./concepts/inference-token-economics.md). - **The first published price per task: $0.99 per outcome.** Intercom sells its Fin support agent by the finished job, charged once per conversation however many steps it takes (a sales qualification is $9.99) — the token count is hidden inside the price, and its fall accrues to the seller ([Intercom](../raw/intercom-fin-outcome-pricing-2026-09.md), vendor pages as of 2026-09-24). See [cost per task](./concepts/cost-per-task.md). - **Nobody on this rung can tell you what a token costs a lab to make.** Benedict Evans puts inference at **40–50% gross margin** but leaves out training, "currently far larger than revenue"; his conclusion is that the plumbing-or-lasting-margin question is unresolved ([Ways to Think About Token Pricing](../raw/ben-evans-ways-to-think-about-token-pricing.md)). That is [the margin question](./concepts/token-pricing-margin-question.md), still open. - **At the thin-margin layer, price is still a product choice, not a cost floor.** For one model (Llama 4 70B output), Q2-2026 list prices spread ~6× — **$0.65** on Together's batch tier to **$4.20** on Cerebras — tracking latency, service guarantees and hardware rather than cost to serve ([Q2 2026 provider pricing matrix](../raw/independent-inference-provider-pricing-matrix-q2-2026.md), an analysis-grade blog, one model, list prices). ## What has changed since July — from the sibling rungs Between 2026-08-06 and 2026-09-24 this rung took in no new source. Its neighbours did, and some of what they now hold bears directly on token cost. These are **context with their own links, not this rung's evidence**: - **`serving`** holds the KB's two measured cost-to-serve results: DigitalOcean's H200 at **$20.32 → $0.45 per million output tokens** from batch 1 to batch 128 ([raw](../../serving/raw/measured-h200-token-serving-cost-digitalocean.md), 2026-07-08) and Epoch's capacity model — serving capacity growing ~3.4×/year against demand ~10× ([raw](../../serving/raw/epoch-inference-compute-crunch-serving-capacity.md)). Both were once compiled here; they are registered on `serving`. It also now holds the KB's first goodput result: splitting prefill from decode on the same eight GPUs raised requests meeting their service level from 26 to 73 per 100 ([raw](../../serving/raw/vllm-prefill-decode-disaggregation-moriio-goodput.md)). - **`harnesses`** holds a controlled cost-per-task result: changing only the orchestration layer cut cost per task **$0.21 → $0.12 (−41%)** at equal quality ([raw](../../harnesses/raw/harness-effect-orchestration-token-economics.md), 2026-07-08) — the harness moved the task price more than the choice of model did. - **`compilers`** holds kernel-level gains on unchanged chips — FlashMLA's FP8 decode kernel 1.64× faster on Hopper ([raw](../../compilers/raw/flashmla-fp8-sparse-decode-dsm-crossover.md)), FlashAttention-4 at 71% utilisation on B200 ([raw](../../compilers/raw/flashattention-4-blackwell-kernel-pipelining.md)) — the kind of software mechanism behind "the cost fall came from software, not silicon". - **`capital-structure`** holds the cost of money behind GPU fleets: CoreWeave borrowed at SOFR+450 in May and SOFR+550 in August 2026 ([May](../../capital-structure/raw/coreweave-ddtl-5-0-3-1bn-2026.md), [August](../../capital-structure/raw/coreweave-ddtl-5-5-2-6bn-2026.md)) — a widening spread that feeds straight into the per-hour cost of a GPU — and the [depreciation conveyor](../../capital-structure/wiki/concepts/depreciation-conveyor.md) concept this rung used to hold. - **`energy`** holds record US generation forecasts (EIA, 2026-09-09) alongside a Texas pause on connecting new data-centre load (ERCOT, 2026-08-03): energy plentiful in total, power scarce at the site ([EIA](../../energy/raw/eia-steo-record-generation-2026-2027.md), [ERCOT](../../energy/raw/ercot-batch-zero-timeline-pause-2026.md)). - **`hardware`** holds SK hynix's 76% operating margin for Q2 2026 — memory scarcity stated as a margin ([raw](../../hardware/raw/sk-hynix-2q26-financial-results.md)). ## What we do not know yet - **Almost nothing here was measured first-hand.** The one own-dataset source is a Token Price Index reading of a published pricing page, not a cost measurement, and that index is retired. No cost curve on this rung has been reproduced locally; `vllm-cost-meter` has not been run. - **Five of the seventeen sources were read at abstract level only** — Cost-of-Pass, AI Tokenomics, STAGE-Claw, WorkBench Revisited and Photons = Tokens — and two of them carry the rung's task-price numbers ($0.17–$6.55 per task; the open-weight-versus-frontier cost split). - **No measured cost for any frontier model exists anywhere in the KB.** Every lab-margin figure stays uncheckable until one does. - **Two sources disagree about what drives the price fall, and the rung cannot settle it.** Du attributes ~103.7% of the cost fall to productivity and −0.9% to GPU hardware — a residual, but a pointed one. OpenAI's own account of its July cut (kernels and speculative decoding, no new chips) now sides with Du on one case, as a vendor claim. Matsuoka builds his model on memory and hardware vintage ([Memory Scarcity](../raw/memory-scarcity-open-models-ai-industry-restructuring-2026-2030.md), reconciled to the `hardware` rung's full close read). Neither has been checked against the other on shared data. - **List prices are not paid prices, anywhere on this rung.** Caching discounts, batch tiers and negotiated rates are invisible. The gap is unmeasured. - **The macro channel rests on a stipulated number.** The Inference-Cost Phillips Curve's size term (λ̄ = 0.18, i.e. 18% of firms' marginal cost being AI inference over 2022–2026) has no stated source, and its cost series ignores utilisation ([Phillips Curve](../raw/economics-of-ai-inference-phillips-curve.md)). See [the macro channel](./concepts/inference-cost-macro-channel.md). - **The energy leg is top-down only.** 326 TWh of projected 2028 US AI energy could support ~6.5×10¹⁷ tokens a year — energy is not the near-term limit on token *volume* — but there is no $/kWh-to-$/token conversion for any real chip ([Photons = Tokens](../raw/photons-equals-tokens-physics-ai-economics-knowledge.md), abstract only). See [the energy cost of tokens](./concepts/energy-cost-of-tokens.md). - **The measurement vocabulary is agreed but unused.** Goodput — requests per second meeting a stated service level — is defined ([LLM Inference Handbook](../raw/modular-llm-inference-handbook.md), a vendor reference with no measurements), but no cost figure on this rung is quoted against it. ## Concept Map ```mermaid graph LR M[Margin question] --> C[Cost-measurement problem] M --> D[Price-decline distribution] M --> R[Price-rung persistence] C --> T[Cost per task] D --> T D --> R E[Energy cost of tokens] --> C X[Macro channel] --> M I[Inference & token economics] --> D I --> R ``` ## Concepts | Concept | Sources on this rung | Evidence | Last Updated | |---------|---------|----------|-------------| | [Token Pricing & the Inference-Margin Question](concepts/token-pricing-margin-question.md) | 11 (0 peer-reviewed) | Moderate — still undecidable | 2026-09-24 | | [The Cost-Measurement Problem](concepts/cost-measurement-problem.md) | 5 | Moderate (one full-text measured preprint) | 2026-09-24 | | [Cost Per Task (Cost-of-Pass)](concepts/cost-per-task.md) | 5 (4 abstract-read + 1 published price) | Weak-moderate; one published task price | 2026-09-24 | | [The Price-Decline Distribution](concepts/price-decline-distribution.md) | 5 | Strong for direction, weak for precision | 2026-09-24 | | [Price-Rung Persistence](concepts/price-rung-persistence.md) | 4 | Dated archive — every GPT-5.6 rung now broken | 2026-09-24 (correction box) | | [Inference & Token Economics](concepts/inference-token-economics.md) | 3 (+4 on sibling rungs) | Moderate | 2026-09-24 | | [The Energy Cost of Tokens](concepts/energy-cost-of-tokens.md) | 1 (abstract-read) | Weak-moderate, early | 2026-09-24 | | [Inference Cost as a Macro Input](concepts/inference-cost-macro-channel.md) | 1 | Weak, early | 2026-09-24 | ## Entities | Entity | Type | Sources | |--------|------|---------| | [OpenAI (pricing)](entities/openai-pricing.md) | Company (pricing lens) | 3 | | [Anthropic (economics)](entities/anthropic-economics.md) | Company (economics lens) | 2 (+1 culled) | | [Independent Inference Providers](entities/independent-inference-providers.md) | Market segment | 1 | | [vllm-cost-meter](entities/vllm-cost-meter.md) | Product (instrument) | 1 | | [Intercom (Fin)](entities/intercom-fin.md) | Product (task-priced agent) | 1 | ## Timeline See [timeline.md](timeline.md) for the dated sequence, and [frontier.md](frontier.md) for what is stale and what would refresh it. ## Sources | # | Title | Type | Published | Read depth | Status | |---|-------|------|-----------|------------|--------| | 1 | [Ways to Think About Token Pricing](../raw/ben-evans-ways-to-think-about-token-pricing.md) | analysis | 2026-07-09 | full essay | compiled | | 2 | [LLM Inference Prices Have Fallen Rapidly but Unequally](../raw/epoch-ai-llm-inference-price-trends.md) | technical-report | 2025-03-12 | full data insight | compiled | | 3 | [Token Price Index — GPT-5.6 rung hold](../raw/tpi-gpt-5-6-rung-hold-2026-07-16.md) | technical-report (own dataset) | 2026-07-16 | primary observation | compiled | | 4 | [Memory Scarcity, Open Models, and the Restructuring of the AI Industry](../raw/memory-scarcity-open-models-ai-industry-restructuring-2026-2030.md) | preprint | 2026-07-08 | reconciled to `hardware` full read | compiled | | 5 | [Beyond Per-Token Pricing](../raw/beyond-per-token-pricing-concurrency-aware-cost-methodology.md) | preprint | 2026-06-10 | full text | compiled | | 6 | [The Economics of AI Inference (Phillips Curve)](../raw/economics-of-ai-inference-phillips-curve.md) | preprint | 2026-05-19 | full text | compiled | | 7 | [Tiered Super-Moore's Law](../raw/tiered-super-moores-law-inference-price-evolution.md) | preprint | 2026-03-30 | full text (2026-09-11) | compiled | | 8 | [Cost-of-Pass](../raw/cost-of-pass-economic-framework-language-models.md) | preprint | 2025-04-17 (rev. 2026-02-26) | abstract | compiled | | 9 | [AI Tokenomics](../raw/ai-tokenomics-economics-tokens-computation-pricing.md) | preprint | 2026-06-10 | abstract | compiled | | 10 | [Q2 2026 Provider Pricing Matrix](../raw/independent-inference-provider-pricing-matrix-q2-2026.md) | analysis | 2026-04-24 | full article | compiled | | 11 | [STAGE-Claw](../raw/stage-claw-state-based-agent-benchmark-cost-per-task.md) | preprint | 2026-06-09 | abstract + search-surfaced figures | compiled | | 12 | [WorkBench Revisited](../raw/workbench-revisited-workplace-agents-two-years-on.md) | preprint | 2026-06-10 | abstract | compiled | | 13 | [Photons = Tokens](../raw/photons-equals-tokens-physics-ai-economics-knowledge.md) | preprint | 2026-02-23 | abstract | compiled | | 14 | [LLM Inference Handbook](../raw/modular-llm-inference-handbook.md) | technical-report (vendor) | undated | landing + metrics chapter | compiled 2026-09-24 | | 15 | [OpenAI GPT-5.6 price cut (Luna −80%, Terra −20%)](../raw/openai-gpt-5-6-price-cut-fast-mode-2026-07-30.md) | technical-report (vendor) | 2026-07-30 | forum post, changelog, pricing page in full; blog post not readable (403) | compiled 2026-09-24 | | 16 | [Anthropic Claude Fable 5.1 cache-read cut](../raw/anthropic-claude-fable-5-1-cache-read-price-cut-2026-09-01.md) | technical-report (vendor) | 2026-09-01 | product page + pricing page in full | compiled 2026-09-24 | | 17 | [Intercom Fin, $0.99 per outcome](../raw/intercom-fin-outcome-pricing-2026-09.md) | technical-report (vendor) | ~2026-09-23 (page update) | help article + pricing FAQ in full; proposed article 401 | compiled 2026-09-24 | By type: 9 preprints · 6 technical reports (4 of them vendor disclosures) · 2 analyses · **0 peer-reviewed papers**. ## Read next - [beyond-per-token-pricing-concurrency-aware-cost-methodology](../raw/beyond-per-token-pricing-concurrency-aware-cost-methodology.md) — the only fully-read, measured cost study on the rung, and the reason every list-price cost claim here is soft. - [cost-of-pass-economic-framework-language-models](../raw/cost-of-pass-economic-framework-language-models.md) — the framework that turns a token price into a task price. - [tiered-super-moores-law-inference-price-evolution](../raw/tiered-super-moores-law-inference-price-evolution.md) — the price-decline paper, now read in full, with the caveats that come with it. ### Sources (17) - Ways to Think About Token Pricing — Benedict Evans · 2026-07-09 — https://www.ben-evans.com/benedictevans/2026/7/9/ways-to-think-about-token-pricing - LLM Inference Prices Have Fallen Rapidly but Unequally Across Tasks — Ben Cottier, Ben Snodin, David Owen, Tom Adamczewski · 2025-03-12 — https://epoch.ai/data-insights/llm-inference-price-trends - Token Price Index — GPT-5.6 holds the prior-generation price rungs — MenFem Token Price Index (own dataset) · 2026-07-16 — https://developers.openai.com/api/docs/pricing - Memory Scarcity, Open Models, and the Restructuring of the AI Industry, 2026-2030 — Satoshi Matsuoka · 2026-07-08 — https://arxiv.org/abs/2607.07207 - Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation — Chitral Patil · 2026-06-10 — https://arxiv.org/abs/2606.11690 - The Economics of AI Inference: Inflation Dynamics, Welfare Costs, and Optimal Monetary Policy under the Inference-Cost Phillips Curve — Gustav Olaf Yunus Laitinen-Fredriksson Lundström-Imanov · 2026-05-19 — https://arxiv.org/abs/2605.20281 - Tiered Super-Moore's Law: Price Evolution, Production Frontiers, and Market Competition in LLM Inference Services — Mingdeng Du · 2026-03-30 — https://arxiv.org/abs/2603.28576 - Cost-of-Pass: An Economic Framework for Evaluating Language Models — Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, James Zou · 2025-04-17 — https://arxiv.org/abs/2504.13359 - AI Tokenomics: The Economics of Tokens, Computation, and Pricing in Foundation Models — Quanyan Zhu · 2026-06-10 — https://arxiv.org/abs/2606.24616 - AI Inference Providers Compared: Q2 2026 Pricing Matrix (independent-provider observed pricing) — Digital Applied Team · 2026-04-24 — https://www.digitalapplied.com/blog/ai-inference-providers-pricing-matrix-q2-2026 - STAGE-Claw: Automated State-based Agent Benchmarking for Realistic Scenarios — Liang, Yu, Wang, Guo, Hu, Cao, Zhao, Liu, Zeng, Cai, Liu · 2026-06-09 — https://arxiv.org/abs/2606.10394 - WorkBench Revisited: Workplace Agents Two Years On — Olly Styles, Sam Miller · 2026-06-10 — https://arxiv.org/abs/2606.13715 - Photons = Tokens: The Physics of AI and the Economics of Knowledge — Alec Litowitz, Nick Polson, Vadim Sokolov · 2026-02-23 — https://arxiv.org/abs/2603.06630 - LLM Inference Handbook — Modular — https://handbook.modular.com - OpenAI cuts GPT-5.6 Luna 80% and Terra 20%, and says why (announcement + API changelog + pricing page) — OpenAI · 2026-07-30 — https://community.openai.com/t/announcing-a-major-price-drop-for-5-6-terra-and-luna-and-fast-mode-for-5-6-sol/1388484 - Anthropic, Introducing Claude Fable 5.1: cache reads cut 75% — Anthropic · 2026-09-01 — https://www.anthropic.com/claude/fable - Intercom, Fin AI Agent outcomes: $0.99 per outcome — Intercom · 2026-09-23 — https://www.intercom.com/help/en/articles/8205718-fin-resolutions --- Cite the canonical URL of the rung or source you used. — MenFem