The Cost-Measurement Problem

Every margin claim on this rung is a fraction. The top half — the price per token — is published and easy to read. The bottom half — what that token cost to make — is measured almost nowhere. "Inference runs at 40–50% gross margin" (Evans, 2026-07-09) is a statement about a cost that the essay never defines: it gives no utilisation, no cost unit and no as-of date. This page exists to keep that hole visible, because every other question on the rung sits on top of it.

The one fully-read measurement this rung holds shows how big the hole is. On the same GPU, the cost of a million output tokens moved by more than an order of magnitude depending only on how busy the GPU was. A cost figure without a utilisation figure is not a cost figure.

What changed since July (dated 2026-09-24). Two measured cost sources that this page used to lean on — DigitalOcean's H200 benchmark and Epoch's serving-capacity model — are registered on the serving rung, not this one. They are kept below as see also context with their own links, not as this rung's evidence. The earlier claim on this page that "a measured cost floor now exists on this rung" no longer holds: on this rung's own shelf, the cost side is still one preprint (Patil) and one vocabulary (the Modular handbook).

1. The per-token cost is a property of your traffic, not of the hardware

Patil (arXiv 2606.11690, 2026-06-10, full text read) surveyed public LLM cost calculators — Helicone, LiteLLM, YourGPT, llm-prices.com and the LLM Inference TCO Calculator — and found that every one treats GPU utilisation as a fixed number the user types in, a preset, or a silent 100%. None measures it against real load. That assumption is not a rounding error; it is the biggest term in the answer.

On one H100 setup, priced at $6.98 per GPU-hour (Azure H100 NVL on-demand list, a $13.96/hr two-GPU node split between co-tenants), effective cost ran from $0.21 to $15.25 per million output tokens in the June 2026 test window. Read that range carefully:

• $0.21 and $15.25 are different models. $0.21 is Qwen3-30B-A3B in FP8 at full load; $15.25 is Mixtral 8x7B in FP16 at one request per second. The 73× gap between them mixes model choice with load.
• The same-model spread is the cleaner number: 17.5× to 36.3× from idle to full load (17.5× for Mixtral FP16, 36.3× for Qwen3-30B-A3B FP8). Across an ordinary 1–10 requests-per-second enterprise load the penalty is 2.5–24×.
• The driver is one variable the operator controls — the offered request rate — which sets how many requests are in flight at once (Little's Law). No open-source calculator exposes it.
• The bias has a clean formula: an estimate that ignores utilisation understates true cost by exactly 1/U. That makes self-hosting look cheaper than it is, worst of all for low-traffic teams — the ones most likely to be weighing it.
• It repeats on older silicon: on A100 80GB PCIe ($3.67/GPU-hr, 56 runs) the load-driven spread is 7.0–11.4×.

Carry the price basis whenever the dollar level is quoted: reserved or spot pricing multiplies it by roughly 0.3–0.7×, and a dedicated single-tenant node roughly doubles the single-GPU figures. The ratios are the assumption-light finding. Patil also released vllm-cost-meter, which attaches to a live vLLM server and reports cost against real traffic.

2. Cost against what? Goodput, not throughput

The LLM Inference Handbook (Modular, read 2026-08-06) supplies the vocabulary a cost claim needs before it can be checked. The key word is goodput: requests completed per second while meeting a stated service level (for example, first token within 200 ms and the whole answer within 3 seconds). Throughput counts all work done; goodput counts only work done acceptably.

Why it matters here: a cost per token computed against raw throughput flatters any system that is fast but too laggy to use. Batching is the lever behind most "our inference is cheaper" claims — bigger batches raise total tokens per second and lower the cost per token, but they also raise the wait for each user. So any vendor's cost per token is one point chosen on a speed-versus-cost curve, and without the batch size and the service level it cannot be compared with another vendor's.

Honest limits: the handbook is a vendor reference (Modular sells an inference stack), every number in it is an illustration, and its latency thresholds ("under 500 ms feels responsive") are rules of thumb with no cited research. Cite it for a definition, never for a figure. No cost figure on this rung is quoted against a goodput denominator — Patil's sweeps ran without a latency cutoff, which the paper itself names as future work.

3. The unit itself may be wrong — $/PB instead of $/token

Matsuoka (arXiv 2607.07207, 2026-07-08) prices inference in dollars per petabyte of memory bandwidth delivered, arguing that for the decode step — generating output tokens one at a time, which is limited by memory bandwidth — that unit is the same whichever model you run. What you really buy is bandwidth-seconds; tokens are a model-dependent way of packaging them. Two providers charging the same per million tokens can be selling very different amounts of the underlying resource.

This is a rival unit, not a refinement, and the rung records it as unresolved. Its scope is decode only; the closed form (paper Eq. 1) assumes 100% uptime, so every $/PB figure is a best-case floor. This rung has not read the PDF itself; it relies on the hardware rung's full close read (hardware close-read).

4. The demand side of the fraction is suspect too

Matsuoka also argues that public token trackers overstate the demand anyone pays for, and that every demand projection made before Q2 2026 predates the industry's shift from maximising tokens to minimising them. He states the direction, not the size. His sharpest practical suggestion: measure the solvency question in dollars, not tokens — the warning sign is "tokens growing while dollars stall".

So a margin claim has to survive three separate measurement problems at once: an unmeasured cost (Patil), a disputed cost unit (Matsuoka's $/PB), and an inflated demand count (Matsuoka's tracker critique).

5. Where an observed price meets a cost

The Q2 2026 provider matrix (Digital Applied, published 2026-04-24) is the only list-price source on this rung that can be set beside a cost. Its own text notes that Together's $0.65 per million batch tokens for a 70B open model sits at a measured cost-to-serve of about $0.45–$0.64 for a 70B model at high load — thin-to-zero margin at the batch tier, with real margin only at the faster, SLA-backed tiers. That measured cost is DigitalOcean's, registered on serving (see below). The comparison is open 70B models only and says nothing about frontier-model economics.

See also — sibling rungs (context, not this rung's evidence)

• serving — DigitalOcean's measured H200 cost (2026-07-08): raw close-read. One H200 at $3.44/GPU-hr running Llama-3.3-70B FP8 on vLLM moved from $20.32 per million output tokens at batch 1 to $0.45 at batch 128 — about 44× from traffic shape alone — with an own-a-GPU-versus-serverless break-even near 73% utilisation. It is the independent dollar reproduction of Patil's point on different silicon. One SKU, one open model, one date.
• serving — Epoch's serving-capacity model (2026-05-25): raw close-read. Calibrated to 111 real InferenceX runs; decode runs at about 30% of peak memory bandwidth, and serving capacity grows about 3.4× a year against demand of about 10× — a supply gap that is one reason list prices can stay sticky. Tokens per second, no dollars.
• serving — prefill/decode disaggregation measured in goodput (2026-04-07): raw close-read. On the same eight MI300X GPUs, requests meeting the service level rose from 26 to 73 per 100 at 8 req/s — 2.5× goodput from a scheduling change. The first goodput-denominated result in the KB, and a direct example of section 2's point: the cost per acceptable token moved with no new hardware. Concept page: prefill-decode disaggregation.
• capital-structure — the depreciation conveyor: concept page. Matsuoka's argument that incumbents stay cheaper than entrants because amortised fleets keep arriving — the "whose cost is lower, and why" half of this page.

Why this is the rung's load-bearing concept

The rung's through-line is converting a token price into a task price, and the cost per task is a cost. If cost moves 17–36× on the same GPU with traffic shape, then cost per task is not a number you can look up; it is a number you have to measure on your own load. That is the whole argument for running an instrument rather than citing a calculator — and no one has run one here yet.

Key Claims

• Every public LLM cost calculator surveyed (Helicone, LiteLLM, YourGPT, llm-prices.com, LLM Inference TCO) treats GPU utilisation as a fixed input, never measured against load. Evidence: moderate — full text read; single-author preprint (Patil)
• On one H100 setup at $6.98/GPU-hr (Azure H100 NVL on-demand list), effective cost ran $0.21–$15.25 per million output tokens across configurations (June 2026); for a single model the idle-to-full-load spread is 17.5–36.3×. Evidence: moderate — measured throughput × a named list price (Patil)
• Underutilisation penalty 2.5–24× across 1–10 requests per second. Evidence: moderate (measured, 42 H100 benchmarks) (Patil)
• An estimate that ignores utilisation understates cost by exactly 1/U. Evidence: strong (a mathematical identity) (Patil)
• The spread repeats on A100 80GB PCIe at 7.0–11.4× (56 runs). Evidence: moderate (Patil)
• FP8 helps the tested mixture-of-experts models about 2.2–2.4× more than the dense model (+69 to +74% vs +31% peak throughput). Evidence: weak — n=3, author asks for broader validation; the dense-model FP8 gain reverses on chips without native FP8 (Patil)
• Goodput — requests per second meeting a stated service level — is the denominator that stops a cost figure flattering a fast-but-unusable system. Evidence: strong as a definition; the source is a vendor reference with no measurements (Modular handbook)
• $/PB of bandwidth delivered is proposed as a model-independent cost unit for decode. Evidence: moderate (preprint; scope-limited to decode; best-case floor) (Matsuoka)
• Public token trackers overstate paid demand; pre-Q2-2026 projections predate the shift to token minimisation. Evidence: moderate (preprint claim; direction only, no magnitude) (Matsuoka)
• The cheapest observed independent-provider list price (Together, $0.65/M batch, 70B) sits at a measured 70B cost-to-serve of ~$0.45–$0.64/M. Evidence: moderate — list price observed on this rung; the cost figure is the serving rung's DigitalOcean measurement (Digital Applied)
• Evans' 40–50% inference gross margin states no utilisation, cost unit or as-of date. Evidence: strong (the absence is visible in the source) (Evans)
• No measured cost for any frontier or proprietary model exists on this rung or its siblings. Evidence: strong (absence, checked across the rung and serving on 2026-09-24)

Benchmarks & Data

Every figure carries its as-of date. Cost numbers go stale fast.

Metric	Value	As of	Nature	Source
Effective cost, one H100 setup, across configs	$0.21 – $15.25 / M output tokens	2026-06	measured × $6.98/GPU-hr list	Patil
Same-model spread, idle → full load	17.5× (Mixtral FP16) – 36.3× (Qwen3-30B-A3B FP8)	2026-06	measured ratio	Patil
Underutilisation penalty, 1–10 rps	2.5 – 24×	2026-06	measured ratio	Patil
Llama 3.1 8B FP16 cost at 1 / 10 / 100 rps	$7.60 / $0.79 / $0.31 per M	2026-06	measured × list	Patil
Load-driven spread, A100 80GB PCIe	7.0 – 11.4× (56 runs, $3.67/GPU-hr)	2026-06	measured ratio	Patil
FP8 peak-throughput gain, MoE / dense	+69 to +74% / +31% (n=3)	2026-06	measured, low-n	Patil
Cheapest independent-provider list (Together batch, 70B)	$0.65 / M output tokens	Q2 2026	observed list	Digital Applied
Inference gross margin (industry)	40–50%, training excluded	2026-07-09	asserted, no method	Evans
Context: one H200, 70B FP8, batch 1 → 128	$20.32 → $0.45 / M	2026-07-08	measured — serving rung	DigitalOcean
Context: goodput from prefill/decode split, 8× MI300X	26 → 73 SLO-meeting requests per 100	2026-04-07	measured — serving rung	vLLM / AMD

Conflicts on the Record

• Unit conflict. Per token versus per petabyte of bandwidth. Everything else on this rung is per token; nothing reconciles the two. Open.
• Method conflict. Patil's 17–36× same-model spread does not prove Evans' 40–50% wrong; it makes it uncheckable as stated. The same applies to an attributed "70%+" Anthropic margin that this rung once carried — its source (a podcast) was culled from the KB on 2026-08-06, so it is no longer evidence here at all (see Anthropic (economics)).
• Demand conflict. Matsuoka's "trackers overstate demand" runs against the bank and analyst demand forecasts logged in the frontier file on 2026-07-17. Those forecasts are located web evidence, not registered sources. Open.

Open Questions

• What does a frontier or proprietary model cost to serve? Nothing on this rung or on serving measures it. Until something does, every lab-margin figure stays uncheckable.
• Can the one-off measurements become a series? Patil (H100/A100, June 2026) and DigitalOcean (H200, July 2026) are single points. No Blackwell, TPU or Trainium cost; no long-context or reasoning-workload cost.
• What does any of this cost at a stated service level? No cost figure here is priced against goodput.
• Is $/PB model-independent beyond decode — for prefill, long context and reasoning, the growing share?
• How large is the token-tracker overstatement? Direction only so far.
• What would it take to measure MenFem's own cost per task? vllm-cost-meter on a live server; the repository itself has not yet been inspected.

Related Concepts

• Token Pricing & the Inference-Margin Question — the umbrella question; it cannot be answered without this page's missing number.
• The Price-Decline Distribution — measures price, not cost; this page is why the two must not be confused.
• Cost Per Task (Cost-of-Pass) — needs a per-attempt cost, which is exactly the number this page says is missing.
• Price-Rung Persistence — the price side, which is published and so unaffected by this problem.
• Independent Inference Providers — the list price that sits on a measured cost.

Backlinks

Pages that reference this concept:

• Token Pricing & the Inference-Margin Question
• The Price-Decline Distribution
• Cost Per Task
• The Energy Cost of Tokens
• Inference Cost as a Macro Input
• vllm-cost-meter

Changelog

• 2026-09-24 (recompile, no new source) — Removed DigitalOcean and Epoch from this page's own evidence: both are registered on serving, and the page was citing raw files that no longer exist on this rung. They now sit under "See also" with working links; their slugs stay in the frontmatter sources, where the site resolves each to the serving ledger that owns it. Corrected the headline spread: the $0.21–$15.25 range spans different models (73× mixes model and load); the same-model spread is 17.5–36.3×. Corrected "survey not enumerated" — the full text names the calculators. Compiled the Modular handbook (ingested 2026-08-06, never compiled) as a new goodput section. Added the serving goodput result as context. Removed the culled Patel figure from the claims. Fixed the depreciation-conveyor link (the page lives on capital-structure).
• 2026-07-23 — Added DigitalOcean's H200 cost and Epoch's capacity model (both since moved to serving), and the Together-batch cross-check. Patil's $/GPU-hour input found: $6.98.
• 2026-07-22 — Created from Patil (2606.11690) and Matsuoka (2607.07207).