vllm-cost-meter

Type: Product — open-source measurement tool (inference-economics lens: the instrument)

Released alongside Patil's concurrency-aware cost preprint (arXiv 2606.11690, 2026-06-10). It attaches to a live vLLM server and reports real $/M-tokens against the operator's own traffic, rather than against a preset or a user-supplied utilization assumption.

It is logged as an entity because this topic's central defect — the cost-measurement problem — is a measurement defect, and this is the only named instrument in the KB that addresses it directly. The topic used to own one instrument on the price side (the MenFem Token Price Index, retired 2026-08-19 and now a closed archive). It has never owned one on the cost side. This is the candidate — and as of 2026-09-24 it has still not been run, nor its repository inspected.

What it would settle, if run. MenFem's core bet routes through cost per task. Patil's finding is that cost per task is not a lookup — it moves up to 36.3x on identical H100 hardware purely with offered request rate. A cost-per-task number sourced from a calculator is therefore an estimate of someone else's traffic shape. A meter attached to a live server against real traffic is the only way to produce one that means anything.

What is now known (2026-07-23, full text read). The repository is github.com/pChitral/vllm-cost-meter (stated in the arXiv Comments field). It attaches to a live vLLM server's Prometheus metrics and reports real $/M-tokens against the operator's own load; the paper's own figures are computed at $6.98/GPU-hr (Azure H100 NVL on-demand list). What is still not verified: licence, maturity, maintenance status, and whether it has been used outside the paper — the repo itself has not been inspected. Do not present it as a vetted tool; a look at the repository is the correct step before any run.

Key Contributions (to this topic)

• A live-traffic cost meter, not a calculator — attaches to a running vLLM server and reports real $/M-tokens against the operator's own load. Evidence: moderate (stated in the preprint, full text read; repo not inspected) (Patil)
• Operationalises C_eff = f(H, M, Q, lambda, L) — makes offered request rate λ, the variable no surveyed open-source calculator exposes, a first-class measured input. Evidence: moderate (Patil)
• Validated across 42 benchmarks spanning dense, ultra-sparse MoE and sparse MoE models, with a 56-run A100 replication sweep. Evidence: moderate (Patil)

Mentioned In

• The Cost-Measurement Problem — the defect this instrument exists to fix.

Related Entities

• OpenAI (pricing) — the price side of the same fraction; published prices are observable, costs are not.
• Anthropic (economics) — the margin claim that this class of instrument would be needed to check.

Changelog

• 2026-09-24 — Noted that the Token Price Index is retired and that the meter has still not been run or its repository inspected. No other change.
• 2026-07-23 — Full-text touch-up: repo URL identified (github.com/pChitral/vllm-cost-meter); reads live vLLM Prometheus metrics; paper's cost basis is $6.98/GPU-hr (Azure H100 NVL on-demand). Repo internals still not inspected.
• 2026-07-22 — Created from the Patil preprint ingested this compile. First cost-side instrument logged in this topic; explicitly marked unverified (abstract only, repo not inspected).