vllm-cost-meter

Type: Product — open-source measurement tool (inference-economics lens: the instrument)

Released alongside Patil's concurrency-aware cost preprint (arXiv 2606.11690, 2026-06-10). It attaches to a live vLLM server and reports real $/M-tokens against the operator's own traffic, rather than against a preset or a user-supplied utilization assumption.

It is logged as an entity because this topic's central defect — the cost-measurement problem — is a measurement defect, and this is the only named instrument in the KB that addresses it directly. The topic already owns one instrument on the price side (the MenFem Token Price Index, which reads published per-token prices to the cent). It owns nothing on the cost side. This is the candidate.

What it would settle, if run. MenFem's core bet routes through cost per task. Patil's finding is that cost per task is not a lookup — it moves up to 36.3x on identical H100 hardware purely with offered request rate. A cost-per-task number sourced from a calculator is therefore an estimate of someone else's traffic shape. A meter attached to a live server against real traffic is the only way to produce one that means anything.

What is now known (2026-07-23, full text read). The repository is github.com/pChitral/vllm-cost-meter (stated in the arXiv Comments field). It attaches to a live vLLM server's Prometheus metrics and reports real $/M-tokens against the operator's own load; the paper's own figures are computed at $6.98/GPU-hr (Azure H100 NVL on-demand list). What is still not verified: licence, maturity, maintenance status, and whether it has been used outside the paper — the repo itself has not been inspected. Do not present it as a vetted tool; a look at the repository is the correct step before any run.

Key Contributions (to this topic)

• A live-traffic cost meter, not a calculator — attaches to a running vLLM server and reports real $/M-tokens against the operator's own load. Evidence: moderate (stated in the preprint abstract; repo not inspected) (Patil)
• Operationalises C_eff = f(H, M, Q, lambda, L) — makes offered request rate λ, the variable no surveyed open-source calculator exposes, a first-class measured input. Evidence: moderate (Patil)
• Validated across 42 benchmarks spanning dense, ultra-sparse MoE and sparse MoE models, with a 56-run A100 replication sweep. Evidence: moderate (Patil)

Mentioned In

• The Cost-Measurement Problem — the defect this instrument exists to fix.

Related Entities

• OpenAI (pricing) — the price side of the same fraction; published prices are observable, costs are not.
• Anthropic (economics) — the margin claim that this class of instrument would be needed to check.

Changelog

• 2026-07-23 — Full-text touch-up: repo URL identified (github.com/pChitral/vllm-cost-meter); reads live vLLM Prometheus metrics; paper's cost basis is $6.98/GPU-hr (Azure H100 NVL on-demand). Repo internals still not inspected.
• 2026-07-22 — Created from the Patil preprint ingested this compile. First cost-side instrument logged in this topic; explicitly marked unverified (abstract only, repo not inspected).