Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation

Read discipline — debt cleared. The prior version of this file was built from the arXiv abstract page only and flagged the missing $/GPU-hour input as the load-bearing unread detail. That detail is now FOUND from the primary text: $6.98/GPU-hour (Azure on-demand list price for an H100 NVL node at $13.96/hr for two GPUs; §4.1 + Table 1). The $0.21–$15.25/M-output-token range is therefore no longer "an unstated input" — it is anchored to Azure H100 NVL on-demand list pricing at $6.98/GPU-hr.

Artifact defect worth recording. arXiv's /pdf/2606.11690 endpoint serves a truncated 7-page render of a paper the abstract Comments field states is 26 pages, 9 figures. The 7 pages end mid-sentence in §4.3; §5 (Results) and §6 (Discussion) are absent from that PDF. This is the classic silent-underread trap — a valid-looking PDF that is a fraction of the real paper. The full text was recovered from the arXiv HTML5 render (/html/2606.11690v1). Figures sourced only from the render (the §5 per-cell cost table, the A100 detail, the crossover λ) are labelled as render-derived below; the $6.98/GPU-hr anchor and the $15.25/$0.87 endpoints were confirmed in BOTH the directly-read PDF pages and the render.

Abstract

"Every public LLM cost calculator we surveyed treats GPU utilization as a fixed input -- entered by the user, baked in as a preset, or silently assumed at 100% -- never measured against the operator's actual load. We show that this assumption is the dominant source of error: on identical H100 hardware, effective cost spans $0.21 to $15.25 per million output tokens, an underutilization penalty of 2.5-24x across low-to-moderate enterprise offered loads (1-10 rps) and up to 36.3x near idle (lambda=1 rps) -- driven by one operator-controlled variable, offered request rate lambda, which sets in-flight concurrency via Little's Law and which no open-source calculator exposes. Because calculators take utilization as a user-supplied input, any utilization-naive estimate understates true cost by exactly 1/U, systematically mispricing self-hosting -- most severely over-selling it for low-traffic workloads. We propose a measurement methodology that parameterizes the relationship as C_eff = f(H, M, Q, lambda, L), validate it with 42 benchmarks across dense, ultra-sparse MoE, and sparse MoE models, and release vllm-cost-meter, an open-source cost meter that attaches to a live vLLM server and reports real $/M-tokens against the operator's own traffic. We further show that FP8 quantization benefits the MoE architectures we tested roughly 2.2-2.4x more than the dense model (+69 to +74% vs. +31% peak throughput; n=3, broader validation needed), and our data are consistent with active parameter count, not total model size, being a primary predictor of saturation economics. To rule out single-hardware confounding we repeat the core sweep on A100 80GB PCIe (56 runs): the load-driven spread reproduces at 7.0-11.4x, the active-parameters ordering survives at FP8, and the dense-FP8 advantage inverts on silicon without native FP8 tensor cores -- a hardware-conditional caveat the framework already accommodates."

THE LOAD-BEARING DETAIL — the $/GPU-hour input (FOUND)

The dollar figures are throughput measured on vLLM, converted to $/M-tokens by Eq. 3:

C_eff = (P_GPU × 10^6) / (3600 × Θ_achieved(λ, L))        [$ per million output tokens]

and the utilization-naive counterpart the paper attacks, Eq. 4: C_naive = (P_GPU × 10^6) / (3600 × Θ_max), whose ratio to Eq. 3 is exactly Θ_max / Θ_achieved = 1/U.

P_GPU — the one number the abstract never disclosed — is stated in §4.1 and Table 1 (p.6):

• Node: 2× NVIDIA H100 NVL 96GB, 96 GB HBM3 per GPU (192 GB total), NVLink 900 GB/s, 80 CPU cores, 640 GB RAM.
• Cloud SKU: Azure Standard_NC80adis_H100_v5.
• Price: $13.96/hr for the node → $6.98/hr per GPU.
• Single-GPU configs (C1–C4) are costed at $6.98/GPU-hr, "which is $13.96/hr node amortized across the two H100s under the assumption that the second GPU is independently utilized by a co-tenant; a dedicated-node operator should double C1–C4 numbers." (So a dedicated single-tenant operator effectively pays $13.96/GPU-hr for those configs.)
• Pricing mode (§3.2): P_GPU is the Azure on-demand list price. "reserved and spot instances multiply C_eff by a roughly constant scalar (0.3–0.7×) and do not alter offered-load–cost dynamics."
• A100 comparison hardware: $3.67/hr/GPU on Azure's NC_A100_v4 family (render-derived).

Consequence for citation. The $0.21–$15.25/M range is real, measured throughput at $6.98/GPU-hr Azure H100 NVL on-demand list. Anyone quoting the level must carry that basis: on reserved/spot it falls ~0.3–0.7×; on a dedicated (non-co-tenant) node the single-GPU figures roughly double; on cheaper silicon (A100 at $3.67/GPU-hr) the whole curve shifts. The ratios (2.5–24×, 36.3×, 7.0–11.4×) remain the assumption-light finding; the level is now anchored rather than floating.

Key Contributions

• The denominator is unmeasured everywhere. Every public LLM cost calculator surveyed (Helicone, LiteLLM, YourGPT, llm-prices.com, and the LLM Inference TCO Calculator — >90% accept only model name + token counts) treats GPU utilization as a fixed input — user-entered, preset, or silently 100% — never measured against actual load.
• The magnitude of the error is quantified, with a named price basis. On identical H100 hardware at $6.98/GPU-hr, effective cost spans $0.21 to $15.25 per million output tokens (June 2026 window) purely from offered request rate.
• A clean analytic statement of the bias: a utilization-naive estimate understates true cost by exactly 1/U, which systematically over-sells self-hosting, most severely for low-traffic workloads.
• A parameterisation: C_eff = f(H, M, Q, λ, L) — hardware, model, quantization, offered request rate, latency SLO. Offered rate λ sets in-flight concurrency via Little's Law (N ≈ λ · W); the paper is explicit that λ is the offered rate, distinct from the engine's in-flight batch size.
• An instrument, not just a critique: vllm-cost-meter (github.com/pChitral/vllm-cost-meter), open-source, attaches to a live vLLM server and reports real $/M-tokens against the operator's own traffic via its Prometheus metrics.
• An architecture finding: FP8 quantization benefits the tested MoE architectures ~2.2–2.4× more than dense — flagged n=3, broader validation needed.
• A cross-hardware replication: the sweep repeats on A100 80GB PCIe (56 runs); the load-driven spread reproduces at 7.0–11.4×, and the dense-FP8 advantage inverts on silicon without native FP8 tensor cores.

Two penalty framings (both are in the text — do not conflate)

• 2.5–24× — the underutilization penalty across the 1–10 rps low-to-moderate enterprise-load window (the abstract's headline).
• 17.5–36.3× — the full per-configuration span from λ=1 (idle edge) to saturation: 17.5× for Mixtral 8x7B FP16, 36.3× for Qwen3-30B-A3B FP8 (footnote 1, p.1; intro, p.2). 36.3× is the idle-edge extreme.

Anchor examples (primary text, p.2 intro)

• Mixtral 8x7B FP16 @ λ=1 rps on an H100 = $15.25/M output tokens — "more expensive than Claude Sonnet 4.6."
• Same model @ 25 rps = $0.87/M — the peak-throughput cost. "Same GPU. Same model. A 17.5× cost ratio between λ=1 and saturation."
• $0.21/M = Qwen3-30B-A3B FP8 at saturation (§5.3, render-derived) — the absolute floor across all configs.

Results (MEASURED — benchmark runs, June 2026)

Table 3 — dense-model cost across offered load (Llama 3.1 8B FP16, H100, $6.98/GPU-hr; render-derived)

Self-host vs API crossover (§5.6 / §6, render-derived)

No single universal crossover λ — it is a surface in (λ, L, H) space, not a point. Model-dependent thresholds:

• Mixtral 8x7B FP16 crosses over vs Claude Sonnet 4.6 ($15.00/M output) near λ ≈ 1.5–2 rps (at λ=1 it is $15.25/M, i.e. more expensive than the API).
• Llama and Qwen single-GPU models cross below λ ≈ 1 rps under the $6.98/GPU-hr co-tenant denominator.
• Caveat the paper stresses: "published serverless per-token prices are unconditioned on any production service-level objective," so API-vs-self-host comparisons are not like-for-like on latency.

Setup (for reproducibility)

• Models (Table 2): Llama 3.1 8B Instruct (dense, 8B/8B active), Qwen3-30B-A3B-Instruct (ultra-sparse MoE, 30B/3B active), Mixtral 8x7B Instruct (sparse MoE, 46.7B/12.9B active). A deliberate sparsity gradient; chosen to span regimes, not to be frontier-quality.
• Precisions: FP16, FP8.
• Sweep: λ ∈ {1, 5, 10, 25, 50, 100, 200} requests/sec. 100 warmup (discarded) + 500 measured requests per config. Synthetic random tokens, input 512 / output 256. Prefix caching disabled (--no-enable-prefix-caching). Poisson arrivals (--burstiness 1.0).
• Engine: vLLM 0.19.0 (PyTorch 2.10.0+cu128) for headline numbers; SGLang 0.5.10 (PyTorch 2.9.1+cu128) run on the same apparatus but excluded from headline to keep the pricing denominator single-engine.
• Held constant (isolating λ): engine choice, pricing mode (Azure on-demand list), 512:256 I/O ratio, tensor-parallel degree (part of H), Poisson arrivals. A §5.7 Gamma-arrival probe (CV=2) found negligible cost impact at one config; treated as a conservative lower bound on real bursty-workload dispersion.

Limitations (author-acknowledged)

• n=3 for the FP8-benefit-by-architecture claim; author explicitly says broader validation is needed.
• The dense-FP8 advantage inverts on hardware without native FP8 tensor cores — hardware-conditional.
• "Active parameter count predicts saturation economics" is stated as consistent with the data, not established.
• Single-author preprint, not peer-reviewed. Author is an "Independent Researcher" with an unnamed employer; the work is stated to be independent of it.
• Latency SLO L is held implicit (all sweeps run without an SLO cutoff); incorporating L as an active admission-control dimension is named as future work (§6.9).

Source: Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation by Chitral Patil (Independent Researcher), arXiv:2606.11690 [cs.DC], submitted 2026-06-10. Full text read 2026-07-23 via arXiv HTML5 render (the /pdf endpoint serves a defective 7-page render of the 26-page paper). Code: github.com/pChitral/vllm-cost-meter.