vLLM

Type: Product (open-source LLM serving framework)

vLLM is the framework underneath this rung's only measured dollar figure, and therefore underneath the number the rung owns. The DigitalOcean benchmark runs vLLM 0.24.0 serving Llama-3.3-70B FP8 on a single H200, sweeping batch and concurrency from 1 to 256.

Its relevance here is structural rather than as a product review: vLLM's continuous batching and paged KV management are the mechanism that produces the 44x cost spread. The framework is what converts spare concurrency into amortised cost, so the spread is a property of how it is operated, not a property of the chip.

Key contributions to this rung

• The configuration behind $20.32 → $0.45 per million output tokens across batch 1→256 on one H200 at $3.44/GPU-hr (DigitalOcean, as-of 2026-07-08)
• Independently reproduces the utilisation thesis with anchored dollar levels rather than ratios (DigitalOcean)

Scope of the number

One SKU, one model, one framework version, 1024-token in/out, max_model_len 4096. The close-read is explicit that this is a reproducible reference point, not a universal cost. Anyone quoting the 44x should carry that sentence with it.

Mentioned in

• KV Cache Management — paged attention as the reference implementation
• Frontier — supplies the number this rung owns