Rung 02 Serving & RuntimeSwitch rung

Running the Box Honestly

Case Study·Advanced·prefill-decode-disaggregation, kv-cache-management, speculative-decoding, kv-cache

Price a token, choose own vs rent, fix stutter without breaking chat, and refuse to multiply headline speedups nobody measured together

You run a small inference service on dedicated H200s, serving a 70B open model with vLLM: the configuration DigitalOcean measured (Llama-3.3-70B in FP8, $3.44 per GPU-hour). Your cofounder wants to quote customers a cost per million tokens and bolt on every optimisation in the papers. Every number you can lean on here was measured or modelled by someone else, and nothing has been run on your own box yet. Make the calls the evidence actually supports.