An Interpretable Latency Model for Speculative Decoding in LLM Serving

Why this one first for serving: every other speculative-decoding source is a speedup number. This is the bound — the model that says when the speedup exists and when it disappears. Gate 4 asks for the constraint, not another win.

Abstract (verbatim)

"Speculative decoding (SD) accelerates large language model (LLM) inference by using a smaller draft model to propose multiple tokens that are verified by a larger target model in parallel. While prior work demonstrates substantial speedups in isolated or fixed-batch settings, the behavior of SD in production serving systems remains poorly understood: request load varies over time, and effective batch size emerges from the serving system rather than being directly controlled or observed. In this work, we develop a simple and interpretable latency model for SD in LLM serving. We infer effective batch size from request rate using Little's Law and decompose per-request demand into load-independent and load-dependent components for prefill, drafting, and verification. We validate our model using extensive measurements from vLLM across verifier and drafter model sizes, prefill and decode lengths, request rates, draft lengths, and acceptance probabilities. The model accurately describes observed latency, explains why speedups often diminish as server load increases, and characterizes how draft length, acceptance rate, and verifier-drafter size shape latency across serving conditions, with implications for configuring SD in deployed systems. We further show how the framework extends to mixture of experts models, where sparse expert activation changes the effective service costs across load regimes. Together, our results provide a structured framework for understanding SD in real LLM serving systems."

The problem it names

Almost every published speculative-decoding result is measured at batch size 1 or a fixed batch. In a real server nobody sets the batch size — it emerges from arrival rate and service time, and it is not directly observable. So a configuration tuned on a benchmark (long draft lengths, aggressive speculation) can be the wrong configuration the moment the server has queue depth. That gap between benchmark and production is the paper's whole subject.

Method

1. Infer effective batch size from request rate via Little's Law (L = λW), rather than treating batch size as a knob.

2. Decompose per-request demand into load-independent (c₁) and load-dependent (c₂) components for each of the three phases — prefill, drafting, verification.

3. Assemble a closed-form per-request latency:

   L = [ c₁,p + (g/E)(c₁,v + k·c₁,d) ] / [ 1 − RPS·( c₂,p + (g/E)(c₂,v + k·c₂,d) ) ]

   where k is draft length, g the generation length, and E the expected number of accepted tokens per speculation cycle. The denominator is the utilisation term — as RPS·(service demand) approaches 1, latency blows up, and speculation adds to that demand.

4. Validate against extensive vLLM measurements across verifier/drafter sizes, prefill and decode lengths, request rates, draft lengths and acceptance probabilities.

What was measured

• Models: Llama 3.1 (8B, 70B); the Qwen3 family (0.6B–235B); GPT-OSS-20B (MoE).
• Prefill/decode lengths: {256, 512, 768, 1024} tokens.
• Acceptance rates: 50%–100%.
• Draft lengths: 1–10 tokens.
• Request rates: synchronous (batch = 1) up to near-saturation.

Findings

• Speedup is a low-load phenomenon. At low load the speculative arm is favourable consistently (c₁ ratio < 1). As utilisation rises, the load-dependent cost of drafting and verification enters the denominator; once that ratio exceeds 1, speedup falls with utilisation. Speculation buys latency with throughput, and a busy server has no throughput to spare.
• Acceptance probability is the escape. The erosion holds "unless acceptance rate α ≥ 90%" — at very high acceptance, speculation stops being wasted work and the added demand is repaid.
• Draft length must be retuned for load. Fixed costs (c₁) scale roughly linearly with model size; load-dependent costs (c₂) also scale linearly but interact with draft length k. The consequence, stated plainly: draft lengths optimised at batch = 1 are suboptimal under load. A production system that copied its k from a paper is mis-tuned by construction.
• MoE models beat the dense prediction at low load. Sparse expert activation means a speculative batch does not touch every expert. The framework corrects with an expert-coverage factor φ(T) = 1 − (1 − m/M)^T for T tokens over M experts with m active. Adding it lifts fit from R² 0.83 → 0.98 for Qwen3-30B-A3B — so the MoE correction is not a rounding term.

Limitations (authors' own)

• Model targets the stable pre-saturation regime; behaviour at and past the saturation boundary is out of scope.
• Mean latency only — tail/percentile distributions are left to future work, which is a real gap because SLOs are written on p99, not the mean.
• Coefficients are system-dependent (GPU, scheduler, serving engine) and must be re-fit per deployment; the structure transfers, the numbers do not.
• Fitted against vLLM implementation details specifically.

How it bears on MenFem

• "KV Cache Compression Will Cut Inference Costs 50%" and the cost-decline call both assume serving-side wins compose. This paper is the direct counter-case: speculative decoding's win is consumed by the same utilisation it competes for. Wins on the latency axis and wins on the throughput axis trade against each other, and stacking their headline percentages overstates the total.
• It is the bound behind the 2026-09-06 batch-speculative-decoding candidate rather than another speedup number — take the constraint before the result.
• Teaching moment: the speedup you benchmark on an idle server is the speedup you will not get on a busy one. A fifth-grader version exists: guessing ahead only helps while the kitchen is quiet; when it is packed, the guesses are extra orders.