DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

Abstract

Verbatim: "Speculative decoding accelerates Large Language Model (LLM) inference by decoupling draft generation from target verification. While recent parallel drafters efficiently propose long token sequences in a single forward pass, they suffer from rapid acceptance decay due to a lack of inter-token dependencies. Furthermore, indiscriminately verifying these extended blocks wastes critical batch capacity on tokens with high rejection risks, severely degrading throughput in high-concurrency serving systems. We introduce DSpark, a speculative decoding framework that unifies high-throughput parallel generation with adaptive, load-aware verification."

Key Contributions

• Semi-autoregressive architecture — couples a parallel (DFlash-based) backbone with a lightweight sequential module that restores intra-block token dependency, mitigating suffix (acceptance) decay while keeping drafting cheap. Two head variants: a Markov head (first-order dependency via low-rank factorization) and an RNN head (full within-block prefix history).
• Confidence-scheduled verification — a lightweight linear confidence head emits a per-position survival probability; a hardware-aware scheduler prunes low-confidence tokens so verification budget is spent where acceptance is likely.
• Post-hoc Sequential Temperature Scaling (STS) — calibrates the confidence estimates left-to-right, reducing Expected Calibration Error (ECE) so the cumulative-acceptance estimates are trustworthy.
• Hardware-aware prefix scheduler (Algorithm 1) — greedy global sort of candidate tokens by cumulative survival probability, incremental batch admission with early stopping to preserve causal correctness; routes verification budget by load profile and token confidence.
• Production deployment inside the DeepSeek-V4 serving system, with engineering adaptations for asynchronous scheduling and CUDA-graph replay.
• Open-source release of DSpark checkpoints and the DeepSpec training framework.

Methodology

Draft configuration: 3 MoE layers with mHC, sliding-window attention of 128, maximum block size γ = 5, Markov head (rank-256 factorization). Training uses a three-term loss: cross-entropy (weight 0.1) + distribution-matching total-variation loss (weight 0.9) + confidence binary cross-entropy (weight 1.0). Trained on 1.3M samples from Open-PerfectBlend (17.6% chat / 39.4% math / 38.9% code), 10 epochs, sampling temperature 1.0.

Verification is standard rejection sampling and remains exact / distribution-preserving: "The token x_k is accepted with probability min(1, p^t_k(x_k)/p^d_k(x_k))" and "the first rejection at position k discards all subsequent tokens." The confidence scheduler only decides how many tokens to verify under load — it does not alter the acceptance rule, so there is no quality degradation.

Results

Offline (Table 1)

Targets: Qwen3-4B / 8B / 14B, Gemma4-12B. Domains: math (GSM8K, MATH500, AIME25), code (MBPP, HumanEval, LiveCodeBench), chat (MT-Bench, Alpaca, Arena-Hard). Metric = accepted length per round.

• vs Eagle3 (autoregressive baseline): +30.9% (4B), +26.7% (8B), +30.0% (14B).
• vs DFlash (parallel baseline): +16.3% (4B), +18.4% (8B), +18.3% (14B).
• Representative Qwen3-4B accepted length: DSpark 6.11 (GSM8K) / 5.38 (HumanEval) / 3.54 (MT-Bench) vs DFlash 5.40 / 4.74 / 2.96 vs Eagle3 5.14 / 4.16 / 2.26.

Ablations: a 2-layer DSpark beats a 5-layer DFlash baseline; the sequential head adds only 0.2-1.3% latency while lifting accepted length up to ~30%; the gap widens at longer proposal lengths (γ ∈ {4,8,12,16}). Confidence-based pruning raises chat acceptance rate from 45.7% to 95.7% under a static-threshold sweep; raw confidence ROC-AUC 0.81-0.90 but overconfident (ECE 3-8%), cut to ~1% ECE after STS.

Production (Figure 7) — the headline

• Baseline: MTP-1 (the prior production single-token speculative-decoding system).
• Hardware / setting: DeepSeek-V4-Flash and DeepSeek-V4-Pro inference engines, live user traffic.
• Measurement basis: first-party. The paper states it "deployed DSpark within the DeepSeek-V4 serving system to assess its performance under live user traffic" and reports "raw telemetry data sampled directly from live user traffic."
• Per-user generation speedup, at matched throughput: V4-Flash 60%-85%, V4-Pro 57%-78%.
• Aggregate-throughput improvement at moderate SLA: V4-Flash +51% (at 80 tok/s/user), V4-Pro +52% (at 35 tok/s/user).
• Under strict interactivity constraints the system enters a different operational regime with much larger nominal improvements (V4-Flash 661% at 120 tok/s/user; V4-Pro 406% at 50 tok/s/user). These are regime-shift numbers, not the headline speedup and should not be quoted as the typical gain.
• Load-adaptive behaviour (Figure 8): verification budget adapts from ~4-6 tokens under moderate concurrency down to lower values as load rises.

Verification note (against prior KB record)

• First-party / matched-throughput / self-measured: CONFIRMED. The 60-85% headline is a per-user speedup vs the prior production baseline MTP-1, at matched throughput, from live-traffic telemetry on DeepSeek's own fleet. This is vendor self-measurement of the authors' own system — moderate evidence at best, to be attributed as such.
• The prior abstract-only record captured "60-85%" (the V4-Flash range) but not the V4-Pro 57-78% range or the MTP-1 baseline framing — both now grounded in full text.

Limitations

Author-stated: "For complex queries with inherently low acceptance rates, this upfront drafting compute is unrecoverable. Future optimizations could introduce difficulty-aware early exiting within the draft model." The asynchronous scheduler also needs engineering adaptation for non-smooth hardware capacity curves and CUDA-graph replay compatibility. Beyond these: it is a preprint (not peer-reviewed) and the production numbers are self-measured with no independent reproduction.

Source: arXiv:2607.05147 — DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation, Cheng, Yu, Shao et al. (Peking University; DeepSeek-AI), 6 July 2026. Full text retrieved via arXiv HTML5 on 2026-07-23.