Expert Scattering in MoE Decoding

Nearly every frontier-scale open model on this rung is now a mixture of experts: a big pool of sub-networks ("experts"), of which each token uses only a few. Kimi K3 uses 104 billion of its 2.8 trillion parameters per token (Kimi K3 technical report); DeepSeek V4-Pro uses 49 billion of 1.6 trillion (DeepSeek V4). That is why these models are cheap to run for their size. This page is about a hidden cost that sparsity brings back in.

The problem. When a model produces text one token at a time, the GPU mostly waits for weights to arrive from memory. Speculative decoding is the standard fix: a small model guesses the next few tokens and the big model checks them all in one pass, so one load of the weights pays for several tokens. That only works if every guess uses the same weights. In a mixture of experts, different guesses can be routed to different experts, so checking four guesses can mean loading four sets of experts. The EcoSpec authors call this expert scattering: the extra guesses you check to go faster drag in extra memory traffic that slows you down (EcoSpec).

The fix, and what it bought. EcoSpec leaves the checking rule alone (so the output is unchanged) and only chooses which guesses to check, preferring ones that reuse experts already being loaded. The paper is explicit that this is a memory-bandwidth gain, not a compute one.

Key Claims

• Loading one expert moves 44.04 MB from memory on DeepSeek-V3.1 (FP8); 0.2 fewer experts per layer per step saves about 0.5 GB per step. Evidence: moderate — preprint, full text read (EcoSpec)
• On Qwen3-235B, choosing guesses by expert reuse cut estimated memory reads per checking step from 99.3 GB to 88.1 GB and checking time from 0.832 s to 0.730 s. Evidence: moderate — reads are estimated from expert counts, not measured with hardware counters (EcoSpec)
• Average end-to-end speed over plain decoding rose 1.22× → 1.36× (Qwen3-235B, vs EAGLE-3), 1.14× → 1.31× (GPT-OSS-120B), 1.10× → 1.15× (DeepSeek-V3.1, vs MTP); best single result 1.62×. Evidence: moderate — 8×H200, one configuration (EcoSpec)
• The benefit shrinks as batches grow: at four requests per batch, Qwen3-235B's speculative baseline fell to 1.00× and EcoSpec to 1.09×. Evidence: moderate (EcoSpec)

Why it matters for the token price

Sparsity lowers compute per token, but in decode the bill is memory traffic, and speculative decoding's saving on memory traffic is exactly what sparsity erodes. So the cheapness of a sparse model is partly given back at serving time unless the serving stack knows about experts. The size of what can be won back depends on the model: Qwen3-235B and GPT-OSS-120B had room to reuse experts; DeepSeek-V3.1, with evenly spread routing, had almost none. A buyer cannot read this off a price list, but it is part of why two models with similar active-parameter counts can cost different amounts to serve.

Open Questions

• Does the gain survive at the batch sizes production servers actually run (well above 4)?
• Is there a routing design that keeps quality while making consecutive tokens share experts — i.e. can the model be trained for this, rather than the server working around it?
• Measured, not estimated, memory traffic.

Related Concepts

• Open-Weight Frontier Parity — the sparse open models this cost applies to
• Speculative decoding (serving rung) — the serving technique itself
• HBM4 memory architecture (hardware rung) — the memory bandwidth being spent

Backlinks

• Models index