Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Conditional memory as a sparsity axis orthogonal to MoE, via Engram (O(1) n-gram lookup). Sparsity Allocation problem yields a U-shaped scaling law (compute vs static memory); scaled to 27B params with gains on BBH +5.0, ARC-C +3.7, MMLU +3.4, NIAH 84.2→97.0.
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Abstract
The paper proposes conditional memory as a complementary sparsity axis to Mixture-of-Experts (MoE). Where MoE scales capacity through conditional computation (selectively activating neurons), conditional memory scales capacity through conditional retrieval (selectively looking up stored knowledge). It is instantiated as Engram, a module that modernizes the classic n-gram embedding into a constant-time (O(1)) lookup table — giving the model a dedicated memory mechanism instead of forcing it to simulate retrieval through expensive neural computation, a capability Transformers natively lack.
Key Claims
- Conditional memory is a distinct sparsity axis from MoE. MoE = conditional compute; Engram = conditional static-knowledge lookup. The two are complementary, not competing.
- A Sparsity Allocation problem governs the trade-off, and it is U-shaped. Allocating capacity between neural computation (MoE) and static memory (Engram) yields a U-shaped scaling curve with an optimal interior allocation point — neither pure-compute nor pure-memory is optimal.
- Engram scales — demonstrated up to 27B parameters of conditional memory.
- Gains are largest in reasoning, not just retrieval — the lookup mechanism helps general reasoning more than naive knowledge recall.
Results
Benchmark improvements from adding Engram:
- MMLU: +3.4
- CMMLU: +4.0
- BBH (Big Bench Hard): +5.0
- ARC-Challenge: +3.7
- HumanEval: +3.0
- MATH: +2.4
- Multi-Query NIAH (long-context retrieval): 84.2 → 97.0
Significance
This reframes "memory" inside LLMs as an explicit architectural primitive with its own scaling law, rather than something distilled implicitly into MoE weights. The U-shaped allocation result is the load-bearing finding: it argues frontier models should budget capacity across compute-sparsity (MoE) and memory-sparsity (Engram) jointly. Author overlap with the DeepSeek team situates it adjacent to the same lab driving open-weight frontier MoE design.
Source: arXiv:2601.07372 — Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models, Cheng et al., 12 January 2026.