Conditional Memory (Scalable Lookup)

Conditional memory is an architectural proposal that gives large language models a second axis of sparsity, orthogonal to Mixture-of-Experts (MoE). The distinction is clean:

• MoE = conditional computation — selectively activate a subset of neurons/experts per token.
• Conditional memory = conditional retrieval — selectively look up stored knowledge from a static table per token.

The mechanism is Engram (Cheng et al., DeepSeek/PKU, Jan 2026): a modernized n-gram embedding that provides constant-time (O(1)) lookup. Instead of forcing the model to simulate retrieval through expensive forward-pass computation — something Transformers do only implicitly and inefficiently — Engram hands the model a dedicated, scalable memory primitive.

Why It Matters: A New Scaling Axis

The load-bearing result is the Sparsity Allocation problem: given a fixed capacity budget, how much should go to neural computation (MoE) versus static memory (Engram)? The answer is a U-shaped scaling curve with an optimal interior allocation — neither all-compute nor all-memory is best. This reframes "knowledge stored in weights" as something that can be deliberately budgeted as a distinct resource, and the paper scales Engram to 27B parameters of conditional memory on this basis.

Key Claims

• Conditional memory is a sparsity axis distinct from and complementary to MoE — compute-sparsity (MoE) and memory-sparsity (Engram) can be combined. Evidence: strong (paper with ablations) (Engram)
• A U-shaped scaling law governs the compute-vs-memory allocation — optimal capacity split is interior, not at either extreme; scaled to 27B params. Evidence: strong (Engram)
• Lookup helps reasoning more than raw recall — largest gains are on reasoning/code/math, not just knowledge benchmarks. Evidence: strong (Engram)

Results

Open Questions

• Does the U-shaped allocation hold at frontier (trillion-parameter) scale, and does it interact with hybrid attention?
• How does conditional memory compose with agent memory architectures (external retrieval stores) — substitute, or stack?
• Is the static-lookup table a new attack surface (knowledge poisoning) the way agent memory stores are?

Related Concepts

• Agent Memory Architectures — agent-level memory (external stores, write-manage-read); conditional memory is model-weight-level memory by contrast
• Chain-of-Thought Reasoning — Engram's reasoning gains complement test-time deliberation approaches

Changelog

• 2026-06-24 — Initial compilation from Engram paper (arXiv 2601.07372). New page: conditional memory as a sparsity axis orthogonal to MoE.