WATCHActive
Transformer Successors: State-Space Models and Linear Attention Are Worth Watching
Transformers' quadratic memory scaling is the KV-cache problem. State-space and linear-attention models scale linearly — hybrids blending both look like the likely path.
Transformer attention scales O(n²) in memory — the fundamental cause of KV cache and HBM bottlenecks
State-space models (Mamba, RWKV) offer O(n) linear memory scaling — potentially 10x more efficient
ai-infrastructurememory-over-computememory