WATCHActiveMed3Y

Transformer Successors: State-Space Models and Linear Attention Are Worth Watching

By MenFem Editorial·AI Infrastructure — Model Architecture·13 April 2026·Methodology·
ai-infrastructurememory-over-computememoryarchitectureSSMMambatransformers
Share
Transformer Successors: State-Space Models and Linear Attention Are Worth Watching

Key Points

  • Transformer attention scales O(n²) in memory — the fundamental cause of KV cache and HBM bottlenecks
  • State-space models (Mamba, RWKV) offer O(n) linear memory scaling — potentially 10x more efficient
  • No SSM has matched Transformer quality at 100B+ parameters yet — the scaling question is unanswered
  • Hybrid architectures (Mamba + attention layers) are the most likely practical path
  • If SSMs win at scale, the HBM and KV cache compression theses partially unwind

The Transformer architecture's self-attention mechanism scales O(n²) in memory with sequence length. This quadratic scaling IS the KV cache problem — it is why long-context inference is expensive, why reasoning models generate huge memory footprints, and why HBM is the bottleneck. State-space models (Mamba, RWKV) and linear attention variants offer O(n) memory scaling — linear instead of quadratic. If these architectures close the quality gap with Transformers at scale, they would obsolete the entire KV cache optimization stack AND reduce HBM requirements per inference pass. Hybrid architectures (some attention layers + some state-space layers) are showing promising results, getting Transformer-quality reasoning where it matters and linear scaling everywhere else. This is the most likely path to adoption — not a wholesale replacement but a gradual blend. This is the wildcard in the AI infrastructure thesis. If SSMs win at scale, the HBM and KV cache compression calls partially unwind. If they don't, the memory wall thesis gets stronger. Either way, this is worth tracking closely.

Research Log

Sources rebuilt primary-first (catalogue II item 69): 3 primary, 0 secondary kept.

source: docs/plans/markets-work-catalogue-2026-09-07.md

Re-underwritten 2026-09-06. The watching call was right in direction: hybrid linear-attention/state-space blocks are now the default in the largest open models rather than a research curiosity. KEEP, conviction to MEDIUM; an early VALIDATED is defensible at the next review if the blend holds in a frontier closed model.

source: docs/plans/call-reviews-2026-09-06-thematic.md

Sources filled from the shelf and the 2026-09-06 review docs (catalogue item 3). 3 entries.

source: docs/plans/markets-work-catalogue-2026-09-06.md

Bull Case

A hybrid SSM-Transformer architecture achieves frontier quality at a fraction of the memory cost. Memory becomes abundant, inference costs collapse faster than hardware alone could deliver. The bottleneck shifts entirely to compute quality and data curation.

Bear Case

SSMs plateau at medium scale. Transformers continue to dominate through 2030. KV cache compression and HBM scaling remain the only viable paths to cheaper inference. The memory wall thesis strengthens.

What would prove this wrong

This is a watching call, so it breaks in either direction. The reason for holding it stops being true if the gap never closes — no hybrid or state-space model reaches frontier quality and the architecture stays a curiosity, in which case the attention spent watching it was the cost. It stops being true just as surely if a pure state-space model displaces the Transformer outright, because this call bets on the blend rather than the replacement, and the KV-cache and HBM theses it hedges would unwind faster than it anticipates.

Catalysts

100B+ parameter SSM training runOther

The first state-space model trained at frontier scale. If quality matches Transformers, it is a paradigm shift for all infrastructure investment theses.

Hybrid architecture adopted by frontier labProduct Launch

Anthropic, Google, or OpenAI integrating SSM layers into production models would validate the approach at the highest level.

Risk factors

SSMs may never match Transformer quality at scaleHigh

The O(n²) attention mechanism may be doing something fundamentally necessary for complex reasoning. SSMs might only work well for certain task types.

Adoption drag from Transformer ecosystemMedium

Transformer tooling, training infrastructure, and workforce expertise create massive switching costs even if SSMs prove competitive.

Conviction

Conviction History

Low→Med

6 Sept 2026

Review 2026-09-06: both falsifier legs NOT FIRED; the attention/SSM blend became the default in frontier open models (Qwen3.5 397B-A17B at 3:1 Gated DeltaNet:attention). Conviction LOW → MEDIUM.

Key Metrics

Transformer Attention Scaling
O(n²)
SSM Memory Scaling
O(n)
Largest SSM Trained
~8Bparams

Sources

See all AI Infrastructure — Model Architecture calls →