Skip to content
Sign in
LONGActiveHigh1Y

Parameter Scaling Is Dead. Multi-Dimensional Scaling Is the New Paradigm.

By MenFem Editorial·AI Infrastructure — Scaling Strategy·13 April 2026·Methodology·
ai-infrastructurememory-over-computescalingMoEinference-time-compute
Share
Parameter Scaling Is Dead. Multi-Dimensional Scaling Is the New Paradigm.

The argument, taken apart

Turn it, hover a ring, a moon or a weight
One call, taken apart
Rings are the falsifier’s legs. Moons are the catalysts it named. Weights are its risks, longer for higher severity.
The claim

AI scaling has moved from raw parameter count to four memory-hungry dimensions — inference-time compute, MoE, data curation, architecture — making memory bandwidth the universal bottleneck.

What would make it wrong · 2 legs
  1. Leg 1 · Not fired
    A step change comes from a bigger dense model
    “Every frontier release in the window is sparse and reasoning-first.”
    read 6 Sept 2026
  2. Leg 2 · Not fired
    Inference-time compute stops paying for its memory
    “GPT-6 Astra sold on task time, not parameters.”
    read 6 Sept 2026
The window
13 Apr 2026 to 13 Apr 2027
54% gone · 169 days left
The reviews
Reviewed on 3 days · last 7 Sept 2026
29 Apr · 6 Sept · 7 Sept
Column = the claim, taller for more conviction · rings = falsifier legs, lit when firing · moons = catalysts · weights = risks, longer for higher severity
  • Leg 1, not fired: A step change comes from a bigger dense model, read 6 Sept 2026: “Every frontier release in the window is sparse and reasoning-first.”
  • Leg 2, not fired: Inference-time compute stops paying for its memory, read 6 Sept 2026: “GPT-6 Astra sold on task time, not parameters.”
  • o3-class reasoning becomes standard
  • 1T+ parameter MoE in production
  • medium severity: Diminishing returns on inference-time compute
  • low severity: New scaling law revives parameter scaling

Key Points

  • Parameter scaling hit diminishing returns — GPT-4 to GPT-5 gap smaller than GPT-3 to GPT-4
  • Inference-time compute (o3) generates 10-100x more tokens per query — massive KV cache pressure
  • MoE architectures use parameters sparsely (~12.5% active) but all must be in memory — memory efficiency matters more than total FLOPS
  • Data quality scaling shifts value from compute volume to curation — smaller, better models win
  • Multi-dimensional scaling makes memory the universal bottleneck across ALL scaling axes

The AI industry has shifted from pure parameter scaling — making models bigger with more compute — to multi-dimensional scaling across at least four axes: inference-time compute, mixture-of-experts architecture, data quality curation, and memory-efficient design. This shift makes memory optimization MORE important, not less. Each scaling dimension creates its own memory pressure: - Inference-time compute (o3-style reasoning): generates 10-100x more tokens per query, creating massive KV caches - Mixture of Experts (Mixtral, Grok): uses ~12.5% of parameters per forward pass, but ALL parameters must be in memory. MoE is fundamentally a memory architecture. - Data quality: training on curated, higher-quality data produces better models at smaller parameter counts — shifting value from compute to curation and making efficient inference even more important - Architecture innovation: hybrid models, speculative decoding, and other techniques all trade memory management complexity for quality and speed The net effect: memory bandwidth is now the universal bottleneck across ALL scaling dimensions. Companies optimizing for multi-dimensional scaling will outperform those still chasing parameter counts.

Research Log

Re-underwritten 2026-09-06. NOT FIRED: every frontier release in the window is sparse and reasoning-first (K2 Horizon 375B total / 23B active, 3 Sep; GPT-6 Astra sold on task time, not parameters). KEEP, conviction HIGH unchanged; an early resolution is worth considering at the next review.

Bull Case

Multi-dimensional scaling becomes the permanent paradigm. Memory optimization companies see demand from every scaling vector simultaneously. The memory-over-compute thesis is validated by industry capex reallocation from training clusters to inference infrastructure.

Bear Case

Parameter scaling makes a comeback via novel training methods or synthetic data breakthroughs. The shift to multi-dimensional scaling was a temporary pause while training data and compute scaled, not a permanent regime change.

What would prove this wrong

The reason for holding this stops being true if a frontier lab ships a step-change whose gain is attributed to a bigger dense model rather than to inference-time compute, sparsity or data curation — the call's own LOW-severity risk names that event, and if it happens the paradigm did not shift, it paused. It also stops being true if inference-time compute plateaus: if extended reasoning stops paying for the memory it costs, the strongest of the four axes goes away and memory stops being the universal constraint.

Catalysts

o3-class reasoning becomes standardMacro

When all frontier models offer extended thinking modes, inference memory demand permanently increases by an order of magnitude.

1T+ parameter MoE in productionProduct Launch

Trillion-parameter MoE models running with ~125B active parameters. Memory bandwidth becomes the absolute binding constraint.

Risk factors

New scaling law revives parameter scalingLow

A breakthrough in training efficiency could make bigger dense models viable again. Unlikely but not impossible.

Diminishing returns on inference-time computeMedium

o3-style reasoning may plateau for many practical tasks. If extended thinking doesn't deliver proportional quality gains, users won't pay the memory cost.

Conviction

Conviction History

Med→High

29 Apr 2026

Both named catalysts have triggered: (1) o3-class reasoning is standard across Anthropic/OpenAI/Google; (2) 1T+ MoE in production (DeepSeek-R1 671B/37B-active, Mixtral 46.7B). Multi-dim scaling is no longer a forecast.

Key Metrics

o3 Token Generation vs Base
10-100x
MoE Active Parameters
~12.5%
Scaling Dimensions
4 (inference, MoE, data, architecture)

Sources

See all AI Infrastructure — Scaling Strategy calls