Rung 04 / Memory & Compute Architecture

Hardware

The silicon rung of the inference stack, read memory-first.

3.35 TB/s

Memory bandwidth of one H100 SXM's 80 GB of HBM3 — the wall the arithmetic waits behind

23
Sources
7
Concepts
10
Entities
A memory module with copper heat spreaders, on paper.

In scope: the memory hierarchy and the memory wall, HBM and DRAM scaling, bandwidth and capacity limits, processing-in-memory, accelerator and NPU architecture, advanced packaging, and interconnect including optical interconnect and co-packaged optics. Memory is the binding constraint of this cycle. Out of scope: photonic compute (computing in light, as distinct from communicating in light) and quantum computing — frontier science with no near-term effect on what a token costs — and consumer buying advice, which is an editorial beat rather than research.

An HBM die stack cut away at the corner, copper through-silicon vias exposed in section.
Memory wallHBMDRAMNPUsPackaging
Sources compiled for this topic
TypeSourcePublished
PAPERMemory Scarcity, Open Models, and the Restructuring of the AI Industry, 2026-2030
Satoshi Matsuoka · RIKEN Center for Computational Science (R-CCS)

Quantitative scenario model showing the 2026 DRAM/HBM price surge creates a persistent entrant-incumbent inference-cost gap that never closes over 2026-2030 (3.2x in 2026, narrowing to ~1.9x in 2027, re-widening to 3-4x by 2029-30) via a depreciation-conveyor mechanism; five probability-weighted scenarios (Rotating Landlord Oligopoly 25%, Commoditization Crash 25%, Jevons Absorption 20%, System-Layer Re-differentiation 18%, Geopolitical Bifurcation 12%) map the resolution.

2026-07-08
REPORTInside the NVIDIA Vera Rubin Platform
Kyle Aubrey · NVIDIA

Six-chip co-designed AI supercomputer platform: 50 PFLOPS FP4 inference, 288GB HBM4 at 22TB/s, 5x improvement over Blackwell

2026-01-05
REPORTNVIDIA, ASML, TSMC and Synopsys Set Foundation for Next-Generation Chip Manufacturing
NVIDIA Newsroom · NVIDIA / ASML / TSMC / Synopsys

NVIDIA cuLitho computational-lithography library deepens integration with ASML, TSMC, and Synopsys. GPU-accelerated mask synthesis becomes a structural enabler for sub-2nm production and a softener for TSMC's deferral of high-NA EUV

2026-04-23
REPORTNVIDIA Vera Rubin Ramps Into Full Production to Power Agentic AI Factories Worldwide
NVIDIA Newsroom · NVIDIA

NVIDIA confirms the Vera Rubin platform is ramping into full production (May 2026), production shipments begin fall 2026 across 350+ factories in 30 countries; 10x agent throughput at scale vs Grace Blackwell.

2026-05-31
REPORTTSMC — 6-K: June 2026 Revenue Report (filed 2026-07-13)
Taiwan Semiconductor Manufacturing Company Limited · TSMC / SEC EDGAR

June 2026 consolidated revenue NT$442.68B (+6.2% MoM, +67.9% YoY); H1 2026 revenue NT$2,404.48B (+35.6% YoY).

2026-07-13
REPORTTSMC — Q2 2026 Earnings Release + Quarterly Management Report
Taiwan Semiconductor Manufacturing Company Limited · TSMC / investor.tsmc.com

Full Q2'26 print (primary, TSMC IR): GM 67.7% above the 65.5-67.5% guided ceiling, OpM 60.3%, revenue US$40.20B/NT$1,270.38B at top of guidance, HPC 66% (60->61->66 progression), Q2 capex US$15.70B/H1 US$26.80B, 2nm 3% of wafer revenue with N2-ramp dilution active.

2026-07-16
REPORTASML — Q2 2026 Financial Results
ASML Holding N.V. · ASML / asml.com

Q2'26 beat (EUR9.33B net sales / 54.0% GM, both above guidance) + FY26 guidance raise to EUR43-45B/54-56% GM (+20.5% midpoint since the Feb 20-F). Beat engine was Installed-Base/service upside (+EUR300M), not new systems.

2026-07-15
REPORTIntel Q2 2026 Results (Actual Print) — 8-K Ex-99.1
Intel Corporation · Intel Corporation (SEC 8-K Ex-99.1)

ACTUAL Q2 2026: revenue $16,128M (+25% YoY, seventh straight guide-beat); GAAP EPS $(2.16) is entirely a $12,529M non-cash CHIPS Act Escrowed-Shares mark, non-GAAP EPS $0.42 and GAAP operating income POSITIVE $1,796M; Foundry loss narrowed to $(2,089)M/(36.2)% (first simultaneous margin+dollar improvement); gross capex FELL 41% YoY to $2,652M (the deeply negative Adjusted FCF is an Apollo/Fab 34 JV buyout, not capex). Panther Lake in HVM on ASML High-NA EUV. No FY26 capex-dollar guide; 18A yield still undisclosed.

2026-07-23
REPORTTSMC — 6-K: August 2026 Revenue Report (filed 2026-09-10)
Wendell Huang, SVP & CFO · TSMC / SEC EDGAR

August 2026 net revenue NT$514.81B (+10.1% MoM, +53.3% YoY); Jan–Aug 2026 NT$3,386.87B (+39.3% YoY). Monthly foundry throughput proxy; YoY rate decelerating (67.9% June → 53.3% August) while absolute level climbs.

2026-09-10
REPORTSK hynix Announces 2Q26 Financial Results
SK hynix Inc. · SK hynix

2Q26 operating margin 76% on KRW 79.32tn revenue (+257% YoY), operating profit KRW 60.54tn (+557%); HBM4 mass shipments began in Q2; LTAs with ~10 customers. Memory scarcity stated as a margin, not a capability spec.

2026-07-29
ANALYSISCustom Silicon Inflection 2026: Hyperscaler ASICs vs NVIDIA GPU
Wega Chu, Dylan Patel, Daniel Nishball et al. · SemiAnalysis

Deep technical analysis arguing NVIDIA's rack-as-product co-design strategy deepens lock-in despite custom ASIC growth at 44.6% CAGR

2026-02-25
ANALYSISThe Angstrom Era Arrives: TSMC Enters 2nm Mass Production and Unveils 1.6nm Roadmap
TokenRing Research · TSMC / Industry Analysis

TSMC N2 hit mass production Q4 2025 (65-75% yield, Apple got >50% of capacity); A16 (1.6nm) with backside-power Super Power Rail targets late 2026 volume — first departure from FinFET in a decade

2026-01-21
ANALYSISHBM Architectural Shakeup: HBM4, HBM4E, C-HBM4E — 3nm Base Dies Enable 2.5x Performance
Anton Shilov · Tom's Hardware / TSMC / GUC

HBM base dies move from DRAM to 3nm logic (TSMC N3P) — enables 2.5x bandwidth (3 TB/s per stack), 2x channels, C-HBM4E adds custom base dies with near-memory compute

2025-12-02
ANALYSISMemory-Centric Computing: A Paradigm Shift for Sustainable and Efficient Systems
Onur Mutlu (ETH Zurich) — synthesis of public talks and papers · ETH Zurich / SAFARI Group

Argues processor-centric computing is fundamentally broken: 60–90% of system energy is data movement, not compute; DRAM can compute (RowClone, Ambit); reliability threats (RowHammer, RowPress, column disturbance) force memory intelligence anyway; JEDEC + trillion-dollar incumbency is the bottleneck, not technology

2026-04-21
ANALYSISTSMC to Quadruple Advanced Packaging Capacity: Reaching 130,000 CoWoS Wafers Monthly by Late 2026
TokenRing / FinancialContent · FinancialContent

TSMC CoWoS ramp: ~35K wafers/mo (late 2024) → 75K (end 2025) → 130K target (end 2026); NVIDIA secures ~60% of capacity.

2026-02-05
ANALYSISSK hynix 2026 Outlook: HBM3E Dominates, HBM4 Dual Strategy Amid 3 Market Headwinds
TrendForce · TrendForce

HBM3E 12-Hi 36GB ≈ 2/3 of 2026 HBM shipments; HBM4 16-Hi 48GB targets Q4 2026; SK Hynix ~60% share, 2026 capacity pre-booked by NVIDIA + OpenAI.

2026-01-05
ANALYSISGlobal AI Server Shipments Forecast to Grow Over 28% YoY in 2026, with a Rising Share of ASIC-Based Systems
TrendForce · TrendForce

TrendForce: 2026 AI server shipments +28% YoY; ASIC-based systems reach 27.8% of shipments (highest since 2023), GPUs 69.7%; top-5 NA CSP capex +40% YoY; Google TPUs sold externally (Anthropic).

2026-01-20
ANALYSISThe Custom AI ASIC State of Play (May 2026) — Broadcom Deals, Google TPUs, Meta MTIA & Beyond
Tom's Hardware · Tom's Hardware

May-2026 ASIC state-of-play: Broadcom ~60% / Marvell ~35% of the custom-AI-ASIC co-design market; Marvell ~$11B AI ASIC revenue 2026; programs — TPU v7 Ironwood (4,614 FP8 TFLOPS, 192GB HBM3E, 7.37 TB/s), Maia 200, Trainium3 (Q2 2026), MTIA, OpenAI Titan.

2026-05-26
ANALYSISClose read: Vintage Breakeven (§6) and the Demand-Measurement Critique (§9.1/§9.3) — Matsuoka 2607.07207
MenFem Research Desk · MenFem

Primary-text close read of the full 22-page PDF. Corrects the parent ingest on four points: author is RIKEN R-CCS not independent; vintage breakeven is a 16-cell grid (4 vintages x 2 regimes x 2 HBM branches) not 3 rows; the 2026 vintage's regime exposure was INVERTED (worse under sticky ~31.3%, not coupled ~24.5%); custom-silicon build cost is $0.082/PB not $0.072. AUTHORITATIVE wherever it differs from the parent.

2026-07-20
ANALYSISTSMC CoWoS Supply-Demand Gap Reportedly Narrowing from 20% to 10% by End-2026 as Capacity Expands
TrendForce (citing Economic Daily News, Reuters, Commercial Times) · TrendForce

June-2026 CoWoS refresh (projection, as-of 2026-06-15): TSMC monthly capacity 120K-140K wafers by end-2026 (industry ~200K incl. 50-60K OSAT); supply-demand gap narrowing ~20% to ~10% by end-2026, shortage moderating in 2027; >60% TSMC expansion by 2027, CAGR >80% 2022-27; CoPoS pilots mid-2027, MP 2028-29 with NVIDIA Feynman first customer.

2026-06-15
ANALYSISTSMC Debuts A13 Technology at 2026 North America Technology Symposium
TSMC / Industry coverage · TSMC

TSMC unveils A13 (1.3nm direct shrink of A14) and N2U process nodes at NA Tech Symposium — A13 production targeted for 2029 for AI/HPC, N2U a more affordable variant for phones/laptops/AI; Arizona advanced packaging facility announced

2026-04-23
ANALYSISTSMC Delays Use of ASML's High-NA EUV Machines Over Cost Concerns
Bloomberg · Bloomberg / TSMC / ASML

TSMC publicly defers high-NA EUV adoption until at least 2029, citing ~$400M per-machine cost (vs ~$200M for current EUV). Plans to squeeze more performance from existing EUV through process optimization (A13, N2U) — major signal for ASML revenue

2026-04-22
ANALYSISNVIDIA Snaps Up AI Chip Packaging Capacity as TSMC Expands in U.S.
CNBC · CNBC / NVIDIA / TSMC

NVIDIA has reserved the majority of TSMC's most advanced packaging capacity (CoWoS), making advanced packaging — not fabrication — the next bottleneck for AI accelerator supply

2026-04-08
Hardware — Memory & Compute Architecture | Knowledge Base | MenFem