Rung 04 / Memory & Compute Architecture
Hardware
The silicon rung of the inference stack, read memory-first.
3.35 TB/s
Memory bandwidth of one H100 SXM's 80 GB of HBM3 — the wall the arithmetic waits behind
23
Sources
7
Concepts
10
Entities
In scope: the memory hierarchy and the memory wall, HBM and DRAM scaling, bandwidth and capacity limits, processing-in-memory, accelerator and NPU architecture, advanced packaging, and interconnect including optical interconnect and co-packaged optics. Memory is the binding constraint of this cycle. Out of scope: photonic compute (computing in light, as distinct from communicating in light) and quantum computing — frontier science with no near-term effect on what a token costs — and consumer buying advice, which is an editorial beat rather than research.
