Processing-In-Memory (PIM) & Memory-Centric Computing

Processing-In-Memory reframes the oldest assumption in computing: that memory is passive and the CPU does the work. In modern AI and consumer workloads, 60–90% of total system energy is data movement, not computation — a DRAM access costs 800× a floating-point op, and up to 64,000× when sensors and storage are included. Meanwhile, state-of-the-art data-center processors spend 80–90% of their time waiting for memory. The implication: the AI compute shortage is substantially a memory-movement shortage, and the correct fix is not more cores but computation at the data.

Two commercial trajectories are converging on this idea. Near-memory approaches (HBM4E/C-HBM4E with logic on the base die, UPMEM-style bank-level processors, Samsung HBM-PIM, SK Hynix AiM) integrate conventional logic adjacent to memory. Using-memory approaches (RowClone, Ambit) exploit the analog behavior of DRAM cells to compute within the array — bulk copy via consecutive row activations, bitwise AND/OR/NOT/majority via concurrent multi-row activation. Critically, researchers (SAFARI / ETH Zurich, Mutlu group) have shown several of these ops run reliably on unmodified, commodity DRAM by violating nominal timing parameters.

Reliability scaling is forcing intelligence into memory regardless of whether the industry wants PIM. RowHammer, RowPress, and column-disturbance mechanisms make modern DRAM a physical-security liability without on-die logic. DDR5 already embeds activation counters; this is the first rung on a ladder that ends at Self-Managing DRAM — memory that schedules its own refresh and defenses and can signal "not now" to the CPU.

The bottleneck is no longer physics — it's paradigm. JEDEC (~390 companies) rarely converges on radical interface changes; the "Self-Managing DRAM" paper was rejected six times over three and a half years. Onur Mutlu frames the full shift as a "Copernican Revolution" likely to take decades. Near-term, the investable surface is narrower: HBM4E/C-HBM4E base-die compute, commercial PIM DRAM (HBM-PIM, AiM, UPMEM), CXL 3.x composable memory, and hyperscaler topology-aware software stacks that can exploit any of it.

Key Claims

• 60–90% of system energy is data movement across consumer apps (Chrome, video codecs, TF inference) and ML workloads (LSTMs, transducers). Evidence: moderate — single-researcher framing, needs independent corroboration (Mutlu synthesis)
• 80–90% of data-center processor time is spent waiting on memory. Evidence: moderate, attributed to Google (Mutlu synthesis)
• DRAM access ≈ 800× FP op, 6,400× int add, 64,000× including storage/sensors. Evidence: moderate (Mutlu synthesis)
• RowClone and Ambit operate on unmodified DRAM by violating nominal timing parameters. Evidence: moderate, demonstrated in lab (Mutlu synthesis)
• HBM4E base dies on TSMC N3P enable C-HBM4E near-memory compute — one of three C-HBM4E customization levels. Evidence: strong (HBM4 Shakeup)
• DDR5 embeds activation counters for RowHammer defense. Evidence: strong — ratified spec (Mutlu synthesis)
• RowPress induces bit flips with orders-of-magnitude fewer activations than RowHammer. Evidence: moderate — recent finding (Mutlu synthesis)
• UPMEM acquired by Qualcomm (June 2025) — RISC-V DPU cores embedded in standard DDR4/DDR5 DIMMs; up to 259× speedup for large-batch MLP inference documented. Evidence: strong — acquisition confirmed by PitchBook and CB Insights; technical architecture documented in multiple arXiv surveys (Mutlu synthesis)
• Samsung HBM-PIM (Aquabolt-XL) — 2.5× system performance, 60% energy reduction measured on Xilinx Virtex Ultrascale+ (Alveo) AI accelerator. Evidence: strong — Samsung + Hot Chips 33 disclosed (Mutlu synthesis)
• SK Hynix AiMX card (32 GB, GDDR6-AiM) ran Llama 2 70B at Hot Chips 2024 / AI HW Summit 2024; 1.25 V operating voltage vs 1.35 V standard → ~80% data-movement power reduction. Evidence: strong — SK Hynix technical disclosure (Mutlu synthesis)
• RowPress documented on commodity DDR4 — bit flips with orders-of-magnitude fewer activations than RowHammer. Evidence: strong — arXiv:2406.16153 (Luo et al., SAFARI) (Mutlu synthesis)
• Self-Managing DRAM framework for in-DRAM autonomous operations — appeared at MICRO 2024 (Yaglikci, Luo, Mutlu). Evidence: strong — peer-reviewed venue (Mutlu synthesis)
• RowHammer paper won 2024 Jean-Claude Laprie Award for dependable computing — signals field-wide recognition of the reliability angle. Evidence: strong (Mutlu synthesis)
• Only hot KV blocks benefit from near-memory compute — uniform HBM-PIM stacks make all layers pay for PIM logic; vertical heterogeneity (dense capacity layers + PIM compute layers) is the proposed fix. Evidence: moderate — architecture proposal, abstract-level only (TokenStack)
• TokenStack: 1.62× geomean token throughput, 1.70× SLO-compliant capacity, 30–47% lower per-token energy vs AttAcc. Evidence: moderate — academic result vs an academic baseline on production-derived traces; NOT a product measurement, full text not ingested (TokenStack)
• PIM's leading candidate workload is hot KV-cache attention during LLM decode — not vector-DB or recommendation serving. Evidence: moderate — one architecture paper, but it is the first with a named, large workload (TokenStack)
• StreamDQ (SK Hynix): near-memory weight dequantization on the HBM base die — DeQuantization Blocks unpack INT4/INT8/FP8 weights to FP16/BF16 on standard loads, bypassing GPU CUDA-core dequant with no ISA change; up to 7.08× GEMM speedup, 90.23% lower energy, 54.68% end-to-end latency reduction, 2.20× decode throughput; per-DQB 0.127 mm² / 0.355 W in 12 nm, 11.36 W/stack across 32 DQBs. First primary memory-maker technical source in this KB. Evidence: moderate — SK Hynix preprint, authors' own simulated/synthesized results, not a shipping product; advantage shrinks at large batch (StreamDQ)
• AMMA: replacing GPU compute dies with 16 HBM-PNM cubes (4×4 mesh, UCIe 3.0 D2D links) removes the GPU from the center of decode attention — 44 TB/s aggregate package bandwidth at 240 W (vs H100's 3.35 TB/s at 700 W), 15.5× lower attention latency and 6.9× lower energy vs H100 at 1M-token context (also 1.1–2.5× vs Rubin, 1.4–3.4× vs NeuPIMs depending on sequence length/workload); names NVIDIA Rubin's GPU-LPU platform as the GPU-centric baseline it argues against. DSE finding: per-cube compute saturates ~96 TFLOPS for GQA decode before bandwidth becomes the bottleneck, and the reordered-collective comms scheme scales 2.7× (8K tokens) → 65.4× (1M tokens). Evidence: moderate — preprint, simulated results vs H100/Rubin/NeuPIMs baselines, not independently replicated; full text ingested 2026-07-24 (AMMA)

Two Approaches Compared

Dimension	Processing Near Memory (PNM)	Processing Using Memory (PUM)
Where compute lives	Logic layer in 3D-stack / on HBM base die / per DRAM bank	Inside the DRAM cell array itself
Logic used	Conventional ALUs, processors	Row activations, charge sharing
Examples	C-HBM4E NMC, HBM-PIM, AiM, UPMEM	RowClone, Ambit
Workload fit	General-purpose + ML kernels	Bulk memcpy, bitwise ops, RNG
Productization status	Commercial (Samsung, SK Hynix, UPMEM) + roadmapped (C-HBM4E 2026–27)	Research + unmodified-DRAM demonstrations
Programming model	Firmware + runtime extensions; CUDA-adjacent	New primitives; ISA-level changes required
Key barrier	Topology-aware software, yield, heat	JEDEC interface, determinism guarantees

Energy Arithmetic

Operation	Relative Energy (vs 64-bit FP op)
64-bit FP multiply-add	1×
32-bit integer add	~0.1×
DRAM read/write (access)	800×
DRAM access vs 32-bit int add	6,400×
DRAM + storage + sensor chain	~64,000×

Reliability Pressure Forcing Memory Intelligence

Mechanism	What it is	Status
RowHammer	Repeated row activations leak charge into adjacent rows → bit flips	Exploited in the wild; DDR5 activation counters partially mitigate
RowPress	Long-held row activation induces flips with orders-of-magnitude fewer activations	Recently discovered; no standardized defense yet
Column disturbance	Thousands of rows affected simultaneously	Newly identified
DDR5 activation counters	On-die logic triggers adjacent-row refresh	Shipping
Self-Managing DRAM	Memory schedules own refresh/defense, can defer CPU requests	Research; Mutlu paper accepted after 6 rejections

PIM Finds Its Killer Workload: Hot KV-Cache Attention

The standing objection to PIM has always been "compelling energy arithmetic, no workload." As of mid-2026 there is a specific candidate answer, and it is the KV cache — because during decode every new token re-reads all prior KV state, making attention a bandwidth- and capacity-heavy memory task rather than a compute task (KV Cache Management, Prefill/Decode Disaggregation).

TokenStack (arXiv:2605.05639, May 2026) is the first HBM4-era PIM architecture in this KB with a named workload and a quantified result. Its central observation reframes the design problem: only hot KV blocks benefit from near-memory compute. Weights, activations and cold KV mainly need dense storage and GPU-visible bandwidth. So:

• A uniform HBM-PIM stack makes every layer pay for PIM logic it mostly doesn't use.
• A dedicated-PIM design (their baseline, AttAcc) recovers capacity but shrinks the HBM bandwidth left for GPU-side work.
• TokenStack's answer is vertical heterogeneity within the stack — dense capacity layers plus PIM-enabled compute layers, with the HBM4 logic base die as a stack-local controller for cross-layer DMA, layered address translation, attention-side gather/broadcast, and inline quantization during migration. Software side: topology-aware KV placement, workload-aware eviction, bounded replication.

Reported: 1.62× geomean token throughput, 1.70× SLO-compliant serving capacity, 30–47% lower per-token energy vs AttAcc on production-derived traces across four models.

How much this actually resolves. The frontier's standing gap "commercial PIM product specs" is not closed by this — TokenStack is an academic architecture measured against an academic baseline, abstract-level in this KB, with no memory maker committing to a heterogeneous-layer stack on any public roadmap. What it does close is the narrower and more interesting gap: what C-HBM4E level-3 near-memory compute would be for. The answer is no longer "vector-DB and recommendation, probably" — it is hot-KV attention in LLM decode, which is the largest and fastest-growing inference workload in existence.

The Base Die Stops Being Passive: Three Works, One From a Memory Maker (Jul 2026)

The load-bearing shift in mid-2026 is not any single benchmark — it is that memory has stopped being modelled as a passive component. For fifty years the base die under an HBM stack was a dumb byte-mover: it stored, refreshed, and returned data, and every arithmetic op happened on the processor. Three independent 2026 works now argue the base die should do arithmetic, and they map a spectrum of how far the GPU gets displaced. Critically, one of the three is authored by a memory maker itself — the first time in this KB that the party who actually fabricates the stack has put its name on a "the base die computes" design.

Work	Where compute goes	What the base die does	Displacement of the GPU	Headline result	Source authority
TokenStack (May 2026, Peking Univ. per registry)	PIM layers inside the stack	Stack-local controller: cross-layer DMA, address translation, gather/broadcast, inline quantization	Adds PIM alongside the GPU	1.62× throughput, 1.70× SLO capacity, −30–47% energy vs AttAcc	Academic preprint, academic baseline
StreamDQ (Jul 2026, SK Hynix)	DeQuantization Blocks on the base die	Inline weight dequantization (INT4/INT8/FP8→FP16/BF16) on standard loads, one DQB per pseudo-channel controller, 3-bit sideband tag, no GPU ISA change	Offloads dequant from the GPU; GPU still runs the GEMM	7.08× GEMM, −90.23% energy, −54.68% end-to-end latency, 2.20× decode throughput; 0.127 mm² + 0.355 W per DQB in 12 nm	Preprint, but from the memory maker — simulated/synthesized, not a shipping product
AMMA (Apr 2026, UC San Diego / Columbia / Yonsei / NVIDIA / Samsung Semiconductor)	16 HBM-PNM cubes replace the GPU compute dies	Per-cube logic-die microarch for decode attention in a 4×4 UCIe-3.0 multi-chiplet mesh (44 TB/s / 240 W package, ~2× bandwidth vs H100)	Removes the GPU compute die entirely from decode attention	15.5× lower attention latency, 6.9× lower energy vs H100 at 1M-token context; per-cube compute saturates ~96 TFLOPS then bandwidth-bound	Academic + NVIDIA/Samsung preprint, H100/Rubin/NeuPIMs baselines

Why StreamDQ is the different kind of signal. TokenStack and AMMA are academic architectures arguing what a memory maker should build. StreamDQ is SK Hynix engineers (Icheon) describing what SK Hynix intends to put on a custom-HBM base die — the first primary memory-maker technical source this KB has held on any subject. It is still a research preprint with simulated results, not a committed shipping product with a spec sheet, and its own advantage shrinks at large batch (in memory-bound / small-batch decode dequant is a large fraction of runtime; in compute-bound large-batch GEMM it is not). But the provenance is categorically different from an academic proposal: the party who fabricates the stack is now on record that the base die should compute.

The architecture side of the memory-maker gap narrows; the supply side does not. This is the distinction to hold precisely. StreamDQ closes — partially — the "no primary memory-maker source at all" gap on the architecture / near-memory axis. It does nothing for the supply, ASP, capacity, and allocation axis, which still rests on one stale January outlook plus TrendForce figures cited second-hand. That second gap closes, or doesn't, at the SK Hynix (2026-07-29) and Samsung (2026-07-30) prints — both next week, both desk-verified against the makers' own filings. See SK Hynix.

Commercial Signals to Track

• UPMEM → Qualcomm acquisition (June 2025, confirmed) — first pure-play PIM exit to a major semi. Tech: RISC-V DPU cores per DDR4/DDR5 DIMM. Watch for post-deal product announcements integrating UPMEM into Qualcomm's AI-PC / edge-AI / data-center stack.
• Samsung HBM-PIM (Aquabolt-XL) — HBM2-based, validated at 2.5× perf and 60% energy cut on Xilinx Alveo. No confirmed hyperscaler production deployment yet — watch for HBM3/HBM4-PIM successor products as Samsung scales HBM capacity +50% in 2026.
• SK Hynix AiMX (GDDR6-AiM) — 32 GB card running Llama 2 70B demoed at Hot Chips 2024; 80% data-movement power savings. Watch for LPDDR6-AiM variant targeting on-device AI (CES 2026 roadmap hint).
• C-HBM4E near-memory compute — level-3 customization in the HBM4E roadmap; hyperscaler topology-aware stacks are the gating item. NVIDIA Rubin Ultra (1 TB HBM4E, 8 stacks) is the natural first host.
• CXL 3.x composable memory — memory disaggregation sets the interface conditions for memory-centric design at rack scale.
• JEDEC DDR6 spec drafts — watch for activation counter evolution and any hint of CPU-memory interface flexibility (Self-Managing DRAM primitives would be the breakthrough signal).
• RowPress and column-disturbance exploits in the wild — first published CVE will force a JEDEC response and validate the "reliability forces intelligence" thesis.

Investment Implications (preliminary)

• Reframes NVIDIA's moat from FLOPs to memory-system integration (NVLink, Grace–Hopper coherence, rack-scale memory bandwidth).
• Elevates memory makers — SK Hynix, Samsung Memory, Micron — from capacity suppliers to active-compute participants as C-HBM4E NMC and HBM-PIM/AiM gain traction.
• Fabric & interface layer (Astera Labs, Marvell, Broadcom, Rambus, Synopsys/Cadence IP) is the arbitrage seat as CXL 3.x and custom D2D interfaces proliferate.
• Standards-body dynamics are a genuine alpha source: JEDEC activity is a forward indicator 3–5 years out.
• Pure-play PIM is thin — UPMEM was the notable independent; post-acquisition this is mostly a "hidden inside an incumbent" story.

Open Questions

• Do the 60–90% data-movement energy figures replicate on independent hyperscaler workloads (Meta, Microsoft benchmarks)?
• Does C-HBM4E NMC find a production workload beyond vector-DB / recommendation serving, or does it stay a niche accelerator? (Partially answered 2026-07-22/23: TokenStack names hot-KV attention; StreamDQ adds inline weight dequantization, and it comes from an actual memory maker (SK Hynix). Still open on the "production" half — both are preprints with simulated results, neither is a shipping product with a spec sheet.)
• Does SK Hynix's StreamDQ base-die-dequant design (or anything like it) appear on a committed custom-HBM base-die roadmap, or does it stay a research preprint? The 2026-07-29 print is the first place a roadmap commitment could surface.
• Is a vertically heterogeneous HBM stack (some layers dense, some PIM-enabled) manufacturable at yield and thermally viable, or does per-layer differentiation break the stack economics?
• Does any of SK Hynix / Samsung / Micron put a heterogeneous-layer or KV-targeted PIM stack on a public roadmap?
• What's the realistic 3-year PIM TAM — is it a 5% sidecar to HBM, or a 30%+ reshaping of memory ASP?
• Do hyperscaler custom stacks (TPU, Trainium, MTIA, Maia) get first or last access to C-HBM4E topology-aware software?
• Does a CXL 3.x-native memory-centric reference architecture emerge, or does it stay a loose constellation of vendor-specific features?
• What's the attacker's-view timeline on RowPress exploits in the wild, and does it force a JEDEC spec revision?

Related Concepts

• HBM4 Memory Architecture — C-HBM4E near-memory compute is the first commercial wedge of PIM into mainstream AI accelerators
• Custom Silicon vs GPU — memory-system integration is where ASICs either catch NVIDIA or don't
• Rack-Scale AI Compute — rack-as-product design lets hyperscalers ship topology-aware memory software
• KV Cache Management — the workload PIM has been looking for
• Prefill/Decode Disaggregation — decode is the memory-bound phase where PIM would pay

Backlinks

Pages that reference this concept:

• Memory-Centric Computing (Mutlu)
• HBM4 Architectural Shakeup
• StreamDQ — Near-Memory Weight Dequantization (SK Hynix)
• AMMA — Multi-Chiplet Memory-Centric Attention
• KV Cache Management
• SK Hynix

Changelog

• 2026-07-24 — Deepened AMMA (arXiv:2604.26103) from abstract-level to full text (arXiv HTML v2): added package-level magnitudes (16 HBM-PNM cubes, 4×4 mesh, UCIe 3.0 D2D links; 44 TB/s aggregate bandwidth / 240 W vs H100's 3.35 TB/s / 700 W), per-baseline detail (vs Rubin 1.1–2.5×, vs NeuPIMs 1.4–3.4×), and the DSE finding (per-cube compute saturates ~96 TFLOPS then bandwidth-bound; comms-reordering speedup 2.7×→65.4× from 8K→1M sequence length). Institutions resolved: UC San Diego, Columbia, Yonsei, NVIDIA, Samsung Semiconductor. No new sources.

• 2026-07-23 — Added StreamDQ (arXiv:2607.08993, SK Hynix — the KB's first primary memory-maker technical source) and AMMA (arXiv:2604.26103). Added the "base die stops being passive: three works, one from a memory maker" spectrum (TokenStack adds PIM alongside the GPU → StreamDQ offloads dequant from it → AMMA removes the GPU compute die entirely). Made the memory-stops-being-a-passive-component framing explicit and recorded the architecture-vs-supply distinction: StreamDQ narrows the memory-maker gap on the near-memory/architecture axis but not on supply/ASP/capacity/allocation, which the 2026-07-29 (SK Hynix) and 2026-07-30 (Samsung) prints address. Flagged StreamDQ as a simulated preprint, not a shipping product. +2 sources.

• 2026-07-22 — Added TokenStack (arXiv:2605.05639) as the first HBM4-era PIM architecture with a named workload and quantified result (hot-KV attention; 1.62× throughput / 1.70× SLO capacity / 30–47% energy vs AttAcc). Recorded the "only hot KV benefits from NMC" design argument. Explicitly scoped what this does and does not resolve of the frontier's PIM gaps — it answers what NMC is for, not commercial product specs. +1 source.

• 2026-04-21 — Initial compilation from Mutlu (ETH Zurich / SAFARI) synthesis. Cross-linked to HBM4 concept since C-HBM4E NMC is the commercial bridge.

• 2026-04-21 — Primary-source pass: UPMEM/Qualcomm acquisition confirmed (June 2025); Aquabolt-XL numbers (2.5× / 60%) and AiMX/Llama 2 70B demo added; RowPress (arXiv:2406.16153) and Self-Managing DRAM (MICRO 2024) cited; RowHammer paper's 2024 Jean-Claude Laprie Award noted. Evidence levels upgraded accordingly.