1 · The Memory Wall

The story starts with a paradox: GPUs keep getting faster, but memory can't keep up.

Over the past 20 years, server compute (FLOPS) has grown 3x every 2 years. DRAM bandwidth, meanwhile, has grown only 1.6x, and interconnect (the links between chips) only 1.4x over the same cycle. The gap between "how fast you compute" and "how fast you read data" keeps widening - engineers call this the Memory Wall.

The practical consequence: a GPU like NVIDIA's H100 packs thousands of compute cores running in parallel. When all of them request data from memory at once, requests queue up and cores sit idle. Engineers call this state compute starvation - the processor starving for data.

With AI, the problem gets worse. A large LLM (hundreds of billions of parameters) needs its entire weight set loaded into GPU memory before it can run inference. For every token generated, the model re-reads hundreds of gigabytes of data. AI workloads are almost entirely memory-bound - it's memory read speed, not compute speed, that decides whether a model runs fast or slow.

2 · HBM - High Bandwidth Memory

High Bandwidth Memory (HBM) solves the Memory Wall with an idea that's simple in concept but brutally hard to manufacture: stack multiple layers of DRAM on top of each other, wire them together with thousands of through-silicon vias, then place the whole stack right next to the GPU on the same substrate.

3D architecture: die stacking, TSVs, and the silicon interposer

An HBM stack consists of:

• Logic die (bottom layer): houses the memory controller, manages I/O, ECC, refresh, and power distribution.
• 8-12 layers of DRAM die stacked on top, each a separate DRAM memory chip.
• TSV (Through-Silicon Via): thousands of "copper pipes" running straight through the silicon, connecting the DRAM layers vertically. This is the backbone that lets data flow between layers without routing back out to the edge.
• Microbumps: tiny solder points between dies that provide the mechanical-electrical connection.

The whole HBM + GPU cluster sits on a silicon interposer - a thin silicon slab with thousands of microscopic wires. TSMC calls this technology CoWoS (Chip-on-Wafer-on-Substrate). The interposer enables a 1024-bit (HBM3) or 2048-bit (HBM4) bus between GPU and memory - 16-32 times wider than DDR5.

The "wide, slow, stacked" philosophy

DDR5 and GDDR6X run at very high clock speeds over a narrow bus. HBM goes the opposite way: an extremely wide bus at a moderate clock speed. Instead of pushing speed, HBM raises bandwidth through extreme parallelism - 16 independent channels, each split into 2 pseudo-channels, allowing simultaneous reads and writes.

A side benefit: because its clock speed is lower, HBM consumes less power per bit than GDDR. Less electromagnetic interference (EMI), less heat, less circuit-design complexity - which is why HBM suits data centers, where every watt saved multiplies across tens of thousands of GPUs.

3 · AI Data-Center GPUs: Who Sells What

The AI data-center GPU market today is almost synonymous with NVIDIA. Here are the three architecture generations currently running in major data centers:

The table makes the trend clear: every GPU generation roughly doubles its HBM capacity. From 80 GB (H100) to 192 GB (B200) to 288 GB (Vera Rubin). Each GPU needs 4-6 HBM stacks. Each data center needs tens of thousands of GPUs. Multiply it out - and HBM demand explodes.

GB200 NVL72: rack-scale AI

NVIDIA doesn't just sell individual GPUs. The GB200 NVL72 is a complete rack: 72 B200 GPUs + 36 Grace CPUs, linked via NVLink 5 at 1.8 TB/s per GPU. One such rack packs 72 x 192 GB = 13.8 TB of HBM3E - roughly equivalent to nearly 14,000 laptop DDR5 modules.

4 · Who Benefits? Three Memory-Chip Giants

Only three companies in the world manufacture HBM: SK Hynix (South Korea), Samsung Electronics (South Korea), and Micron Technology (United States). This is a near-absolute oligopoly.

Why does HBM push RAM prices up?

This is the core economic mechanism many people overlook:

Producing 1 GB of HBM consumes 3 times the silicon wafer of 1 GB of DDR5. The reasons:

• Lower bit density: HBM3 has a density of ~0.16 Gb/mm², while equivalent DDR4 reaches ~0.30 Gb/mm² - nearly 2x higher, because HBM must leave room for TSVs.
• Stacking yield loss: when 12 dies are stacked, the probability of a defect multiplies exponentially. One bad die means the whole stack is scrapped.
• Advanced packaging: the CoWoS interposer is its own separate silicon wafer, adding an expensive manufacturing step with limited capacity.

When the three makers shift 23% of their wafer capacity to HBM (which carries fatter margins), the wafer supply left for DDR5/LPDDR5 shrinks. Demand for DDR5 (from PCs, ordinary servers, smartphones) hasn't fallen. Falling supply + steady demand = DRAM prices rise.

The result: DRAM prices rose 90% in Q1 2026 alone versus Q4 2025. Samsung and SK Hynix have already raised HBM3E prices another ~20% for 2026. Micron has even decided to exit the consumer market entirely to concentrate its full capacity on AI data centers - where margins run many times higher.

5 · HBM's Engineering Challenges

HBM isn't a "silver bullet." The technology comes with several challenges manufacturers must solve:

Heat

12 stacked DRAM layers mean 12 heat sources piled on top of one another, right next to a GPU die running at hundreds of watts. Heat can't escape easily because it's "trapped" between layers. Data centers must use vapor chamber cooling or liquid cooling - adding significant infrastructure cost.

Power delivery network (PDN)

Thousands of parallel signal traces generate very large transient currents. The power delivery network must be designed with extreme precision - requiring dense power/ground grids and decoupling capacitors on both the interposer and substrate.

Signal integrity

With thousands of signal traces running in parallel across just a few millimeters, crosstalk and jitter become major issues. Solutions include matched trace lengths, differential signaling, and a separate clock domain (the command bus runs at half the frequency of the data bus).

Yield and packaging cost

CoWoS capacity is a physical bottleneck. TSMC is the near-exclusive supplier of CoWoS. Each interposer is its own silicon wafer, limited in size (reticle limit ~858 mm²). Bigger GPUs with more HBM stacks require bigger interposers - which are harder to manufacture.

6 · The Paradox: Expensive GPUs, Sitting Idle

Here is the biggest paradox in AI infrastructure: companies pour billions of dollars into GPUs, then leave them sitting idle most of the time.

A GB200 NVL72 rack costs around $3-4 million. At 30-40% utilization, that means $1.2-1.6 million worth of hardware is sitting idle waiting for data from memory at any given moment. Multiply that across thousands of racks in a large data center - and the waste runs into hundreds of millions of dollars.

Why don't GPUs run at full capacity?

Many assume GPUs sit idle for lack of workload. In reality, the main cause is insufficient memory bandwidth to "feed" the GPU:

• Compute starvation during inference: During training, large batch sizes help amortize latency. But during inference (especially single-user), each query must read the entire set of model weights from HBM. The GPU has thousands of tensor cores but only a handful of memory channels - cores queue up waiting for data. Once memory controller utilization exceeds 80%, tensor cores are effectively sitting idle waiting for data.
• Batch size mismatch: AI inference typically runs small batches, not enough to fill the GPU pipeline. The result: compute units sit idle while the memory unit runs at full load.
• Networking bottleneck: In multi-GPU training, data must be synchronized across GPUs over NVLink or InfiniBand. If the interconnect is slower than compute, the GPU waits on synchronization. Optical interconnect is becoming the next bottleneck - if optics don't scale, GPUs keep sitting idle.
• Scheduling overhead: Container startup, model loading, queue wait - every time a job switches, the GPU has to reload model weights (hundreds of GB) into HBM. Load time equals idle time.

Training vs. inference: two different problems

Training is less affected because large batch sizes (thousands of samples per batch) help amortize memory latency. GPUs run near full utilization in continuous training workloads.

Inference is a much bigger problem. Every user query is effectively batch size 1. The model must re-read its entire weight set for every token generated. This is why GPU inference throughput often maxes out at 40-100 tokens per second, while architectures built specifically for inference (like Groq's LPU, with on-chip SRAM and ~80 TB/s bandwidth) hit 300-750 tokens per second. The difference doesn't come from compute power - it comes from memory read speed.

7 · DeepSeek Engram - Breaking HBM's Monopoly?

In January 2026, DeepSeek published a paper that shook the field: Engram - a "conditional memory" architecture that separates static knowledge from dynamic reasoning, cutting reliance on expensive HBM.

The core idea

Traditional transformers force every layer to simultaneously handle two jobs: (1) looking up facts ("Paris is the capital of France") and (2) reasoning ("if X then Y"). Both run on the GPU, both read from HBM.

Engram splits these two jobs apart:

How it works (technical)

Tokenizer Compression: Engram normalizes equivalent tokens (e.g., "Apple," "APPLE," "apple") into one canonical form, shrinking vocabulary size by 23%. Pipeline: NFKC - NFD - diacritic stripping - lowercasing.

Multi-Head Hashing: Instead of a massive N-gram lookup table (infeasible at scale), Engram uses K different hash functions for each N-gram level. Lookup costs only O(1) - no attention, no GPU compute required. Results from the hash heads are aggregated to reduce the impact of collisions.

Context-Aware Gating: Not every looked-up piece of information fits the context. A gating mechanism uses the model's hidden state as a query to check: does the retrieved embedding actually fit the sentence being processed? If not - discard it. If it does - fold it into the compute stream.

Benchmark results

Optimal ratio: 20-25% of the parameter budget for Engram, 40% for MoE. Engram-27B cuts the expert count from 72 down to 55, redirecting the freed parameters into a 5.7B embedding module at layers 2 and 15.

Implications for the industry

If Engram scales (and DeepSeek has already open-sourced the module), the consequences are significant:

• Reduced pressure on HBM: AI models could run on "moderate HBM + large DRAM" configurations instead of packing everything into expensive HBM.
• Rising demand for DRAM/CXL: If static knowledge shifts to system memory, demand for DRAM capacity will surge - benefiting all three memory-chip makers, though at lower margins than HBM.
• Geopolitical angle: China faces restrictions on buying high-end GPUs (and the HBM that comes with them). Engram would let competitive models run on weaker hardware - blunting the effectiveness of US chip export bans.
• Future architecture: Instead of treating compute and memory as one block, split them into two independently scalable resources - the same way cloud computing separated compute from storage.

8 · Looking Ahead

The HBM market is projected to reach $54.6 billion in 2026 (up 58% YoY) and keep growing as every GPU generation demands more memory. "Memory Supercycle" is the term analysts use to describe this super-growth phase for memory chips.

But Engram and architectures like it raise a question: will HBM still be the only bottleneck five years from now? Or will the AI industry learn to decouple memory from compute - the way it decoupled storage from compute a decade ago?