Nvidia says the fight for AI memory will not be won by packing more chips onto a GPU — it will be won by designing every layer of the machine together. The company’s answer to a brutal memory market is what it calls full-stack codesign: GPUs, CPUs, networking, storage and software shaped as one system so that each byte of scarce high-bandwidth memory does more useful work.
The stakes are concrete. Agentic workloads push AI memory to its limits just as supply tightens and prices climb. Yet across the Blackwell Ultra and Vera Rubin generations of its AI memory roadmap, Nvidia grew HBM capacity per GPU by roughly 3% while lifting agentic throughput per dollar by up to 30x. This piece unpacks how that gap is possible — and why it matters for anyone who pays for AI.
Table of contents
- Why AI Memory Pressure Is Spiking
- Inside the AI Factory Memory Hierarchy
- Full-Stack Codesign: Nvidia’s Answer
- Shrink the Weights: NVFP4 and NVL72
- Squeeze the KV Cache and Offload Context
- Compact RAG: Keep the Index Out of HBM
- The Numbers: 3% Becomes 30x
- What the AI Memory Squeeze Means for Buyers
- References
Why AI Memory Pressure Is Spiking
Every inference request forces a GPU to consult two very different kinds of information. The first is the model’s parameters — trillions of values that encode everything the network learned during training. Frontier models now carry trillions of them, and each byte of parameters must sit close enough to the compute to be read constantly.
The second is the KV cache, which holds the context of each active conversation so the model does not reprocess the same tokens over and over. That cache grows with every word a user sends and every word the assistant writes back. Both residents compete for the same scarce, expensive real estate: high-bandwidth memory, or HBM — the AI memory tier every lab is fighting over.
Chatbots made this problem manageable. Agents made it ferocious. An agentic run strings together repeated model calls and tool calls while carrying prior context forward, and a single trajectory can process up to 3 million tokens. Working memory for one request therefore grows and grows instead of resetting each turn.
Retrieval-augmented generation (RAG) piles on more pressure by injecting retrieved enterprise information into prompts — valuable context that also has to be stored and retained. And this all lands while the physical supply of AI memory tightens and prices rise, a squeeze chip executives have been warning about publicly. The demand curve and the supply curve are moving in opposite directions.
The AI memory squeeze is not merely a workload story — it is a commodity story too. Memory makers are steering wafer and packaging capacity toward the AI memory tiers that hyperscalers pay premiums for, while consumer memory prices drift upward. Every agentic workload therefore arrives with two price tags: the engineering cost of managing scarce HBM, and the purchase cost of buying AI memory in a rising market. That double tag explains why Nvidia frames efficiency as a strategic feature rather than a benchmark footnote: in an AI memory market this brutal, tokens per dollar is the statistic CFOs actually read.
Inside the AI Factory Memory Hierarchy
Nvidia’s own analogy makes the constraint intuitive. Treat HBM as a warehouse and memory bandwidth as its loading docks. A bigger warehouse holds more inventory, but the docks decide how much moves in and out per hour. Once the docks saturate, extra shelves just sit there — adding cost and drawing power while response times slip past the service level the workload demands.
An AI factory therefore cannot keep everything in the fastest, most expensive place. What is in motion stays near the line. What is waiting moves to a larger, cheaper space nearby and returns before it is needed. That is a memory and storage hierarchy — and the table below shows the tiers Nvidia describes.
| Tier | Speed | Cost per bit | Typical resident |
|---|---|---|---|
| GPU HBM | Fastest | Highest, most contested | Model weights + hot KV cache |
| CPU memory (LPDDR via NVLink-C2C) | Fast | Lower | Paused context waiting to resume |
| Local SSD | Moderate | Lower still | Spill data local to one server |
| Shared storage / CMX | Slowest | Lowest | Idle context, knowledge indexes |
More HBM does mean more KV cache and more concurrent users — up to a point. Every added user shares the same compute and the same bandwidth, and once data can no longer arrive as fast as the compute can consume it, the system has hit what engineers call the memory wall.
Past that wall, extra capacity idles while still adding cost and power. The data still needs a home, but it does not need the fastest, most expensive one. That single observation drives every technique in the rest of this story.
Full-Stack Codesign: Nvidia's Answer
The reflexive answer to AI memory pressure is to bolt more HBM onto the GPU. Nvidia explicitly rejects that framing. Its first move is speed of adoption — the company was first to bring HBM3e and then HBM4 into an accelerated computing stack, working hand in hand with memory partners. But its deeper play is different: design the entire machine together so every byte earns its keep.
Full-stack AI memory codesign means GPUs, CPUs, networking, storage and software are intentionally shaped as one system rather than optimized in isolation. The goal is not more memory; it is more useful work per byte. How much useful work a byte does, in Nvidia’s telling, comes down to three levers: shrink what must be stored, place it on the right tier, and bring it back to the GPU before it is needed.
The codesign attacks three distinct sources of waste in the AI memory system, each with its own toolkit. The table maps the pressure points to Nvidia’s counter-moves — every one of them discussed in detail in the sections that follow.
| HBM pressure point | Codesign counter-move | Where it happens |
|---|---|---|
| Model weights | NVFP4 quantization + distribution across up to 72 GPUs | Precision formats, NVL72 domain |
| KV cache | NVFP4 KV cache + tiered offload via Vera CPU, Dynamo and CMX | Interconnect, inference software, storage |
| RAG knowledge | Smaller Nemotron embeddings + cuVS indexes kept outside HBM | Retriever models, CPU/storage libraries |
The economics rhyme with the warehouse analogy: a smart operator does not merely buy more warehouse. They shrink the boxes, move slow-selling stock off the prime shelving, and recall it before the next order ships. Codesign industrializes that discipline across silicon, systems and software.
Plenty of vendors sell more HBM, and taller stacks are coming. Nvidia’s argument is that bolted-on capacity collides with the memory wall, while codesign moves the wall itself — because wins compound across layers instead of saturating in one. A shrink at the precision layer, a redistribution at the system layer, and an offload at the storage layer relieve AI memory pressure from three directions at once, and the AI memory savings multiply rather than cancel. That is why the refrain at every Nvidia keynote runs two words long: in the AI memory era, the stack is the product.
Shrink the Weights: NVFP4 and NVL72
Weights are the heaviest inventory in the warehouse, and precision is how fat the boxes are. Nvidia’s NVFP4 format slashes the space weights need compared with FP8 — the lowest precision possible on the Hopper platform. Four-bit weights are no longer a thought experiment: recent releases shipped with NVFP4 checkpoints across the industry.
The adoption list reads like a who’s who of frontier labs — FLUX.2 [klein] from Black Forest Labs, Inkling from Thinking Machines Lab, Beam 1.0 from Reflection, Meta’s Muse Spark, Step 5 Preview from StepFun, and Nvidia’s own Nemotron 3.5 Super VL. Model-builder adoption at this scale signals that 4-bit precision is becoming a normal production choice for faster, more memory-efficient inference, not a research curiosity.
Shrinking each box is only half the move; the other half is rearranging shelves. Within an NVL72 system, model weights can be distributed across up to 72 GPUs, so every GPU stores only a slice of the model.
That redistribution changes what the local HBM is for. With weights occupying a smaller share, more of the fastest memory frees up for KV cache and concurrent users — the exact resources agentic workloads consume. A single big model stops being a single big tenant, and the warehouse starts running many small, dense orders instead of one overflowing one.
Squeeze the KV Cache and Offload Context
The context cache deserves the same discipline as weights. Applying NVFP4 to the KV cache shrinks its footprint relative to FP8, so substantially more conversation context stays resident in HBM before any offload is needed. Smaller footprint, same conversation — the cheapest capacity is the capacity you never have to move.
When context does go idle, it should wait somewhere close. The NVIDIA Vera CPU helps twice over. First, its faster tool execution shortens the time the KV cache sits inactive during tool calls — the shorter the pause, the less offloading matters. Second, its large LPDDR memory, connected to Rubin GPUs over the NVLink-C2C interconnect, gives paused context a nearby place to wait and return quickly.
For longer idle stretches, NVIDIA Dynamo inference software relocates context to NVIDIA CMX context memory storage — a tier engineered for capacity at far lower cost per bit than HBM. Unlike local SSDs, CMX is shared across servers; unlike enterprise storage, it skips the long-term data-protection overhead that ephemeral context does not need.
Dynamo’s KV cache reuse then ties the tiers into one resource pool spanning GPU memory, CPU memory and storage, so context computed once can be reused system-wide. Storage partners lining up behind CMX include Cloudian, DDN, Dell Technologies, Everpure, HPE, IBM, NetApp and VAST — a notable crowd for an AI memory category that barely existed a year ago.
Consider the counterfactual. Without tiering, every idle conversation would pin its KV cache in HBM, and operators would answer growth by buying more of the most expensive AI memory on earth. With tiering, idle state flows down to LPDDR and CMX while HBM keeps serving active requests, so capacity purchasing tracks active users rather than every attached user. In a rising AI memory market, that difference compounds straight into the invoice: the cheapest AI memory byte of 2026 is the one your factory never had to buy, and the cheapest AI memory upgrade is the workload change that makes the old configuration fit.
Compact RAG: Keep the Index Out of HBM
RAG converts enterprise documents into embeddings — numerical representations that let a system search by meaning instead of keyword — organized into a searchable index. Naively, that index is another tenant competing for HBM, and at enterprise scale it can dwarf the model itself.
Nvidia’s counter-move has two parts. Nemotron Retriever models produce smaller embeddings with higher retrieval accuracy — less inventory per document — while the NVIDIA cuVS library lets the index be built and served across CPU memory and storage rather than graphics memory.
The architectural consequence is clean: the knowledge base never occupies HBM at all. Only the handful of passages a model actually needs enter the live context window; everything else stays on cheaper tiers, indexed and ready.
It is the same discipline applied to weights and KV cache, generalized to enterprise knowledge: shrink the representation, place the bulk on the right tier, pull back only what the next request needs. For teams weighing RAG investments, the direction of travel matters — retrieval is becoming a CPU-and-storage workload that feeds AI memory budgets, not a GPU-resident tenant itself.
There is a quieter implication for AI memory architecture: retrieval has been drifting back out of the GPU. For a few years, vector indexes lived as close to the accelerator as possible because latency outranked cost. Smaller Nemotron embeddings and cuVS-style serving flip that tradeoff — enterprise knowledge becomes AI memory that resides on CPUs and networked storage at inference speed. Expect database and search vendors to feel the shift, because the HBM they hoped to occupy is precisely the AI memory tier Nvidia is busy vacating.
The Numbers: 3% Becomes 30x
Time to put the numbers on the table. On the SemiAnalysis AgentX DeepSeek V4 Pro workload, Nvidia reports that its GB300 NVL72 delivers up to 10x higher throughput per dollar than the H200 platform at 100 tokens per second per user — with HBM capacity per GPU roughly doubling. Then, at 125 tokens per second per user, the Vera Rubin NVL72 delivers up to 30x higher throughput per dollar than GB300 NVL72 — with HBM capacity per GPU up only about 3%.
One sentence frames the chart below: each generation turns slightly more memory into dramatically more work per dollar. That gap is the measurable footprint of codesign.
For AI factory operators, the number that matters is how many useful tokens the factory delivers within its power budget while still meeting response-time promises. More tokens per second per watt means more revenue-bearing work from the same wall socket, and a lower cost for every token served.
That is not an abstraction for anyone who has watched agent metering run hot — our piece on Garry Tan’s stunning $200K-a-year AI agent bill shows what token costs look like from the buyer’s side. Efficiency multiplied thirty times over is a different planet for that invoice, and Nvidia’s pitch is that codesign, not just bigger memory budgets, is how the industry gets there.
What the AI Memory Squeeze Means for Buyers
Nvidia’s pitch arrives against a genuinely ugly backdrop. AI memory supply is tight and prices are rising; its own chips are caught in geopolitical friction, as our report on Nvidia chips piling up in Hong Kong showed, and a European chip CEO has warned the AI boom is starving phones and PCs of wafers. In that world, the vendor who squeezes the most work from every byte holds the strongest card.
The comparison table below summarizes what changes for buyers across the platform generations Nvidia describes. The pattern: memory grows modestly, but the machinery around it — precision, distribution, offload, retrieval — does the heavy lifting.
| Platform generation | HBM per GPU | Throughput per dollar | Codesign emphasis |
|---|---|---|---|
| H200 (Hopper) | Baseline, FP8 floor | Baseline | HBM3e adoption |
| GB300 NVL72 (Blackwell Ultra) | Roughly 2x H200 | Up to 10x vs H200 | NVFP4 checkpoints, 72-GPU weight spread |
| Vera Rubin NVL72 | Up only ~3% vs GB300 | Up to 30x vs GB300 | HBM4, Vera CPU + NVLink-C2C, Dynamo + CMX tiering |
Two uncomfortable questions land on every other chipmaker and cloud provider. First: if the leaderboard is decided by system-level efficiency rather than memory gigabytes, what happens to vendors whose pitch is raw capacity? Second: if context, indexes and idle state increasingly live on shared storage tiers run by partners like IBM, Dell and HPE, where does the boundary of the GPU vendor’s empire sit?
Buyers of AI at scale — including the sovereign builders behind projects like our coverage of the Orion AI factory — should read the same signal twice: ask vendors for tokens-per-dollar-per-watt, not spec-sheet memory; and assume the memory hierarchy will keep reshuffling which components earn the margin.
Three practical moves follow from all of this. First, benchmark candidate platforms on tokens per dollar at your real context lengths: AI memory consumption in agentic runs scales with trajectory depth, not with the number printed on a spec sheet. Second, audit where your RAG indexes physically live — moving them off graphics memory frees the AI memory your own agents actually need. Third, watch the new storage tier: CMX-class products mean the AI memory hierarchy is becoming a purchasable category with its own price competition.
Efficiency, in short, is now a procurement decision. The AI memory wall is very real — but as of this week, climbing it is also an itemized line in a quote.
References
NVIDIA Blog: More Performance From Every Byte — How Full-Stack Codesign Solves AI’s Memory Challenge
NVIDIA Developer Blog: 3 Ways NVFP4 Accelerates AI Training and Inference
NVIDIA Dynamo Inference Software
NVIDIA CMX Context Memory Storage
Video Explainer: NVIDIA Full-Stack Codesign and Inference Memory Pressure
More AI coverage: explore Progressive Robot's AI Models, Tools & Releases hub — hands-on reviews, setup guides and benchmarks in one place.
