# Storage Got Promoted, and the New Job is Context Memory

**Author:** WEKA

**Published:** November 11, 2026

![A black-and-white photo of a tall metal drawer unit resting on a stone block, with one open drawer revealing a stack of papers tied with string.](https://cdn.sanity.io/images/ult5g8gw/production/7abb73ddae74b07173c1ee1895990314739a1261-1376x768.jpg)

## TL;DR

Explore how context memory helps AI clusters reuse KV cache and reduce redundant computation.

- Understand why GPU memory cannot keep pace with growing context.
- Assess whether your workloads need a context memory tier.
- Track goodput, cache hits, and redundant prefills.

Last November, [we argued](/article/ai-storage-is-the-new-battleground-for-inference-at-scale) that AI storage would become the battleground for inference at scale. A year later, this thesis is proving out.

The question stopped being how fast you can feed a GPU to train models.The focus transitioned to where you put the memory a GPU no longer has room for as it computes complex inference queries. This shift defined a new storage layer called “[context memory](/learn/ai-ml/what-is-context-memory-primary/)” and it’s showing up as a dedicated tier in cluster architectures that didn’t have one a year ago.

## Why storage became a memory problem in 2026

WEKA’s [Betsy Chernoff](https://www.linkedin.com/in/betsy-chernoff-68ba256/) said during an [interview on theCUBE](https://www.thecube.net/events/nvidia/nvidia-gtc-2026/content/Videos/1cb2e38a-808c-4c30-a5af-36e23169d677) at NVIDIA GTC 2026:

“Storage got a promotion this year. There’s a new third job for storage in AI clusters, beyond keeping GPUs humming with hot data or shared storage across the network. The third job: dedicated nodes specifically for storing context memory or [KV (key-value) cache](/learn/ai-ml/what-is-kv-cache/). A completely new tier of storage in an AI cluster.”

The hardware industry is now building for exactly this development. At FMS 2026, [SK Hynix and SanDisk announced](https://www.prnewswire.com/news-releases/sk-hynix-unveils-first-hbf-standard-specifications-with-sandisk-presenting-ai-memory-solutions-at-fms-2026-302841792.html) the first specs for High-Bandwidth Flash: a NAND-based memory layer that sits between HBM and SSDs.

Betsy added that the industry had almost no warning: “The demand for this was not baked into anyone’s forecast a year ago.”

WEKA’s [Val Bercovici](https://www.linkedin.com/in/valentinbercovici/) then [watched it turn ever since](https://www.youtube.com/watch?v=HwIuEEILths): “At GTC last year, we first started talking about pressure on AI memory in the form of KV cache. It was a very lonely conversation.”

It isn’t lonely anymore. Coding agents changed the dynamic, and by mid-2026, [SemiAnalysis](https://www.linkedin.com/company/semianalysis) data showed [agent traffic had surpassed all human traffic](https://x.com/semianalysis_/status/2062580333770408231?s=12) on the internet.

## Why faster GPUs can’t close the memory gap

This isn’t a problem that better silicon can solve. It’s a math problem. “Even the new Vera Rubin NVL-72 has about 70TB of combined memory. That’s a lot, but when context memory is at petabyte-scale, you hit the wall. Without storage for that data, you have to recompute on the fly,” Betsy said.

And it's about to get tighter: [SemiAnalysis reports](https://newsletter.semianalysis.com/p/vera-rubin-nvl72-vs-gb200-nvl72-inference) that SOCAMM DRAM capacity per Vera Rubin NVL72 rack is being cut in half.

The gap widens with every generation of new gear.

## Why nobody solved this sooner

People tried. Predictive approaches like [MEXT](https://www.mext.ai/)'s are still chasing it from the DRAM side, but agent swarm access patterns are random by nature, which is precisely what defeats prefetch.

The tiering had a hole in it. [Solidigm](https://www.linkedin.com/company/solidigmtechnology/)’s [Ace Stryker](https://www.linkedin.com/in/acestryker/) said: “Traditionally, KV cache tiering flowed through HBM and DRAM, then cold storage (SSDs), with nothing in between, no warm state to hold data for fast reuse back to hot state.”

No warm tier meant no economics. This is what changed.

## What “prefill once, decode forever” actually means for your token bill

Most inference clusters, as [NVIDIA has pointed out](https://developer.nvidia.com/blog/nvidia-vera-cpu-boosts-ai-factory-throughput-to-accelerate-agentic-workloads/?ncid=so-link-736732&linkId=100000429700576), pay to compute the same thing over and over. Val noted, “Today, a busy GPU may be pre-filling redundantly. As new prompts come in, all old context gets evicted and refilled. There’s throughput vs. ‘goodput’ — useful, efficient work. For a long-running agent, instead of refilling 1 million times across 10,000 turns across a day, you prefill once. It’s an order of one, not an order of N.”

📚 Learn more: “[Prefill and Decode: A Technical Guide to the Two Phases of Inference](/learn/ai-ml/prefill-and-decode/)”

This is a change in complexity class, and it shows up in wall clock time: “Day-long agent runs become hour-long runs,” Val added. The measured result, using [WEKA Augmented Memory Grid](/product/augmented-memory-grid/)™ across two racks of Firmus (https://www.linkedin.com/company/firmustechnologies/) GPUs, is “6.5x more tokens from the same CapEx and OpEx. 550% more tokens.” Read more about it in this [case study](/customers/firmus/).

## Does every AI workload need a context memory tier?

No. And be suspicious of anyone who tells you otherwise.

[Animesh Singh](https://www.linkedin.com/in/animeshsingh1/), who runs the GPU fleet behind LinkedIn’s AI platform, has deliberately [not gone there](/video/linkedin-s-ai-infrastructure-secrets-for-1-2b-users): “We started with the hypothesis that we’d need to go from HBM to DRAM to SSD-based storage for certain use cases. So far, most optimizations using KV cache and HBM have been successful. At most, we’re defaulting to CPU memory on GPU nodes. We still haven’t needed to go to external SSDs.”

His reasoning is about complexity, not performance, and it fits his workload: recommendation and ranking models, 7 to 8 billion active parameters, single tenant. Val’s reply is worth sitting with: “Memory tier performance is essential, especially as GPUs get faster and prefills get faster, you still have a high bar for KV cache offloading.”

The bar rises as the silicon improves. “Not yet” is a real answer with an expiration date, and long-context agentic workloads shorten it.

## How to tell whether this tier belongs in your architecture

Here are three checks you can run this quarter:

1. **Measure “goodput,” not utilization.** A busy GPU and a productive GPU aren’t the same thing. The Solidigm analysis above found “real GPU utilization in big clusters runs 50 to 80%. Storage is often the culprit.” We also wrote about this in a previous WEKA Wire, check it out here: [The Difference Between Goodput and Throughput? Your Entire AI Budget.](https://www.linkedin.com/pulse/difference-between-throughput-goodput-your-entire-ai-budget-weka-io-98bje/)
2. **Track KV cache hit rate as a first-class metric.** Shimon lists KV cache hit rate as a core requirement for production inference, alongside a token warehouse built “in a simple, secure, multi-tenant way.” (👉 Check this out: [ICML research](https://x.com/brendanburkex/status/2077412073840214193?s=46) found long-horizon agent workloads suffer premature KV cache evictions in both vLLM and SGLang. Your hit rate may be lower than your dashboard suggests.)
3. **Count your redundant prefills.** If your agents re-read the same context every turn, you already know your answer.

Your cluster diagram lists two storage jobs. Your agents are running the new third right now.

👉 Want to go deeper on what you can do to prevent running out of memory? Read [this article](/learn/ai-ml/ai-memory-wall/).