# The Hidden Infrastructure Crisis Behind Every AI Agent You're Running **Author:** WEKA **Published:** September 30, 2026 ![Black and white image of a data center with an open server rack revealing internal components and cables.](https://cdn.sanity.io/images/ult5g8gw/production/98dccdf4c8b4d6ee8a487794833b2b5b8cc7fac2-1408x768.jpg) ## TL;DR Expose the hidden memory bottlenecks draining AI agent economics and discover smarter context storage. - Quantify how KV cache hit rates drive inference cost and performance. - Explain why GPU memory tiers and legacy storage break at AI scale. - Show how 99% KV cache hits transform unit economics and user latency. The AI industry has a dirty secret: We're burning billions of dollars recomputing the same information over and over again. While many still obsess over model parameters and training costs, there's a massive infrastructure bottleneck quietly constraining the economics of AI, and it's getting worse every day. Vik Sekar, host of the Semi Doped podcast and author of the [Semi Doped newsletter](https://substack.com/@vikramskr/note/c-195752474) on Substack, recently spoke with WEKA’s Val Bercovici about the stark infrastructure realities organizations face in the era of agentic AI and how optimizing storage and memory is the key to unlocking AI’s true value. As the conversation began to unfold, Vik mentioned, “It's amazing that you're saying in 2017 nobody really thought about storage solutions too much. It was considered boring and probably a commodity. But fast-forward not even a decade, things have turned around entirely…. We want to store infinite context, and the need for storage has completely changed.” Among other topics, they discuss how maximizing existing memory through software is the right approach as inference quickly overtakes model training for the resources necessary to keep AI initiatives moving at full throttle. First things first: To understand what’s holding back AI economics from flourishing, it’s important to know the meaning of “KV cache hit rate,” which is the measurement of how often a system can reuse previously computed context from memory (aka KV cache) instead of recomputing it from scratch. Essentially, it's the percentage of time your model retrieves existing computations saved in memory instead of burning GPU cycles to recalculate the same information over and over. Leveraging KV cache is significantly faster, more efficient, and less expensive than repeated recomputation. Industry norms for KV cache hit rates hover around 50-60%, with emerging best practices pushing the rate up to 70%. Keep reading for more on how you can achieve 99% (that is not a typo!) KV cache hit rates, as well as other highlights from Vik and Val’s thought-provoking conversation. 👉 Learn more about [KV cache](/learn/ai-ml/what-is-kv-cache/) ## Why 100,000 tokens require 50 gigabytes of memory in AI systems Most organizations haven't fully grasped the simple and unavoidable fact that 100,000 tokens (roughly 1 MB of data) translates to 50 GB of KV cache working memory, an expansion of 50,000x: “That's because you're vectorizing,” Val explained. “You're adding 10,000 to 20,000 dimensions of intelligence to 100,000 tokens, this 1 MB of data.” This explosion happens at the beginning of every single user session. Now multiply this across hundreds of users launching agent swarms—dozens or hundreds of parallel sub-tasks per user—and you quickly understand why even the most advanced GPU servers are gasping for more memory to keep workloads running efficiently. Loading trillion-parameter models like Claude Opus 4.5 or DeepSeek R1 requires more memory than a single H100 can provide. By the time you add user prompts and context, you've already exhausted the GPU’s high-bandwidth memory (HBM) completely. Vik noted, “As long as you get KV cache hits, the cost of inferencing dramatically drops. If you need to recompute, it's a very expensive proposition.” This is why NVIDIA's announcement of the new [Rubin platform](https://nvidianews.nvidia.com/news/rubin-platform-ai-supercomputer) at CES 2026 is so important. By introducing context memory storage as a first-class citizen in the platform, NVIDIA is acknowledging what the industry has been quietly struggling with: Traditional storage simply cannot keep up with the memory demands of modern AI inference. (Read more about it in this popular WEKA blog, “[The Context Era Has Begun](/blog/ai-ml/the-context-era-has-begun/).”) ## How GPU memory tiers work: HBM vs. DRAM vs. NVMe storage Think of AI memory in three tiers: **Tier 1 (G1):** **HBM**, which lives on the GPU itself and operates at nanosecond speeds. It’s the fastest, but most scarce, resource. Each GPU in an eight-chip server has dedicated HBM, totaling about 1 to 2 TB of aggregate memory. **Tier 2 (G2):** **DRAM **(dynamic random access memory), which sits on the motherboard and gives you about 1 TB of shared memory across all GPUs, operating in microseconds. It’s three orders of magnitude slower than HBM, but still functional for inference workloads. **Traditional storage **constitutes Tier 3 and is where things historically fall apart. Legacy protocols like NFS and Lustre were built for spinning hard drives. Val noted the fundamental problem: “NFS and Lustre in particular, they're very, very popular in the AI world, were literally built around the era of hard disk drives. We have to remind ourselves, this is now history…. These are antiques.” When these protocols try to tier memory to storage, they introduce milliseconds of latency—or thousands of microseconds—because they were never designed for the complex memory semantics AI requires. Here's the problem: 4 TB of total memory per server isn't nearly enough. Applications like Cursor (the very powerful and popular AI coding tool) process tens of trillions of tokens daily, requiring exabyte-scale KV cache storage to maintain efficiency. Val emphasized the scale of this challenge: “This is why Jensen (Huang) called (AI) [the biggest future storage market of the world](https://finance.yahoo.com/news/sandisk-trending-nvidia-ceo-jensen-143114579.html), because when you're dealing with an exabyte of memory, you have to think about some kind of storage semantics.” 👉 Read more about [the memory wall](/learn/ai-ml/ai-memory-wall/) ## What KV cache hit rates mean for AI inference costs and performance The current state of inference economics needs substantial optimization. Most companies—including AI model providers, AI clouds, and enterprises deploying large AI initiatives—are operating at negative unit economics, and the culprit is constant recomputation. Every five minutes (or at best, every hour), systems evict KV cache and force redundant and expensive prefill operations to regenerate context that should have been preserved. But with a strategic context memory infrastructure, it’s possible to achieve 99%—yes, 99%—KV cache hit rates automatically. This changes the game in a huge way. Val emphasized the transformative impact: “That is the nirvana of context engineering when you effectively can automatically get 99% KV cache hit rates. There are no latencies for your users, for big agentic jobs, for voice agents in particular. And more importantly, you're very energy-efficient now because the really energy-intensive, compute-intensive process of prefill is done once and you decode forever for those tokens.” **The actionable insight here: **Check your inference provider's pricing transparency on platforms like [OpenRouter](https://openrouter.ai/). Look at the ratio between cached read tokens and input tokens. A 10:1 ratio is common today. If you see this narrowing to 3:1, 2:1, or even 1.5:1, you're looking at a provider with proper token infrastructure, which will directly benefit your application economics. For AI leaders, the message is clear. If you want positive unit economics, you need to stop endless recomputing, start storing context intelligently, and pay close attention to the memory layer. This is the next frontier of competitive advantage. 👉 For a deeper dive into Val and Vik’s conversation, watch [the full podcast episode](/resources/video/why-ai-storage-will-define-the-future-of-inference-at-scale/). And be sure to [subscribe to the Semi Doped Substack](https://substack.com/@vikramskr/note/c-195752474) for more conversations and commentary like this.