# Tokens, Memory, and the Paradox Nobody Priced In **Author:** WEKA **Published:** October 12, 2026 ![Modern black kitchen appliance with a circular silver knob blurred in motion across its surface.](https://cdn.sanity.io/images/ult5g8gw/production/0075f0f6f20c9b66a2da14a1d5c3e6023046fd6d-1408x768.jpg) ## TL;DR Plan AI infrastructure for rising memory demand as cheaper tokens unlock larger contexts, more agents, and heavier workloads. - Anticipate agentic and multimodal capacity needs. - Reduce repeated token generation with persistent caching. - Balance compute, memory, storage, and networking. Every time AI gets more efficient, the demand for AI infrastructure grows. This is [Jevons paradox](https://www.npr.org/sections/planet-money/2025/02/04/g-s1-46018/ai-deepseek-economics-jevons-paradox) at work, and it’s quickly emerging as a significant dynamic shaping AI infrastructure investment. WEKA’s [Val Bercovici](https://www.linkedin.com/in/valentinbercovici/) speaks about this often: The faster AI gets cheaper and easier to use, the faster memory capacity becomes the binding constraint. And this gap is widening, not closing. ## Why efficiency gains are driving memory demand, not relieving it The instinct in infrastructure planning is to assume that as AI models become more efficient (e.g., less costly to run, quicker to respond) the pressure on underlying systems would ease. But the opposite is happening. “Every time we lower the unit cost, the demand overwhelms and spikes that reduction,” Val noted in a recent interview with [SDxCentral](https://www.linkedin.com/company/sdncentral/). Val pointed to Jevons paradox, a counterintuitive economic principle stating that as a resource becomes more efficient and affordable to use, total consumption of this resource rises rather than falls. It’s a 160-year-old concept that has impacted every industry; it’s now emerging as a major AI challenge that is getting noticed all the way up to the C-suite (read for yourself in this [LinkedIn article](https://lnkd.in/p/gG7VBuKx) from [Box](https://www.linkedin.com/company/box/) CEO [Aaron Levie](https://www.linkedin.com/in/boxaaron/)). “We’re seeing a nice set of pricing segments and pricing tiers where there’s premium tokens, there’s sort of the ‘90% as good tokens at 10% of the price’ open weights tier,” Val continued. “And there’s local tiers where you don’t even really pay per token anymore if the model is small enough to run on your laptop or your phone.” 👉 Be sure to watch Val’s [full interview with SDxCentral](https://www.sdxcentral.com/video-interviews/sdxstudio-the-ai-memory-wall-and-emerging-bottlenecks-with-val-bercovici-weka/). Cheaper tokens don’t reduce memory pressure. They unlock new use cases, longer context windows, more agents, and more users… all of which demand more capacity and make scaling the [memory wall](/learn/ai-ml/ai-memory-wall/) an elusive challenge. ## How context windows and agent swarms exacerbate the problem When ChatGPT launched in November 2022, context windows were small, with dozens or maybe hundreds of tokens at most. Today, million-token context windows are common, and this is before accounting for agentic AI. “Agents are chat sessions on steroids,” Val said. “They have these things called ‘turns’ where they go back and forth, and it’s as if you and I were doing 1,000 back-to-back prompts and responses. That’s what agents and agent swarms do.” Add multimodal workloads such as video data that runs, say, 10x heavier than text, and the memory problem stops growing linearly. Instead, it expands by orders of magnitude. Every efficiency unlock at the model layer translates directly into a new ceiling for memory infrastructure. The actionable takeaway for any team building or procuring AI infrastructure is to plan ahead for agentic and multimodal workloads, not just for current LLM inference needs. ## How cache economics reveals the true cost of memory inefficiency Token-based pricing has become one of the clearest windows into where memory inefficiency actually lives in AI systems and where the Jevons paradox dynamic plays out in concrete costs. “Pick your model, pick your provider,” Val said. “To use round numbers, it’s about $1 per million input tokens, and often something like $5 or $10 per million output tokens, but the new variable is _cached_ input tokens. And the price per million cached input tokens is often 10 cents. The new possibility is to make your cost only the cached input cost, only the 10% cost, if you break this capacity barrier for the memory wall.” Persistent, well-architected memory and caching infrastructure can therefore reduce token costs by a factor of 10. But you don’t achieve this by negotiating better rates; rather, you need to engineer systems that stop regenerating the same tokens repeatedly through prefill, which is one of the most expensive inefficiencies in AI infrastructure today. As these costs drop, Jevons paradox kicks in again: Lower cost per token means more tokens consumed, more agents deployed, and more sessions running in parallel. And if enterprise power users continue to generate tokens at increasingly explosive rates ([OpenAI](https://www.linkedin.com/company/openai/) CEO Sam Altman [recently stated](https://finance.yahoo.com/sectors/technology/articles/sam-altman-dishes-openais-top-191022095.html) their top user is racking up 100 billion tokens per _month)_, it’s a signal of where enterprise AI intensity is heading and why memory capacity is becoming a board-level conversation. ## Why balanced systems will define who leads in AI infrastructure Token routing is becoming a core infrastructure competency. Knowing which queries require frontier models — and which can be handled by open-weight or locally deployed alternatives — isn’t simply a cost-optimization exercise. It’s how organizations manage the compounding demand that efficiency gains keep generating. “Balancing the processing power of the GPUs and the LPUs, balancing the memory and the networking requirements — that’s going to be something that is an art, a black art today, and ultimately I think it’s going to become a science and become more public and democratized,” Val said. The organizations that will lead in AI aren’t necessarily those with the most compute. They will understand the Jevons dynamic well enough to architect ahead of it, building storage, memory, networking, and processing as an integrated system designed for the workloads that increased efficiency will keep unlocking. 👉 Dive deeper into how you can get the most from [AI inference](/learn/ai-ml/how-ai-inference-works/) from the infrastructure you have already with [this guide](/learn/ai-ml/prefill-and-decode/) that explains the prefill and decode phases.