Why Memory, Not Compute, Is Breaking the AI Cost Curve

It’s amazing to be here at the WEKA stage again. Thank you for joining me. So much has changed over the past year. It’s fun to see Snowflake and Alibaba Cloud and all of our other vendors here talk about things that did not exist a year ago.

We’re going to peel some of the layers away from what actually powers these agents and how they directly impact your user experience, my user experience, your cost, my cost, and ultimately what we can all do with agents.

Last year we didn’t really talk about agents that much. They were aspirational. We had Manus on stage here last year. Claude Code was just appearing in May of 2025. Who was using Claude Code a year ago? My early adopters here — fantastic, congratulations. Who was using Manus a year ago? A couple of people as well, myself included.

So again, so much has changed. These were really mysterious technologies a year ago. They’re absolutely mainstream production-ready technologies today.

So what’s changed? The focus, certainly the dollars, the attention, has shifted from training to inference this year.

I want to bring up this really fun, short-lived phenomenon. Who is familiar with the term “tokenmaxxing”? It kind of came up a little bit in the holidays and Christmas last year, a lot in the first couple of months this year. I think it peaked at the NVIDIA GTC conference in San Jose in March, but it’s kind of gone now.

We were talking about annualized spend of up to $5 million per user, $15,000 a day of token costs at the high end. Our friends from SemiAnalysis, who I’ll be speaking with tomorrow on stage as well, actually have a real use case because they’re able to sell reports and models for more than $5 million that they generate with these tokens. Many of us don’t generate products that can sell for $5 million, so tokenmaxxing doesn’t really make sense for a lot of us.

Victor Taelin from High Order Company is one of the most respected AI open source developers out there. He had a more credible estimate in terms of token costs — last year, about $743,000 to develop his product. Still untenable, unsustainable.

As they say, the chickens have come home to roost. As of last week, the CFOs got involved, the CIOs got involved, and they said tokenmaxxing is ridiculous. We can’t spend more on tokens than on the humans that are using them. We have to figure out how to be adults in the room.

This is very familiar for those of us who were part of the cloud disruption 15 years ago. Everybody, especially developers, were very excited about cloud when we could develop apps and deploy them very, very quickly — don’t have to wait for IT, don’t have to provision servers, don’t have to provision firewalls. But by month two, month three, the big AWS bill came. And if you remember phone bills from 15 years ago — multiple pages, thousands and thousands of line items you don’t understand, and a bottom line nobody likes — that’s exactly where we are with tokenmaxxing and token spending today.

Where do we go from here? We can’t afford the tokens we want. How do we break through this bottleneck? It is a memory bottleneck.

One of the best practices is actually routing models. There’s a lot of routing going on in production systems today, in the agents we just heard about. The first thing we do is run a small SLM, a small language model or a small neural network, that assesses and classifies the nature of the request. Is this particular prompt, this particular attachment, this particular RAG (retrieval-augmented generation) workload very heavyweight, requiring a lot of reasoning. Or is it relatively straightforward?

If it’s heavyweight, of course we send it to the premium models, a premium token price from a commercial frontier lab. Most of the time it’s not. Most of the time it’s something you just need to execute, or it can be a low-cost, high-volume, bulk requirement that an Anthropic Haiku-class model can run. The middle of the road can be an Anthropic Sonnet-class model (we’re using some GPT naming conventions here).

Model routing is one of the most important concepts. And after you route to the right model, called sparse models, you do mixture-of-expert token routing inside the model. So it’s almost like a Russian doll: There are layers of routing going on, and the token costs are very important to understand.

There was a lot of confusion last year that’s hopefully clarified this year. We see these proven numbers over and over again. NVIDIA CEO Jensen Huang himself mentioned at GTC a few months ago that for every 100x reduction — 100x lower unit cost of token inference — we see a 10,000x increase, a 1 million percent increase, in token demand.

So the net reality, and why our token costs are out of control right now, is 10,000% growth in actual number of tokens required and actual token costs. This is what we have to address, and the vendors aren’t helping.

There are a couple of really cool frameworks that were pioneered last year that are mainstream this year. If you’re familiar with Steve Yegge and Beads, or Gas Town — a friend of mine, Reuven Cohen, has this really cool framework that used to be called Claude Flow to orchestrate Claude Code. Now it’s called Ruflo. Anthropic’s been paying attention, and over the last couple of weeks they released this very cool new workflows feature that automatically creates subtasks, automatically creates many parallel sessions, so you can complete an ambitious goal.

If you’re not familiar, one of the hottest commands of any coding agent right now is a “slash-goal” command that will run for hours and days. Think about that: Consuming tokens 24/7 for five, six, seven days straight, thousands of hours, millions of dollars of potential token cost, to generate fully functional artifacts. This could be a fully functional website with vulnerability scanning, enterprise authentication, auditing and logging, multiple backends, data consistency, backups. All generated by one command and, hopefully, a very detailed specification of what you want, how to test it, how it should work, and what it should cost.

But what we’re seeing right now — and this is a cheeky play on words — is there’s no such thing as “agent” in production. There's no singular for “agent.” It’s only agent swarms. This is the production reality we have to deal with right now.

So how do we measure this? We can’t manage what we can’t measure. Something I’m incredibly proud of: On the WEKA team, one of the genius product managers I work with — who isn’t here this year, but is actually moving from the West Coast in the United States to this part of the world, so you’ll be seeing more of him, his name is Callan Fox — created a very cool open source repository called KV Cache Tester.

This repo brought observability to KV cache consumption for the first time in the industry, and a way to actually test it in real-world conditions with real-world traces, letting you replay those traces or create synthetic traces.

This is now in use by SemiAnalysis for the next version of the InferenceX benchmark, which we’ve codenamed together AgentX. Every benchmark, including the great SemiAnalysis InferenceX benchmark, has to date only benchmarked small chat sessions, not long-context, multi-turn, long-horizon, highly concurrent agent swarms, which is the reality of the industry today.

NVIDIA has forked this repo and it’s part of AIPerf in the Dynamo framework. AMD, I found out this morning, has just forked this repo too and is now using it for real-world benchmarking. We proudly just published a benchmark, which I’ll talk about a bit later, with Oracle — it was actually Oracle Cloud that published it. Again: real-world model, real-world workloads.

Every color change you’d see in one of these traces is an opportunity for a token to be cached, and we’ll talk about why that’s really important, because that’s how you slash your token costs. That’s how DeepSeek, with version 4, is now shaking the foundations of the AI industry again after 484 days, by being able to slash the cost of tokens, particularly the cache-read costs. Every color change in a real-world agent swarm involves a lot of potential caching, and real-world caching is very, very important.

So the bottleneck we hear a lot, when we talk about OpenAI going public, when we talk about Anthropic going public, when we talk about Cerebras, the IPO that just happened recently — very popular — there's always this talk about there’s not enough compute to actually run AI today. That's a very simplistic way to describe it, because the reality is there's actually abundant compute. It’s not cheap, it’s not always available, but compute is not the bottleneck. Memory is the bottleneck.

Let’s use a very real-world, simple explanation. This is on a Hopper-class system, because the math is absorbable — it’s only about a terabyte per server here. When you load the model weights — this is MiniMax, I believe, M2.5 — just the model weights, before you do anything, is almost 40% of a GPU server. Then you put in the working memory — your sessions, my sessions, swarm sessions — you load that into memory, and that’s another 60-plus percent. Very quickly you’re at 90% utilization of the memory before you’ve actually started the second turn or spawned the second agent subtask. We’re almost out of memory before we even begin. This is the GPU memory, the physical memory. This situation is called the memory wall.

We’re at a point right now where we’re completely imbalanced from a GPU perspective. We have more compute power being generated than memory to accompany the incremental compute power. As we went from Ampere to Hopper, Hopper to Blackwell, Vera Rubin is very much in the news right now — every time we go up a generation, we’re always very happy to see the huge increase in FLOPS (floating-point operations per second). What no one really wants to admit on the same stage is we’re nowhere near increasing the memory capacity as much as we’re increasing those FLOPS.

So we’ve reached an inflection point where the model parameters, the model weights, require so much more memory than we had planned when we actually designed the silicon for these GPUs, that we’ve reached a breaking point. We have to do something different. This is why we have the global memory shortage. If you tried to build a gaming PC for yourself, for your friends, for your kids over the holidays, you were very frustrated that everything was more expensive than you budgeted for — the GPU, of course, but the memory, the storage, the hard drive, everything was more expensive.

There’s this really unfortunate divergence happening right now that’s creating enormous pressure on the industry, because that backlog of GPUs, memory, and storage keeps increasing the prices literally every week. Most vendors, if they quote you a price for a GPU, storage, or memory, will only honor that price for a week at most, maybe two, before their own supply chain increases the price on them and they have to pass that on to you. So we’re under extreme pressure from a supply chain perspective, and at the same time, token demand is absolutely surging because of agents.

Here’s another pressure point on the industry, and this isn’t about our productivity or these cool dashboards and apps we’re generating as agents. This is a harsh reality of the internet. We’re all familiar with Mythos. Who’s heard of this thing called Fable? RIP Mythos. It’s like a mythical model that most humans will never touch. It’s going to remain invite-only, very secretive, very expensive to run, very restricted access for cybersecurity professionals.

But it’s a new era. It’s a loss of the age of innocence, because we can now use these models — hackers, nation-states, ransomware gangs, organized crime, extortionists — now have access to incredibly powerful technology to crack almost any software at will. We’ve seen the number of security incidents spike in the last six weeks or so since Mythos was released.

The real danger here, though, is something the president of Microsoft wrote a book about during the COVID era. Brad Smith wrote a famous book saying every new technology is both a tool and a weapon.

DeepSeek — I love the company, I love their technology, amazing tool — is also an incredible weapon, because the cost of tokens for DeepSeek is 100x lower, cache-read tokens in particular, than any other model, and, if hosted out of China, mathematically precisely 87x lower than hosting DeepSeek anywhere outside of China. I won’t speculate as to why it’s so cheap, but these very intelligent, very low-cost tokens are available to malicious agent swarms right now. I call them red agent swarms.

We have to ask ourselves: Can we afford 24/7 cybersecurity blue agent swarms? Our own cybersecurity protection agents, personally but especially organizationally, at our companies, our governments, our nonprofits? Can we afford to run what’s needed now in a modern security operations center? When you talk to the professionals inside Palo Alto Networks, CrowdStrike, Cisco, inside every major cybersecurity vendor, they’re literally planning right now for running agents continuously — not just one goal or one task, but 24/7 swarms of agents, with those old token-maxing metrics. Can we afford that? Probably not.

If anyone ever asks you whether AI is a bubble, simply point them to where real money is being spent. The most transparent view of token traffic in the world is a wonderful service called OpenRouter, which is also about a two- or three-year-old startup. They just got a bunch of funding recently, not surprising, when you look at the growth.

Where we were last year is literally off the charts at the beginning, over on the left-hand side. Manus and Claude Code were roughly May of 2025. We’re only showing one year’s worth of token consumption here. We see consistent growth week over week, month over month — and then December happens.

What happened in December? Two things. One you all probably remember: Peter Steinberger released OpenClaw, and there was an explosion of token consumption worldwide. Who remembers the promotional pricing and token rate-limit increases Anthropic also had during December of last year? Probably the single most brilliant marketing move in the history of technology, because every technical leader, every hacker, every developer had extra time over the holidays, was getting higher rate limits, lower pricing, and was able to experiment for the first time with Claude Code (I believe it was Opus 4.5 at the time) and the context windows were just large enough that real work could be done, real apps were generated relatively bug-free, relatively well-designed.

Everyone proved to themselves this technology is real. It was the “holy S” moment — basically, “oh my god, this is real.” And look at what happened since then. Whether it’s commercially, whether it’s OpenClaw or Hermes Agent, we’ve seen this amazing spike of token traffic. I gave a version of this talk with this chart just three months ago in New York, and that circle was at 22 trillion tokens. In only three months, we’re already at 45 trillion tokens. If you do some back-of-the-napkin math, we’re looking at one quadrillion tokens per day being consumed. And this isn’t a proxy for the entire market, this is only about 1% of the market, and it’s paid tokens.

So it’s very statistically significant, with this volume of tokens, this breadth of models, this breadth of apps and agents. This is proof that the industry is real, and we can’t even satisfy this demand.

At the beginning of the year, Jensen shared the context memory revolution with us. He said, and he was true to his word, six months ago that we’re out of memory, that we need to extend the working memory of AI into this new concept called “context memory.” It’s a bit of an oxymoron, context memory storage. It’s the fusion of both memory and storage.

Why do we have to do this? Imagine you’re Amazon, the retailer, not the cloud service, you work with a lot of merchants on your marketplace that produce stuff from factories. Imagine if there was no warehouse, no Amazon warehouse. The factories have to produce, and to reach any economy of scale, thousands of units come out every week. If you can’t sell those and you can’t warehouse them, you literally have to throw product into the garbage. Wasted product, wasted energy, wasted material, wasted labor, wasted cycles.

This is exactly how AI inference works today. When we run out of that precious memory, and it happens almost instantly, we have to pay a recompute tax. We have to rebuild the product at the factory — something called GPU prefill — and then try to ship it again to the output tokens, the end consumer.

At WEKA, we decided we’re going to innovate and enable this concept of token warehousing, where yes, you build a product in the factory, but you only prefill once. You don’t keep rebuilding and paying the recompute tax over and over again. What that means is you can actually store the working memory, called KV cache (key-value cache). You can store these key-value matrices at memory speeds. This isn’t an archive process, it’s not really a storage process — it’s placing the in-memory structures, at memory speeds, in a warehouse that’s again accessible at memory speeds whenever you want to decode those tokens. So there’s no recompute tax, no redundant prefills. It’s remarkable in terms of lowering the cost of tokens, lowering the latency, increasing the throughput. That’s a fusion of storage and memory.

The way this is done technically is by taking the four common tiers and hierarchies of memory and blurring the lines a little bit. Jensen held his promise: He said there’s such a shortage of memory that last week SemiAnalysis reported, and it got a lot of attention, that the new Vera Rubins are going to have half the DRAM. Instead of having roughly 4 TB in that G2 tier of CPU DRAM, the DDR5 memory, they’re going to have 2 TB.

Lowering the actual amount of memory in the bill of materials is a very significant event, because when you blur the lines correctly, when you can actually move memory tiers at memory speeds, you need less of that G2 tier. You’re still going to need that G3 tier, because you want the capacity — petabytes and petabytes of memory, not storage, accessible as context memory. That’s what the G3 and G4 tiers offer.

This is what WEKA specializes in, with our Augmented Memory Grid. Dan Nishball from SemiAnalysis and I spent a lot of time on this last year. This is the updated version of last year’s chart, the infamous Pareto curve. It’s very simple: the y-axis is how many tokens you can process (how many input tokens per GPU per watt) and the x-axis is how fast those tokens come out (the output token speed).

In the past you had these very balanced hardware systems, accelerators like Google’s TPU (tensor processing unit), an architecture called a systolic array that gives you a fixed balance of compute and memory, and you recompile your models. If you’re Gemini, inside Google, that’s easy. But even Anthropic has gone through the pain of recompiling their models, which is very complicated. In order to take advantage of these accelerators — I’m not going to label who is what here — you can have some GPUs that give you more bulk capacity of token processing, and other GPUs that accelerate your output tokens per second. That was essentially a very 2025 view, on the left side of this Pareto curve.

What was announced at GTC, what the Cerebras IPO was all about a few weeks ago, is that on the y-axis you can dramatically accelerate the speed of output tokens per second. The reality is it’s for a limited number of users, limited model size, limited context size, but that portion of the agent swarm can be very valuable, like real-time voice agents.

What WEKA specializes in is context memory technology that dramatically increases the bulk capacity, by 10x or more, of input tokens processed. This is a win-win scenario: You get a reduction of up to 90% of your token latency and your token costs. If you’re a provider, you can serve 10x the number of users and sessions, and on the same capex and opex, have 10x the number of tokens.

This is no longer WEKA claiming this — we’ve stopped just boasting about our own technology. These are our partners, scientifically reproducing these results independently, and serving on them. On the Oracle marketplace — again, real-world model, not a special year-old or two-year-old benchmarking model — MiniMax M2.5, a real-world model, Oracle independently benchmarked this technology available on their marketplace and found they could deliver 10 times more MiniMax user sessions, 10 times more MiniMax tokens, without increasing their capex or opex — no extra GPU hardware, no extra storage, no extra memory, no extra energy or water — to generate this amazing result.

And if you’re someone sitting here in Singapore saying, “I don't trust that northern hemisphere company, I want something closer to home in the southern hemisphere,” we partnered with Firmus, who used the Qwen model from Alibaba, one of the bigger ones (Qwen3, 480 billion parameters) for a coding agent with a real-world workload and got similar results: 6.5x the number of tokens.

So very, very credible. And what this really means — I’m glad you’re sitting down, this is probably the most profound thing we’re going to say here — is that out of thin air, with nothing but software, you create five to 10 more entire virtual data centers — a full AI factory, a token factory — by installing this software. You get that additional memory, not paying the recompute tax, and just being able to decode more tokens, lower latency, again with the same energy, the same water usage.

So it’s something we’re very proud of. This is how we scale within all these constraints of the industry — the memory shortage, the GPU shortage, the energy limits we have, the very finite caps on energy, the focus on reducing water usage, the focus on removing rate limits, the focus on lowering latency for agent swarms. The main lever, the main thing you can do to improve your situation, is improve that memory utilization.

With that, I’ve really enjoyed coming here a year after the beginning of this trend last year. I can't wait to see the wild things we’re doing as an industry in 2027. It’s going to be a lot of fun. Thank you very much for your time, everyone.