Inference Is Eating Memory, and Tokens Now Run AI Economics

Catching up: The Mythos-Fable moment arrives, and how model routing defends against the memory crunch

Vik Sekar: Welcome to another Semi Doped podcast. I’m Vik, from Vik’s Newsletter, and with me is Val Bercovici from WEKA, which is an AI data and memory infrastructure company. Val has been on Semi Doped before, and a few months ago we spoke about context memory storage (CMX) right after NVIDIA announced their CMX platform. You can check out that whole conversation. Today is more about recent developments in the world of memory and storage and where things are going in AI inference. Val, thanks for being on the podcast.

Val Bercovici: So much fun to be back, Vik. As we predicted, the only constant is change, so there’s lots of updates to talk about since I was last on.

Vik: Isn’t that exciting? Things are constantly changing. Since we spoke a few months ago, it seems like 10 years have gone by in AI Land.

Val: Exactly. I don’t know if I made this prediction last time, but I expected a Mythos-Fable moment at the end of the year, and it happened. Things are definitely happening faster than we thought.

Vik: That’s amazing. It’s really picking up now. Memory has reached an all-time high and become extremely expensive. A lot of people deploying AI hardware are finding ways to make the best of what they’ve got: increasing utilization rates, pooling resources, offloading to different tiers, whether that’s other kinds of RAM beyond HBM (high-bandwidth memory), or NAND flash. So many techniques have sprouted up in a matter of mere months because we have to do something about the memory situation.

Val: Exactly. Maybe not the simplest, but one of the most obvious and coarse-grained techniques, in my mind, has been model routing. If you’re not always using the most memory-hungry model for every prompt — especially in an agent swarm with hundreds or thousands of turns — and you’re intelligently routing requests to medium and small models, you implicitly get lower memory requirements. That’s an easy button for reducing memory in aggregate. But then we get much more specific with each model’s memory requirements: working memory, KV cache (key-value cache), and so forth.

Vik: Exactly, and that’s what we were talking about last time. The KV cache started getting so big when agents arrived on the scene that you couldn’t keep it all on expensive HBM. It’s a limited resource that has to be preserved and used judiciously. So there was a lot of talk about offloading to DRAM and then, as agents started working more and having greater context length, offloading to context memory storage — the rack NVIDIA calls STX.

What NVIDIA’s STX and CMX mean and how WEKA’s Augmented Memory Grid fits in

Val: The new storage platform is STX. Its biggest use case is going to be context memory storage, context memory extensions in NVIDIA terminology, within the STX hardware and software framework.

Vik: WEKA has hardware for this too, which we spoke about last time: WEKA’s Augmented Memory Grid, which provides storage directly to the GPU via a very fast network connection. The speed you get out of it is comparable to something faster than NAND but slower than DRAM, right? That’s where we left off.

Val: Let’s double-click on that, because it’s actually different. The standard positioning is that you can be faster than NAND with higher-bandwidth connections — faster than local storage, effectively — but in a standard hierarchy you could slot in as slower than DRAM. However, depending on your architecture, and whether you can take advantage of full line-rate network performance or something like NVIDIA NVLink the way WEKA does, you can actually be faster than DRAM.

If you look at the number of PCI lanes in these scale-up domains with NVLink on NVIDIA servers in particular, there are far more PCI lanes — around 128, I believe — than there are PCI lanes from the CPU to DRAM, which is only 32 on the motherboard. This is one of the cool architectural leverage points of WEKA. We recognized at company founding, about 13 or 14 years ago, that for high-performance computing — which is what AI factories are — there is more bandwidth on the network than on the motherboard. PCI is a bottleneck, not an accelerant. This is still something a lot of systems architects and developers struggle with. They assume nothing is faster than what they can access on the motherboard, and they design entire database hierarchies and application latency profiles based on that invalid assumption today, because the network is faster than a motherboard in a GPU network and a high-performance network.

So the memory tiering can involve whatever you connect on the other side of that high-bandwidth network — the NVLink-style network, the east-west compute memory network, whatever you want to call it — and whatever you connect on the other side of that network can be faster than DRAM. The way WEKA connects (optionally, we don’t require that network, but we leverage it maximally), we’re faster than DRAM on a raw bandwidth basis.

Another confusing part is how CPU and GPU memory transfers. KV transfers for example, operate on most GPU servers depending on the generation. Hopper has something called a “bounce buffer” between the GPU and CPU. Blackwell introduced, especially with Grace Blackwell, more sophisticated chip-to-chip interconnects that are faster between GPU and CPU transfers. Vera Rubin continues to advance that with higher-bandwidth and far more scalable memory transfers between GPUs and CPUs. But you can still be faster than DRAM on these servers because of NVLink. There’s a ratio between NVLink bandwidth relative to memory bandwidth, relative to front-end network bandwidth, and as long as you can be line-rate on NVLink, you can be faster than DRAM. It’s counterintuitive, but the math proves out.

Vik: That’s great. Just to recap: The networking on modern hardware is so fast that when you hook up storage using that network, you can get bandwidths faster than DRAM, because motherboards aren’t the fastest — and that’s an assumption a lot of people are making right now. Instead, if you view NVLink as a very high-bandwidth interconnect, you can bring storage to operate with much higher capacity, but at speeds better than DRAM.

Val: Yes. And these networks are really critical right now. We’ve seen a rise in interest in networks. What Mellanox has done with NVLink has been wonderful, and it’s not just NVLink — Mellanox offers CX7, CX8, CX9 network adapters, BlueField 3s and 4s, with multiple varieties of BlueField 4. We’re not picky at WEKA about where the bandwidth comes from. As long as systems architects configure the bandwidth they want for the memory tiering they want, WEKA can support it and continue to deliver radical economic benefits — more tokens per second, more concurrent users per second — which really hits the market sweet spot right now of agent swarms.

Why WEKA now calls itself a data and memory infrastructure company

Vik: Since we last spoke, WEKA shouldn’t be viewed so much as a storage company anymore. In the AI era, given the solutions WEKA provides, it’s far more than that. We touched on this before today’s show. Do you want to double-click on why WEKA should be seen more as a memory company and a company that does AI data infrastructure instead?

Val: Yes, very much. Two of our industry partners, so to speak — one is Groq, which NVIDIA acqui-hired last December for $20 billion, and then of course the popular and successful Cerebras IPO — were clear indicators that the inference infrastructure market is very distinct from the training market that NVIDIA dominates with GPUs.

Even Google is the clearest example for me. Google has had seven generations of TPUs (tensor processing units) applied to both training and inference — radically different markets — through TPU 1, 2, 3, and so on through 7. With TPU 8, announced a few months ago, they were explicit that there’s now a TPU 8 for training but a completely different TPU 8i for inference. This is another example that inference is different, inference infrastructure is different. It’s great when you can leverage the same infrastructure for training and inference — there’s nothing wrong with that — but understand that whether it’s companies like Etched, Cerebras, or other LPU-based, ASIC-based, SRAM-based companies, inference will continue to diversify in terms of infrastructure and diversify away from training.

WEKA is very much following that trend. WEKA has a great legacy of high-performance computing storage, but the new tagline for the company — data and memory infrastructure — reflects the fact that, at a first-principles basis, inference is not storage-centric. Inference is a little compute-centric, as we know, for prefill, and extremely memory-bound for decode and feed-forward. That’s really where WEKA Augmented Memory Grid fits in: You can use it as an inference-only product without any WEKA storage whatsoever. It just happens to be backed by NAND at the bandwidth of memory, and as we just discussed, when that context memory network is large, it’s actually more bandwidth than DRAM.

Inside the four-tier memory hierarchy and why real AI workloads break it

Vik: So depending on the kind of hardware running now — you mentioned Cerebras is one — the whole inference landscape really seems like there’s no single right way to do inference. You could run it all on SRAM, you could run it with LPUs, which is also SRAM or with a combination of HBM and SRAM. SambaNova uses all three. It’s really breaking up all over the place. How do these compare in terms of performance, and where does storage play a role in all of this?

Val: There’s a memory hierarchy, and it’s interesting. When we last talked about it, we discussed it as if it had been around forever, but it’s really only about a year old. It’s a fairly well-established four-tier hierarchy that’s going to have to change now. NVIDIA’s Dynamo team has done a really good job documenting this. They label the top tier, G1, for high-bandwidth memory. G2 is for CPU DRAM, also called LPDDR and SOCAM. For the uninitiated, it’s all the same thing. G3 is local storage, sometimes called the rack SSD in the servers. G4 is remote storage, often NFS or S3-type storage layers.

One thing not to confuse about this memory hierarchy: There’s no smooth graduation from HBM to DRAM to storage, local or remote. These are very rough, jagged transitions — almost a Grand Canyon between them — because we’re talking about orders-of-magnitude higher latency and different bandwidth between those tiers. You very quickly reach cliffs where if you’re OK providing one or 10 tokens per second to a user, with latency and time-to-first-token of 10 seconds and end-to-end latency of hours where it should be minutes, then you don’t have a problem. But in the real world, no one tolerates five or 10 tokens per second output — the human eye and attention span demand about 35 to 50 tokens per second of output minimum, and agents, at machine speed, will take thousands of tokens per second in a multi-turn agent swarm.

So SLOs (service level objectives) matter. That’s where we quickly find that even though a lot of vendors talk about memory tiering, including NAND and storage, in the KV cache hierarchy, it’s kind of irrelevant. Most of the benchmarks we see today that include storage in the KV cache tiering hierarchy reflect workloads you could just run on a DGX Spark or a Mac Studio. You don’t even need a big server for some of these smaller models with small context windows and single-turn chat sessions, or two- to three-turn agent sessions that aren’t representative of reality.

When you’re running real-world workloads today — and SemiAnalysis is actually working on updates to InferenceX that they’ve nicknamed AgentX after my suggestion, actually, so I won’t steal their thunder — but when it comes out soon it’s going to reflect something called input sequence length, or ISL. The context window is much larger than the 8,000, which was the limit in the past — now its hundreds of thousands. They’ve published an analysis showing the median traces for coding agents now are 200,000 to 300,000 tokens, because we’re in the era, since we last spoke, of million-token context windows instead of the 100,000 or 200,000 limits of before.

So when you reflect the reality of large models — trillion-parameter models or models in that class — Minimax is in that class even though it’s not quite a trillion parameters. Kimi GLM 5.2 is the new hotness with large multi-turn sessions and high concurrency the way Claude Code, OpenCode, Codex, Hermes, OpenClaw, actual agents work, and you end up with a radically different workload profile. On one hand, models have gotten so much better in terms of KV cache consumption. I remember, for every 100,000 tokens without optimized compression — for example, sliding window attention and other techniques from DeepSeek V4 — you’re up to 50 GB of KV cache usage for that 100,000 tokens.

Jevons paradox in KV cache: Why compression gains keep getting erased

Val: That number has been reduced, on a unit basis, with TurboQuant and DeepSeek’s other optimizations, to about 5 GB instead of 50 GB. That 90% reduction is great, but what’s happened on the other side? Context windows 10x’d. Agent swarms 10x’d, or 100x’d, in number of concurrent sessions, and on and on — we’ve gone multimodal. The net effect we keep seeing is that for pretty much every 100x reduction in unit KV cache size, there’s a 10,000x increase in consumption of KV cache. So we’re always looking at about 100x more token volume, and it’s reflected in the pricing pushback we’re seeing from enterprises today. Maybe they’re not paying 100 times more than expected, but they’re paying a lot more than expected because of that token volume.

Vik: Right. This is great, because KV cache optimizations are clearly on the way. TurboQuant, for one, scared the daylights out of the market; everyone thought memory was dead, HBM was dead, everything was dead. But it turns out quantization is a good thing, because we only need about one-tenth, 90% lower, KV cache storage for 100,000 tokens. So 100,000 tokens went from 50 GB to 5 GB. That’s great, because with existing models we can run far more agents and do a lot more work. So the demand never went away; it actually increased.

Val: Right. You could almost make a meme out of this. That 50 GB that went down to 5 GB went right back up to 50 GB because the context window size went from 100,000 to 1 million. There’s a give-and-take here, and the net result is always more memory demand.

Vik: So because KV cache is being compressed like this, do you see that continuing? Will we get to maybe 1 GB per 100,000 tokens in the future? What does this mean for KV cache? Since usage dropped through quantization and came back up because of agents, I guess there’s no net increase. Is this something the memory and storage industry should worry about? I don’t know if the increase in usage is exceeding the benefit coming from quantization.

Val: The cliché “this time it’s different” applies here too. Every time there’s a massive new innovation in KV cache compression, it’s not different. It’s Jevons paradox kicking in over and over again: We see a higher increase in demand every time there’s a reduction in unit consumption. Context windows aren’t stopping at one million and it won’t be long before we’re at 2, 5, and 10 million, maybe by the end of this year. Everybody wants more context, and we’re going longer-horizon in our agents. We’re giving agents far more ambitious goals, encouraged by the agent and model providers. The goals now span hours and days, and soon some agents will run for weeks quite regularly. For cybersecurity, they run forever, right. You’ve got to run the security operations center, persistent blue-agent swarms forever now.

So we’re seeing longer multi-turn horizons, more parallelism, more concurrent subtasks. Claude Workflows was one feature introduction that became popular for solving a problem in parallel 10x faster. And we’re going multimodal, not just language. We’re inserting more video, audio, and image frames into these agents. It all results in a continuous explosion of overall token demand and overall KV cache consumption.

Vik: OK. So now that we’ve established this is only going to grow from here, NAND and DRAM, pooled or not, we’re going to need more of it, because we’re only going to run more of these things. That doesn’t spell decline or leveling off in the near term, because context windows will get longer and agents will run longer, and more agents will run per task since you can run them in parallel.

DeepSeek’s cache-read pricing shock and the China pricing gap

Vik: One interesting thing you mentioned earlier was DeepSeek’s optimizations. When DeepSeek V4 came out, there was a lot of emphasis on an SSD-first approach to inferencing, which helped their token pricing, especially when they hit cache. They showed that for some workloads you could hit 95% cache hits, and as long as you keep hitting KV cache, you already have the tokens stored in a high-bandwidth network connection to an SSD, which means you can basically get it for free. Their token pricing for cache hits is so low, it’s insane. How is this playing out with other models? Are other models also hitting cache hits at this level?

Val: I’m glad you noticed that, because we really want to emphasize this key point now: The majority of token volume is agent swarms, by far — 80% to 90%. It’s no longer chat sessions; those are almost irrelevant to the conversation from an infrastructure perspective. It’s interesting. Even the Claude Sonnet 5 announcement the other day, everyone — even Anthropic — said, “Here’s the new pricing for input tokens and here’s the new pricing for output tokens, and it’s a promotional price.” The reason I don’t even quote the number is that it’s really irrelevant, because most of the tokens used by agents are cache-read tokens. With Anthropic’s particular pricing model, you have to pay for cache writes, too, to benefit from cache reads, but we should really only be talking about cache-read pricing. The other pricing is a rounding error.

DeepSeek’s pricing reflects this. They’re dramatically inserting two zeros after the decimal point for price-per-million cache-read tokens, which is a shock to the industry. Not so much the memory industry, but it’s a shock to other models and inference providers. The important asterisk is that pricing is only available out of China. You have to use DeepSeek hosted — the rumor is a Mongolian data center with very low energy costs — but also some secret sauce DeepSeek has in how they’ve integrated V4 Flash and Pro with the 3FS file system, parts of which are open source. They famously open-sourced it last year and wrote papers and blogs about it. Curiously, unless I’ve missed something, they haven’t updated those blogs with what are definitely new innovations and improvements that factor into the dramatically low pricing.

The net of it, and VentureBeat did some math on this and quoted me earlier, is about 87 times lower cache-read pricing from China than the same model, V4 Pro or V4 Flash, hosted in Singapore, the United States, or Europe. So the real conversation is, yes, DeepSeek has completely set the bar for global pricing for people who can inference out of China. But for people who can’t, it’s still by far the best pricing, and there’s a massive margin opportunity for open-weights inference providers to differentiate on cache-read pricing. You may not match DeepSeek’s subsidized pricing, but you can still leverage their innovations — sliding window attention, hybrid compressed attention, or HCA — and offer important reductions in token cache-read pricing, driving effective aggregate agent pricing down, because that’s the one pricing metric that dominates the cost of agents.

Vik: I have a couple of questions on this. First: It’s news to me that pricing comes only out of China. If I run a DeepSeek model out of, say, OpenRouter, I don’t get that pricing?

Val: OpenRouter is wonderful because it shows you the difference. There’s a little slider button you have to switch to say “show ignored” — I’m not sure why they label it that way — and when you click it, it exposes the DeepSeek pricing out of China, so you can compare. Your agent — your Hermes, your OpenClaw agent — can compare side-by-side in real time, the pricing for the exact same model: same input pricing, same output pricing, same cache-read pricing. I should double-check, because I stand on OpenRouter quite a bit daily, whether more cache-write pricing is beginning to appear for DeepSeek providers as well, since I know it has for some other models. You’re going to see providers differentiate not just on cache-read pricing, which is a major point of differentiation, but on cache-write pricing too.

What your agent dashboard doesn’t tell you about cache hit rates

Vik: In terms of cache-read for the real, agentic workloads running today, since typing in a chat window is a negligible portion of the inference market, let’s just talk about agentic workloads. How does somebody make sure they have a cache hit rate of, say, 95%, since that drives pricing down enormously and creates meaningful differentiation? Where is the industry right now in terms of cache hit rates?

Val: We could spend a whole episode just on this question, because it’s very opaque and a little confusing. Let me go through it, since it comes up a lot in other conversations, too. When you look at your agent dashboard — Claude Code, or just your Claude dashboard if you have both Code and Cowork and other Claude products, or OpenClaw, pick your dashboard — it’ll show a cache hit rate. That’s a logical cache hit rate, reflecting the cacheability of the tokens in your agent swarms, which is often very high — every agent pretty much has a cacheability of 95%, because you’re reusing a lot of context for agents, especially agent swarms.

However, the provider’s actual cache hit rate is not one-to-one with the cacheability of your tokens. The effective cache hit rate is very much a function of the memory tiers you have. Most people, by and large, in the first quarter of this year and some in the second quarter, have had very finite, fixed memory tiers. There’s only so much HBM packaged on your GPUs, only so much DRAM on the GPU servers, and while you can try to add storage tiers and KV-offload to storage, this ruins your SLOs. So it’s not that common in production for the popular models and token consumption.

The way to really understand where people have true effective cache hit rates close to the logical rates is through pricing. OpenRouter is again a great proxy for that, because it’s the only way to introduce transparency to real-world cache hit rates. OpenRouter publishes actual by-provider, by-model cache hit rates, and I like this because they change every few minutes based on actual token traffic. Even so, I don’t think those numbers are the real effective infrastructure, hardware, memory cache hit rates — there’s some blend — but nevertheless they’re much closer than what your own agent dashboard reflects.

I like to joke that we’re seeing a lot of “benchmaxxing” in a lot of models right now. For example, everyone trains for SWE-bench, so it’s no longer that relevant to benchmark, though DeepSWE is still a good benchmark. We see the same thing when vendors benchmark their KV-offloading solutions because there’s not a lot of truth left in benchmarking. My personal belief is that profit and loss, transparent pricing, is the ultimate benchmark, and you’re going to see WEKA be more explicit in that space. We want to continue proving out our own advantages, and we think real-world metrics reflected by pricing are probably the best way to benchmark a solution now, as opposed to something in a lab.

How DeepSeek’s sliding window attention actually compresses KV cache

Vik: You also mentioned DeepSeek’s sliding window attention is something worth looking into. What’s unique about it?

Val: If you’ve ever paid attention to how file compression in general has evolved from simple PKZIP-style compression or block-based deduplication toward similarity hashing algorithms — a combination of global similarity, global hashing, and local hashing — the same thing is happening now with KV cache. The same concepts are being introduced at the token level, at the attention level. Sliding window is exactly that: a way to look just at more recent tokens and compress them as much as possible, assuming tokens the particular attention head hasn't paid much attention to recently aren’t as relevant or as compressible.

It’s cumulative — I think there are formally about five different attention mechanisms in DeepSeek V4 now, all applied simultaneously, each specializing in short-term compression, long-term compression, context-relevant compression, and so on. The net result is that impressive 90% real-world reduction in KV cache usage per request. But again, requests are just coming faster and bigger, so it’s necessary to support that. I think the reason they continue to innovate so aggressively on KV cache consumption is that they want to support million-token context windows, and soon 2, 5, 10 million — and the only way to do that effectively is to keep being more efficient and intelligent about how you consume KV cache.

Vik: So the DeepSeek sliding window attention idea is that you keep the window of attention to what’s most relevant right now and keep discarding what isn’t relevant anymore, which saves KV cache.

Val: Almost assume that whatever was attended to before, and isn’t being attended to now, has already been compressed to some extent, so let’s be really aggressive on what we’re attending to right now, and look for opportunities to compress token memory, token attention, in KV cache.

NAND flash tiering for inference: SLC endurance vs. QLC capacity tradeoffs

Vik: On NAND flash storage, I recently wrote an article about what it really takes for an SSD to be AI-ready. SSDs in the past were built for a different use case than what they’re used for now. One theory I have that I want to run by you: NAND itself is now breaking up into several tiers. In the past, the industry always wanted more capacity out of NAND, so it tried to go from single-level cells, which have lower capacity but more endurance and faster write speeds, to QLC (quad-level cells) where you can store four bits, 16 different states, in one cell — which gives a lot more capacity, but endurance is a concern and it’s harder to write to, since you have to differentiate between 16 states. How do you see these NAND tiers working out? Are single-level cells making a comeback?

Val: Yes, it is making a comeback. Just as we discussed earlier, training infrastructure used capacity NAND because you wrote a lot of training data infrequently and read it very frequently, which is ideal for a QLC-type approach. Inference is fundamentally different: It’s memory-centric, so you’re really trying to make NAND appear more like memory and a lot less like storage. Memory doesn’t care whether you write or read at the same rate, because the assumption is there’s no endurance issue — there’s a power issue, a persistence issue, but no endurance issue with DRAM or HBM. That’s not true of NAND flash, so it struggles to act more like true memory from a performance and consumption perspective, as opposed to just a capacity perspective. This is something WEKA anticipated a long time ago, and you’re seeing it largely because of inference alone.

Intel had a really interesting technology with Micron called Optane, or 3D XPoint. It’s a real shame that it went away because today inference is the killer app for Optane, but it’s gone now, based on different materials, phase-change memory, and so forth. What’s trying to replace it is SLC (single-layer cell) flash. If you can’t optimize your writes, your KV cache writes, well enough, you have to buffer that endurance issue, that drive-wear issue, with a more endurant form of NAND flash — an SLC, tier — sometimes complemented by QLC, buffering a lot of writes in SLC and destaging them down to QLC later. Again, this adds complexity and latency, so there’s no free lunch there.

Another approach, which WEKA does, is to anticipate that you’re going to write to NAND flash a lot and read to it a lot, and use NAND flash as opposed to keeping it just for capacity. My joke is there’s no “S” in storage, in NVMe — it’s non-volatile memory express, or memory extensions. When you treat it like a true memory protocol and amortize the writes across a whole fabric of NVMe devices, you don’t need SLC. We don’t have SLC at WEKA, and never have — we can use TLC for a very high write-endurance workload like KV cache. In the future, as we’re able to influence the rest of the inference stack — the scheduling and routing, not just the KV transfer storage layer — we’ll add support for QLC drives as well. But you have to be careful, because you don’t want to break these drives with unintelligent scheduling of writes or unintelligent routing of tokens to somewhere that needs a quick write but isn’t really worth it.

Vik: So you can optimize the controller layer to make sure you don’t clobber the drive, whether it’s TLC or QLC, and intelligently write across the whole array while still preserving endurance.

Val: Exactly. This is a category of storage technology called shared-everything, as opposed to the Hadoop-style shared-nothing that you still see in parallel file systems like GPFS or Lustre. With a shared-everything approach, the client has global awareness, using a RAF protocol, of the queue status — the work-queue status of every NVMe device, every SSD controller. When every client has global visibility into the queue depth of every NVMe device in a fabric — tens or hundreds of thousands of queues across tens of thousands of drives — you can treat NVMe like a cache line for DRAM, and be very intelligent about where you shard the writes, where you load-balance them very intelligently.

At that point, you don’t need expensive SLC tiers to buffer writes. You’re not being suboptimal, you’re not treating devices as blunt storage devices: You’re digging into the devices, optimizing the underlying NAND flash, and cooperating with the controllers rather than offloading work to them.

Vik: So you don’t really see SLC coming in as its own tier, or maybe it’s just going to be used as a buffer?

Val: Let me disclose that I’m talking about how WEKA specifically has decided to optimize flash tiers. I think the industry needs SLC — the industry doesn’t have our patents or our implementation, except for the server partners we work with. Generally, that technology isn’t available outside of WEKA. So there will be a rise in the need for SLC to buffer writes when you can’t amortize them over TLC. NVIDIA is forecasting this themselves, even for CMX. The upside is if you buffer correctly to SLC, you can still use QLC. At WEKA, we’re skeptical, because these aren’t capacity workloads where you have the luxury of time to destage SLC to QLC — they’re very bursty, intensive workloads attempting to emulate memory. We think that, ultimately, one tier, whether SLC or TLC, is the only safe and fast way to offload KV to storage. But there are a lot of smart engineers across the industry and many vendors, so the market will prove what works over time. We just know what works for us right now.

Why over-provisioning NVMe drives protects write endurance

Vik: In a NAND storage rack, you can amortize writes over many drives, but do you also over-provision, and to what ratio? If you have 100 drives in a rack, to make sure you have the right endurance even after amortizing over TLC drives, do you actually provision 20% more?

Val: These are tricks we can always play. If we don’t have SLC — because of economics or supply chain issues — and need to use TLC, or God forbid QLC only, then yes, you’d have to over-provision. The net effect is you’d buy, say, a petabyte of storage but only see 300 TB or 500 TB of effective capacity from that petabyte you’ve purchased and are powering, because the over-provisioning is needed so you don’t break those drives.

Vik: So it could even be a two- or three-to-one over-provisioning ratio.

Val: Yeah, it’s tough, because these are extreme workloads. There’s nothing gentle about emulating memory with NVMe and NAND flash. It’s a very intense workload.

Vik: That’s great insight into how SLC and TLC tiers work and the tradeoffs — definitely added to my body of knowledge today.

Val: It’s nuance, that’s why I said we could spend a lot of time on this alone.

Vik: Let’s move on as there’s a lot of interest in the market right now around high-bandwidth flash (HBF). If people can put flash right next to a GPU and store things there, I would imagine that would be useful. My two questions: Does HBF use SLC, single-layer cells? And what are the real use cases for it?

Val: Great question. HBF will probably have to use SLC in the most common configuration, for the reasons we just discussed. It’s trying to be a great offload tier, in many cases trying to replace DRAM, complementing high-bandwidth memory directly. For me, it always comes down to a packaging decision. SK Hynix has been public about wanting to package it directly on the GPU package itself, not going through buses or networks, wanting it tightly bound to GPUs. I think it’s a great vision. I haven’t seen any GPU vendor directly commit to or adopt that yet, but I think we’ll see it maybe in 2028. There are other vendors that I can’t discuss under NDA that are packaging it differently, on a PCI card, not even with GPUs but with ASICs, non-GPU accelerators.

So there will be different packaging form factors, but ultimately I think it’s necessary, because the opportunity to get this right and optimize the KV transfer layer and get the scheduling right is enormous. The acquisition — I believe it was AMD acquiring Qualcomm’s stake, or was it modular — allowed getting compiler expertise, the way NVIDIA’s Groq acquisition brought over compiler expertise from Google’s TPU team. That’s going to be more important over time to making HBF really usable. As we recompile models to understand HBM and HBF tiers and maybe bypass DRAM tiers altogether, you can get a lot of bandwidth out of NAND flash devices over NVMe, or other protocols, or without NVMe.

There are optimizations available now with $1 trillion of opportunity encouraging optimizations between HBM and HBF with raw NAND flash and/or NVMe.

Vik: You can always replace drives if you’re running that bandwidth over a high-speed network with NVMe drives, and as we discussed with over-provisioning, if something dies you swap it out. But HBF is NAND flash after all, and even if it’s SLC, it has an endurance limit, and you can’t swap it out if it’s packaged next to the ASIC or GPU. Is that going to play out, or is the lifetime of the chip 50 years and it doesn’t matter since we’d probably throw the chip away by then anyway?

Val: I agree — that’s the bag of tricks engineers are throwing at this problem. It’s not one solution, it’s several. Over-provisioning is one. Proper amortization at the NVMe fabric layer is another. Then there’s very explicit scheduling at inference time, but also at model recompilation time, and very intelligent token routing at the MoE level and even at the gross model level.

Vik: So you can engineer the whole thing to the point where it’s not really a problem, and you can still use it.

Val: Not only that, but I’ve already seen anecdotes of people applying Fable to very complex engineering problems and seeing months’ worth of sophisticated systems engineering completed in four hours. One optimistic scenario is that these agent harnesses, the eval loops, the quality eval loops, the judgment models, the guardrail loops and guardrail tokens — when packaged together, some of these deep-tech engineering advances can happen much faster. Although ironically, we need more engineering in the harness to make sure they happen reliably and safely.

How CXL is losing ground to NVLink and RDMA-over-Ethernet

Vik: So much is happening, it’s really interesting to see how it all plays out. The one other thing that’s recently cropped up — and I know we spoke about this last time and you had ideas that I want to check have evolved — is the use of CXL. There’s a lot of talk about using it now since some DRAM isn’t being used on every server, maybe only 50% is used. Because memory is so expensive, even Google has had a change of heart and started using CXL to reclaim unused DRAM. Is CXL a thing now?

Val: CXL has a lot of fans, and I was one of them 10 years ago. But I’m not a fan anymore, and it’s not because I don’t like the technology; it’s because in the real world there are alternatives. Personally, I think Mellanox, and of course NVIDIA’s acquisition and great execution of scale-up domains, kind of killed CXL in one sense. The ability to have this great NVLink-style network, and have it used for high-bandwidth as well as regular DRAM memory, has been a real ceiling for CXL market growth — there’s less and less need. Then you’ve got RoCE, RDMA over Ethernet — you never bet against Ethernet in this industry.

The challenge with CXL is it’s another bus you have to engineer, debug, maintain, and power. That’s not bad in isolation, and it has a great killer app today in utilizing this precious, underutilized DRAM resource in some cases. But in the context of real-world alternatives, I’m pessimistic about CXL’s future, largely because — and I’m biased — I’m able to leverage things like RoCE or NVLink over InfiniBand and deliver better-than-CXL performance with NAND flash economics, cost of goods, and capacities. It’s one thing to pool underutilized terabytes of DRAM; it’s another to pool petabytes and exabytes of NAND flash at the same or better performance.

Vik: Won’t faster network bandwidth also benefit CXL the way it does other approaches?

Val: Yes — the first principles are simple until you have to engineer the real-world issues in. More bandwidth is good, and if you can have faster-than-PCI bandwidth to a pool of DRAM, as Google discovered, there are benefits. It’s just that if there really were such big benefits, I think you’d have seen it in Blackwell, in Vera Rubin from NVIDIA, in Helios from AMD, or even in Feynman, NVIDIA’s already-pre-announced next generation — and you haven’t seen it. You haven’t seen it as a standard Supermicro offering, or as a standard Dell, HPE, or Lenovo offering. I think that absence speaks volumes. There definitely are use cases for it, but for some reason it hasn’t broken into the mainstream.

Vik: Yet, probably, it seems like the shortage of HBM is now causing players like Google to consider it, because capacity issues are pushing hardware makers toward solutions they probably didn’t consider before. That’s another interesting thing to watch play out, because you’re right, there’s a reason it hasn’t been used until now.

Val: There’s a reason for that, and I think there’s one important hint. It was reported by SemiAnalysis about two or three weeks ago that NVIDIA changed the BOM on Vera Rubin and cut the actual amount of DRAM in half — from 4-to-2 or 3-to-1, to 1.5-to-1, depending on the model. That’s a clear indicator that, A) DRAM has gotten too expensive, and B) NVIDIA is projecting, with CMX solutions in the marketplace from a number of vendors including WEKA, that there will be less need for DRAM for inference. So you’re seeing people apply solutions to this problem, and it’s not always just pooling it better, it’s reducing it overall.

Vik: Right, that’s part of what we spoke about earlier. You could make optimizations to the algorithm, use sliding window, do a lot of different things that use less memory overall going forward, rather than just using existing architectures and pooling stuff together. If we find better algorithms, we probably don't need to pool it — it could go either way.

Val: That is I think the prediction we’re making.

Why AMD’s MEXT acquisition and AI-driven cache prediction matter

Vik: Speaking of better algorithms, what’s your take on AMD’s MEXT acquisition? For people who haven’t heard of it, MEXT is a software company that found a way to optimize DRAM use by dynamically offloading unused parts of DRAM to NAND flash, then using AI to predict when that same information will be needed back in DRAM, and preemptively moving it back before the GPU even notices it’s gone. It looks like a cool way to use NAND. What’s your engineering interpretation of what’s going on?

Val: That was fun to review, since I wasn’t familiar with MEXT beforehand, but reviewing their use cases pre-acquisition and the initial positioning post-acquisition, it’s clear that it’s AI technology — machine learning specifically, small language models and small neural networks — optimizing caching algorithms, being more semantically aware than simplistic heuristics like least-recently-used.

So it’s the application of machine learning and deep learning into caching algorithms, but the actual use case, ironically, isn’t yet KV cache offloading — though it has potential to be very good there. We’ve seen popular in-memory databases like Redis implement algorithms that aggressively destage DRAM to NAND flash and retrieve it back, and Redis does position that for KV cache offloading as well. I think it’s an active space, but right now AMD is initially targeting scientific computing such as life sciences, Monte Carlo simulations, seismic analysis, weather analysis. Those are the applications the technology has proven itself in. We may see it appear in KV cache offloading as well.

Vik: I thought it was an interesting use of the predictive nature of an LLM: If you can predict the next word, why not use it to predict what the next page of memory is required and quickly pull it from flash to DRAM? I don’t know how practical or useful it is, but I found the idea interesting.

Val: Your instinct is right. Why has Cursor customized Kimi K2.5 into Composer? Many companies are realizing the bar toward training your own model has come down; it’s much more accessible now. You don’t need million-dollar ML researchers to train your own models anymore, and when you can customize a model for your domain, it doesn’t have to be an LLM at all. It can be an ML model, a very tight neural network — it can be inferenced on a CPU, even a small, low-power CPU, if the model is really domain-specific. It’s just a neural network at that point, not a large language model. I think there will be another Cambrian explosion of use cases and applications for clever small models that do things heuristics peaked at and can no longer optimize or improve.

Small models, big edge: Why robotics doesn’t need LLMs

Vik: Do you think this use case applies to physical AI and robotics, where sensors at the edge process very specific amounts of information? Just one kind of information from a sensor, so you don’t need a large model? Is that a useful use case going forward?

Val: 100%. I think it’ll probably be the reference architecture for robotics and edge inference; we don’t need large language models there. There will be some aggressive cloud connection, whether through Starlink in remote locations or broadband, if you really need to burst to some complex decision a large language model has to make, but 90% to 95%, maybe 99% of inference for robotics will be local and disconnected — air-gapped, so to speak.

Vik: The fact that you could make inference decisions at the point of sensing — put intelligence anywhere — is a very useful edge use case. How useful or good that intelligence is yet to be seen, but in principle you could deploy these little models everywhere.

Val: Yes. The cost of these Raspberry Pi-style SOC (system on a chip) motherboards is really plunging, and fortunately — whether it’s KV cache optimizations, small model quantization, or custom neural network training that doesn’t have to be a large language model at all — this is intersecting really well with affordable SOCs. That results in some really interesting robotics and drone use cases.

Vik: Didn’t Jensen (Huang) mention something about the AI flywheel in this context?

Val: Exactly. It’s a general concept about being able to capture domain-specific data: Train models, then customize with more real-world, domain-specific data, or synthetic data now that you have enough real data to create useful synthetic data, and keep iterating on that loop. You push the frontier with big models, customize through fine-tuning, distilling, quantizing, low-rank adapting — all sorts of customizations — and customize smaller and smaller versions of those models. You’re able to retrain entire small neural networks that are very domain-specific, and get more efficient at processing inference, retaining fresh data, and keeping the flywheel going. It’s a mix, a whole thriving ecosystem of different model types and different types of data. If you keep the flywheel going, it stays relevant with nature’s natural entropy.

How SaaS giants and neoclouds are in a race to own the token stack

Vik: That’s a fascinating idea. Since we’re coming up on time, I want to pick your brain for one prediction: What do you think will happen in the next 12 months, and what are you most excited about?

Val: Zooming out, this is something Jensen references often: Software is fundamentally changing. A year ago, we wouldn’t have predicted all of our engineers would be using AI for most of their daily work right now — it was heresy. So not only is the rate of change happening right now remarkable, but the fundamental nature of software is changing: More and more software isn’t just compiled and run; it's compiled, run, and inferenced. It’s really agents now, being much more intelligent about token consumption. We’re not using the biggest model for everything anymore; we’re using model routing and a mixture of models, and a mixture of experts within models, to be very token-efficient. This means the cost of running a software business is radically different than before.

It’s a high marginal cost. You can’t just leverage token costs across users the way you could leverage cloud instances, databases, VMs, micro-VMs, and containers across users — even with KV cache as a leveraging mechanism. SaaS companies — the reason I believe the “SaaS-pocalypse” is real and a genuine problem — is that no matter how much they figure out new pricing and value models, their OpEx is going through the roof. Their opex is now token OpEx, tokenomics, token consumption. All these engineering solutions we’ve just discussed are ways to manage that, but if you look at token volumes on OpenRouter, week after week, they keep rising. We’ve barely begun mainstream token consumption and persistent agent swarms.

The only way to run a profitable gross-margin software business will be to own more of the token stack. You can continue outsourcing that to an inference provider or model provider, or you can acquire it: Merge with a neocloud, merge with a token factory, vertically integrate more of that very expensive token-generation stack to keep running a high-gross-margin software business. The real question is: With cash flows what they are, will SaaS giants acquire neoclouds before neoclouds are able to acquire the SaaS giants?

Vik: Fascinating. I’ve always said the best way for large companies to save on token costs is to bring inference on-premises, running it at the edge. What you’re suggesting puts that concept on steroids: A big enough software company can acquire a neocloud and effectively own its token factory. There’s still a cost to run it, but the tokens are entirely yours to use. If every big software company starts acquiring neoclouds of some size — they don’t have to be multi-gigawatt data centers —

Val: I predicted this before X, or SpaceX AI, acquired Cursor. I thought something like Workday or Monday.com would merge with a mid-tier neocloud, but now the first domino has fallen with SpaceX AI and Cursor, and older established SaaS companies will have to react as well. Whether the dominoes start falling in the middle or on one side or another, it’s kind of inevitable now that most SaaS companies and most neoclouds will have to merge.

Vik: That’s a fascinating prediction. I’d love to see how it plays out. Thanks so much, Val. It’s always a pleasure chatting with you. You’re a fire hose of information, and I know I’ll be listening to this podcast again myself while reviewing the edits, thinking, “Oh my god, I missed that.” I hope this helps all our viewers too. It’s really a pleasure.

Val: Always a pleasure. We said we’d do it again next time, and I’m looking forward to coming back a few months from now — well before the end of the year, because by then we’ll be surprised again by what happens.

Vik: It’s an eternity. Every three months is an eternity in AI time. So we should do this more often.