# Why AI Inference Margins Are Suddenly Getting Better

**Published:** August 27, 2026

![Three speakers on a stage during a panel discussion.](https://cdn.sanity.io/images/ult5g8gw/production/e456ecbaf698f409b67e778c1d83d2a7f9aa3183-960x540.jpg)

**Watch:** https://fast.wistia.net/embed/iframe/qw9ynl2wtb

Redundant prefill calculations stall agentic inference. Offloading KV cache to storage enables 95% hit rates, boosting GPU ROI.

## Speakers

- **Liran Zvibel** — Co-Founder & CEO, WEKA
- **Dylan Patel** — Founder, SemiAnalysis

## Transcript

Recomputing the same tokens is the expensive mistake hiding inside cheap AI inference. In this short clip from RAISE Summit, Dylan Patel, Founder of SemiAnalysis, and WEKA Co-founder and CEO, Liran Zvibel, explain the fix.

****

**Liran Zvibel: **Looking at large-scale infrastructure, and obviously we've all had tons of experience with training, and a lot of the infrastructure that was built up until now was optimized for training. The GPUs were the bottleneck, and then you were trying to figure out, “How can we fit as many GPUs as we can?” You also had some memory limitations there, but you were spreading it across so many GPUs, you were able to make batch sizes reasonable enough to make good progress. With inference, especially when you’re adding parameter counts, you’re getting to models now that are half a terabyte and more.

In many cases, you can’t even fit them on a single server with the older kind of GPUs. You have to go to a Wide-EP (Wide Expert Parallelism) when you’re generating across a lot of GPUs, and then while you’re running inference, you have a growing amount of context. You probably have heard of KV cache (and this is where it fits in).

So the tokens you’re generating, if you have just one go, you give a prompt to get the answer, it’s a perfect answer, and you go away. There isn’t a lot you can optimize for. But now with agentic workflows, and we all know we never get the right answer the first time. We have a multi-turn.

The more turns you have, you have to have all the answers, all the tokens from the previous turns before, so you can generate the next one. If you have a 10-turn or 20-turn flow, it’s quadratic, so you have to go and recalculate 100 times or 400 times between the 10 and the 20, the same tokens over and over. The kind of technology WEKA is building, and others are building, is the ability to store these tokens, never recompute again, and then push them to the GPUs as fast as you can — because when you’re looking at GPU yield, or how many of the cores are being used for inference, most of the chips are actually wasted.

So if we can make sure that you never calculate the same token twice, and then the GPUs never have to wait for these pre-calculated tokens, we’re going to increase the amount of utilization significantly.

**Dylan Patel: **One of the very interesting aspects of inference over the last two years is chat multi-turn.

People would wait 30 seconds or even an hour between sending a question back to the model. And when you look at inference, there are two main things. There’s prefill, which is calculating the KV cache, calculating the context. That’s very parallel, but the vast majority of the usage was actually in decode, right?

Then there’s generating each individual token. Now, when we look at the agentic flows that people have today, we’re seeing the context length is oftentimes 30,000, 100,000 tokens into the context of a model, and you may only generate 1,000 tokens. Initially I’d say, “OK, well, now I need a ton of compute to calculate all the prefill tokens.”

Today, and even two years ago, you see the cost of a prefill token, right? The input would be 3x or 4x lower than the output token. But one of the things that WEKA as well as many others are doing is now you calculate that once, you store it off in memory, whether it be system memory or storage, and then you pull it back in when you run the turn.

And so now you have a massive decrease in cost because you don’t actually have to do the prefill again. There’s this recomputation of prefill (many people, their infrastructure is still not ready for it), so you have to spend again. If it’s 100,000 context, even though the tokens are 4x cheaper because it’s prefill, it still far outstrips the decode side.

But now, because you’re offloading it and you’re pulling it back in, not only is your time to first token faster because you don’t have to recompute, but also when you pull it in and you start generating tokens, your cache hit rate – i.e., the how often you need to recompute the prefill — is so small, we’re seeing cache hit rates above 95% for many agentic workflows. Which means the cost for a cache hit is not free, but it’s way, way cheaper than these very expensive GPUs.

And now this means your GPU utilization rises drastically, and the amount of time the GPUs are generating new tokens is far higher than the amount of times GPUs are generating or recalculating the context, which is not necessarily providing any value to your operations. You’re only generating the value when you’re decoding tokens, but you have to prefill to get those decode tokens out, and you’ve offloaded that to storage now, so it’s much cheaper.

## Related Videos

- [Power, Tokens, Talent: Inside the $7 Trillion AI Compute Race](/video/compute-as-capital-redefining-ai-s-new-currency)
- [Inference Is Eating Memory, and Tokens Now Run AI Economics](/video/inference-is-eating-memory-and-tokens-now-run-ai-economics)
- [Build for What's Next in AI with NeuralMesh](/video/build-for-what-s-next-in-ai-with-neuralmesh)