# Win Back Wasted GPU Compute With KV Cache Efficiency

**Published:** November 18, 2026

![Rob Strechay and Alan McSeveney seated on stage during an AI Impact Series panel discussion.](https://cdn.sanity.io/images/ult5g8gw/production/6da1685d6ed0016387d52e9764025e33e4067e3f-960x540.jpg)

**Watch:** https://fast.wistia.net/embed/iframe/gkzu5ktqwf

WEKA field CTO Alan McSeveney joins Rob Strechay at the VentureBeat AI Impact Series to explain why GPUs redoing KV cache work lose up to 60% of their compute, how NVMe offload can outperform DRAM, and whether to own or rent AI infrastructure.

## Speakers

- **Alan McSeveney** — Field CTO, Media and Entertainment & AI, WEKA
- **Rob Strechay** — Lead Analyst, VB Intelligence, VentureBeat

## Transcript

### [00:00] GPU efficiency is the top AI infrastructure challenge

**Rob Strechay:** So next up is Alan McSeveney, a field CTO over at WEKA. But Alan also had a life before WEKA where he was the CTO at a company as well. I had a lot of fun talking with you about this. You’ve heard what Byron and Ben (prior speakers) were talking about. What do you see as some of the commonalities between what they’re talking about and when you’re out talking to customers about the challenges they’re seeing?

**Alan McSeveney:** Trying to optimize your own application set to squeeze as much out of, or to be as productive as possible. That’s one place that Byron focused on, getting your KPI right. That’s within your domain, and it's very specific to every customer or every business. The place that we start to see a lot of inefficiencies that people are really struggling with is more on the infrastructure side, and trying to find a way to squeeze more out of a capital investment that they already have.

And those limits could just be purely budget, or they could be additional limits like power availability or data center space. These are real constraints that are really getting in the way of a lot of organizations’ attempts at growth right now. So trying to do more with less, or with what you already have, is very real for a lot of people.

**Rob:** Yeah, and I think Ben was even hitting on that with the HPC cluster and using that, and how many folks were coming in. And I’ll steal this from our upfront. He was talking about it could be a week, it could be a month until you get access to that. But if you can make that more efficient and get more turns on those GPUs, you could easily get more through.

### [01:56] What is GPU ‘busy work’?

**Rob:** But that brings up a pretty interesting thing, because when we were talking about this beforehand, I think part of it was that aspect of what is busy work versus real work for a GPU. Help us understand how organizations may want to distinguish, because, again, busy work isn't really adding to it. And Byron talked to that a little bit as well, the busy work, and pulls based on pull requests and things like that. But help people understand what you mean by the busy work versus the real work.

**Alan:** Model choice was alluded to. A customer asked Byron earlier about targeting the correct model in the first place. Are you running a model of the correct size for the task? Jev AI was just launched, which is available in OpenRouter now, and its sole function is to try and make a decision about where to route a prompt to, so that the reasoning can happen on the right size model. If you can do this with more models spread across the GPUs that you have, and they’re only handling a problem that is of the right size for the model, it’s a much more efficient use case. There’s also context relevance, which is precisely your KPI comment. You have to make sure that the model is being driven by the correct context in the first place to be able to make the correct decisions.

The tokens you’re getting back are actually goodput, not just more tokens. Generating more and more tokens seemed to be a big rush at some point, but now people are realizing, “Actually, I would need more out of less tokens.” It’s the only way that you arrive at something that is truly useful for a company, and the economics add up. At some point VC money runs out, and when the gravy train is over, people need to be left with a functional system that can actually return value.

And so the other part of busywork is GPUs recalculating calculations that they already have done before. I don’t think people appreciate what percentage of a GPU’s time is spent doing that.

### [04:20] How KV cache recomputation wastes up to 60% of GPU compute

**Alan:** So the two main parts to an LLM or its calculations are prefill and decode. In a multi-turn conversation or a multi-turn session, it could be a user, it could be an agent or multiple agents, every single part of the context, the entire history of your session, is the context for this particular turn. So that footprint just grows more than linearly. It grows quadratically in terms of its footprint on memory. But you need a large amount of the memory already taken up by the model and weights, so there’s only so much you can actually allocate to cache. When that cache is actually filled, then the context gets dropped, and it has to be recomputed all over again, the exact same questions and results that were provided before.

So we see in HBM on GPUs, this can be as much as 60% of the compute on a GPU that can actually be recomputing things it’s already done before. That's pretty wild.

**Rob:** That’s a lot. That’s pretty high overhead for what you’re doing, I would assume, if you’re trying to do that. But one of the concepts that comes from that is the fact of this, I guess you could say, repeated memory or long-term context memory, and the separation of the two, or are they the same? And there’s a lot of techniques that are out there for really looking at that. Talk to what you’ve seen with the material impact from an economics perspective of using the repeated or that long context inference.

**Alan:** Running without any strategy to deal with that can leave your GPU running at as low as 30% utilization where it’s actually doing any new work. Get a good strategy and you can see that north of 90%, 93%, 95% in a best-case scenario. Given that when you build your own infrastructure to run AI workloads, more than 80% of your budget goes on GPUs themselves. So for them not to be really firing guns all the time on things you need it to be doing is a colossal waste of money.

### [06:57] NVMe flash can outperform DRAM

**Alan:** So we see different companies and even model builders trying to come up with different techniques to improve the situation. Mooncake is a good example. DeepSeek 4.1 has introduced some new techniques to improve KV cache reuse. I think what we find is that offloading to DRAM seems to be a technique that a lot of people use, but everybody here, I’m sure, knows that the cost of DRAM today is not really a place you want to be having to spend additional money on. And that’s not getting better anytime soon.

So we’ve seen even companies like NVIDIA recognize this problem, and they’ve brought out CMX, which is a cache memory extension module that will be part of the Vera Rubin stack. Because using flash and using NVMe, the actual performance metrics versus using DRAM for this use case, it performs as well or better, scales much better, and is much, much more affordable. I think we’re going to see more people heading in this direction to really bring the costs of their infrastructure spend down.

### [08:19] What is the ideal KV cache retention time?

**Rob:** And I liken this to being, when you’re using things like key-value cache or KV cache, it’s like putting a sticky note out there for a certain amount of time for the agent to use and reuse that context. And then you’re able to also keep all that institutional knowledge back in. It’s the more, I guess you could say, hardened infrastructure from that perspective. How would you see it that way?

**Alan:** Well, the KV cache, ideally, you would keep it around. The sweet spot seems to be around 72 hours. That’s the length of time it takes for a user to stop using a session and return to it, or an agent might be parked and then resume. But while a GPU stops working on a particular session for a given user, then it needs to immediately pick up work for a different user or a different agent, some other session. It can’t be sitting, waiting to see if you might start or resume. So getting all that data off as quickly as possible, let somebody else occupy the GPU, and then get their data back on when they need it, 72 hours seems to be about the ideal number.

**Rob:** So economically, this helps them be able to manage that thrash, I guess you could say.

**Alan:** Well, if you can drop the data off and bring it back on quickly, the number of users that can be served from every single GPU can increase by 4x, 5x for the exact same GPU. So you take an array of GPU servers, and instead of buying more GPUs and throwing GPUs at the problem, you can literally fit 4x or 5x as many people or agents on the exact same infrastructure. I don’t think anybody in the room would turn that down.

**Rob:** No, I don’t think so. All right. We're 10 minutes in. We've got a question all the way in the back.

### [10:26] Should you optimize models or infrastructure first?

**Audience Member 1:** I’m from the Microsoft Copilot team. So first of all, thank you for the talk. Out of all the businesses and companies that you have worked with, what do you see is the first place that the optimization goes to? Is it more dealing with the weights and quantization and fine-tuning and techniques like those, or optimizing your harness around the models? Which direction do you think needs more work?

**Alan:** I think that’s very varied between customers, but context, and the harness, is a huge part of it. I think anybody who’s spent a reasonable amount of time now working with AI and LLMs and trying to implement them in their workflows are getting a pretty good handle on how to reduce hallucination, how to stop waste, and make sure that results are properly validated. So I think that just becomes a maturity in each business’s own workflow. I don’t think there’s enough understanding yet on the infrastructure side on how more headroom could be made available within a system for additional queries and additional users. That still seems to be a pretty large knowledge gap across many customers.

**Rob:** From what I've seen from some of the organizations that we talk to and when I'm at these events, one of the big themes has been, to help from an economics perspective, owning that harness layer so you can own the governance, you can own a lot of the context, you can own a lot of these things that he’s talking about. Then optimizing it underneath there tends to be where a lot of them get really good bang for their buck. And there’s a lot of open source harnesses, and there’s a lot of support around that.

Also, in addition to Jev, you’ll see some stuff coming out with vLLM routing models and stuff like that within the open source community, which will be helping to actually use the KV cache better underneath the hood, on top of these types of solutions. So I think there’s a lot of really cool stuff that’s really pushing that way to help the economics. But the harness is really where I’ve seen most people try to own it versus rent it, because it becomes more expensive from that perspective.

**Alan:** On the infrastructure side, we’ve worked with, for example, Oracle Cloud. So [OCI stood up a nine-server GPU cluster](https://www.weka.io/news/weka-and-oracle-cloud-infrastructure-validate-10x-throughput-gains), fully loaded, and ran this with DRAM as offload for KV cache. And then they switched to using an NVMe-based solution that could not only capture the KV cache, but then load that KV cache centrally back in, and not just the same host, but any host. So any session could stop and resume on any other host in the data center. And the speed at which it could do it, this drove the number of users on the same GPUs up by 10x. The number of tokens it could generate went up from 200,000 in the same timeframe versus DRAM to 2 million. So it makes a pretty big difference if you can optimize for this area, but it doesn't discount what you’re talking about. You still have to get your application and workflow and pipeline as optimized as possible.

### [14:20] Own, rent, or both?

**Audience Member 2:** So I think WEKA is doing some really cool stuff with file systems. I like what I found out about just doing super insanely fast file system access. It’s pretty neat. The problem that we’re describing today about token economics, and then I believe what I’m hearing the answer that you’re proposing is owning the stack versus renting the stack, right? So are you describing building the system architecture yourself versus what you get when you pay for Claude or Anthropic or Bedrock or whatever you’re doing? Because I’m hearing you describing how you can architect the hardware layer versus what you’re doing with KV cache and how that’s being handled. So I’m very curious how you’re approaching the stack level. Is it, “I own this, we have the money, we're doing this,” or, “This is how you can engineer this because you're paying for it, you're renting it”?

**Alan:** Ultimately this ends up benefiting the GPU owner. So that could be a service provider. It could be your own GPUs. You own the GPU, or you’re renting the GPU but you’re not just renting the service on it. If you’re actually renting the hardware and running your own stack, then this is where you get to benefit from these optimizations. If you look at a foundation model provider, you’re paying them per token.

**Rob:** Yeah, so it’s hybrid, right? It can be a hybrid depending on if you’re using cloud instances. You can still get benefit depending on how you’re paying for that instance, and that’s where I think, to your point, obviously if you own the hardware and you keep it busier and you have your AI factory, you’re gonna get even more out of that. But that doesn’t mean you can’t use some of those foundation models and have them tied in through your harness and things like that. You’re not going to be necessarily optimizing for that foundation model. You’re optimizing for everything else.

So that becomes a balance between what you do in-house, and what we’re seeing just talking to organizations is that they are looking at that. Maybe they’re using the foundation model as part of their orchestration layer within the harness, but they’re actually using SLMs, the small language models, that are more personalized to what their task is at hand, and those are on-premises or in a cloud instance that they’re running at that point in time where they can take advantage of things like KV cache and that type of architecture.

Apologize. I worked for Amazon at one point in time and built a service there, so I sometimes forget I'm a geek...

**Alan:** You said you’re not even allowed to use foundation models, right? It’s not an option, so running models yourself ends up becoming a prerequisite. I think as we start to see models that are not the foundation models get up to a point where they’re good enough to perform many workflows, we’ll see more of a trend toward people running things locally or in their own data centers. But there’s also the demand for GPUs is so high that these foundation model companies or service provider companies are really struggling to even be able to have enough GPU to continue to serve tokens.

I even had colleagues today who had to sit twiddling their thumbs because a foundation model couldn’t serve them… It is tough to actually get access to compute on demand. And that won’t be any different for service providers. So running some of your own infrastructure might not necessarily be your first thought, but it may become a necessity.

**Rob Strechay:** Well, I want to thank everybody. We went a little long, but I appreciate everybody’s questions. And Alan, thank you very much for this, and I want to say thank you to WEKA for sponsoring and making this all possible. Thank you very much for coming.

## Related Videos

- [The New Math of Inference](/video/the-new-math-of-inference)
- [Why Inference Economics Are Reshaping Neoclouds](/video/why-inference-economics-are-reshaping-neoclouds)
- [Power, Tokens, Talent: Inside the $7 Trillion AI Compute Race](/video/compute-as-capital-redefining-ai-s-new-currency)
