# AI Broke the Yardstick. Now What? **Episode:** 7 **Published:** October 13, 2026 ![Deep Geeks podcast episode 007 featuring Vernon Turner (Cognizant) and Val Bercovici (WEKA).](https://cdn.sanity.io/images/ult5g8gw/production/a6ec2106b4af3e47f4202b453954a04aea6d3925-1800x1013.png) **Watch:** https://fast.wistia.net/embed/iframe/qupxoynpm5 ## Listen On - [Spotify](https://open.spotify.com/episode/34exgWTXJW07HXEYd5l2jx?si=ZaVXDnzoQIG2sMR0rPhfTw) - [Apple Podcasts](https://podcasts.apple.com/us/podcast/ai-broke-the-yardstick-now-what/id1896864991?i=1000796100105) - [YouTube](https://youtu.be/KoioZyVaXrE) Your data center can hit a near-perfect PUE and you still can't say whether AI is paying off. On this episode of Deep Geeks, Dr. Serena Huang talks with Vernon Turner of Cognizant and Val Bercovici of WEKA about what should replace PUE, the inference costs nobody is counting, and where to start measuring AI workloads. ## Guests and Hosts - **Val Bercovici** — WEKA - **Vernon Turner** — IoT and Sustainability Advisor, Cognizant - **Dr. Serena Huang** — Host, Deep Geeks ## Transcript ### [00:00] The Metric That May Have Served Its Time **Val:** A PUE of 1.5 or two or above kind of disqualifies you from even having an efficient data center conversation. **Vernon:** I was there at the beginning helping to create the PUE framework. So it's ironic that 20 years later I'm there going, you know, it may have served its time. ### [00:18] Meet Vernon and Val **Serena:** Welcome back to another episode of Deep Geeks. I'm your host, Dr. Serena Huang. Today I'm joined by Vernon Turner from Cognizant and Val Bercovici from WEKA. Thank you so much for joining us today. **Val:** Glad to be here. Really thrilled to be back, Serena. **Serena:** Welcome back. This is going to be a great follow-up conversation to our previous episode. ### [00:44] From Watts to Outcomes: Why PUE Falls Short **Serena:** We have spent the last couple decades getting very good at measuring data center efficiency with PUE, and today we're going to dive into that. In the past couple years we've also gotten really good at measuring something else: AI activity, I will call it. How much compute, how many tokens, how many users we have. But that doesn't really tell us whether or not AI is creating value. So we're going to talk about that today all the way down the stack, from watts to tokens to workloads to outcomes. And then we're going to share with our listeners what leaders should actually be measuring in this new AI economy. So excited to get started. I'm going to start with a tough one for Vernon, because you have said this very provocative statement that PUE, which we have been using for two decades, is just not enough. Tell us why. **Vernon:** Yeah, and it is a tough one, because I was there at the beginning helping to create the PUE framework. So it's ironic that 20 years later I'm there going, you know, it may have served its time. That's not to say it's a bad metric. I think it's a metric for the time when we were looking at usage and trying to figure out how much density we could pack into a data center, but we're measuring the power outside coming into the data center. If you think about that metric alone, how much we can pack into the data center over the last 20 years has just gone off the scales. It's not even Moore's law. It's a whole different dimension of how you should look at that. And that made me start to think: 20 years ago, we would measure the effectiveness of the power consumption, the power usage. In other words, how much power was coming into the building and how much was being used, and come up with a very straightforward, user-friendly metric. Just for the sake of history, if you had a metric that was 2.0, then that probably meant you were only half as good at using the energy coming in to you. In other words, you had leakage all over the place. And over the last 20 years, our data center ops, our data center teams, everybody's gotten pretty good at cutting that down if you're an on-prem data center. And if you're one of the large hyperscalers, you're probably running very efficiently anyway, at some metric like 1.2 or 1.1. As a result of that, we're kind of there. How much more can we get out of the curve? How much more is efficiency going to matter? But Serena, you mentioned one thing that I think is important. You talked about efficiency, and this is where I would push back on everybody. PUE is about usage. Today we really are talking about how efficiently we can run a workload based on AI. And that gap is what brought me to this whole conclusion that PUE just may not be the way we need to go. Maybe we need these other metrics that measure how efficiently we're coding when we link it to AI, up the stack, all the way up to the output itself. And one of the things I talk about is AI commerce. In other words, what are you generating from AI, either by agents by themselves, and can you measure the cost it took against the revenue it created? That's a completely different discussion from when we talk about data center efficiencies. **Serena:** I agree, and that sounds so much more challenging than just measuring PUE, all the way to the output. I think that's where most businesses do want to get, but it's very difficult. So we'll dive in today on how we get there. Val, let me ask you, how does this resonate with you so far? ### [05:51] Token Efficiency and Cost per Task **Val:** Oh, completely. This is the most important foundation for just understanding whether we're even in the right ballpark. If we're looking at efficient AI and sustainable AI, we've got to at least start with a decent PUE in the ranges that Vernon mentioned earlier on. A PUE of 1.5 or two or above kind of disqualifies you from even having an efficient data center conversation. I proposed something about six months ago, when I was getting into tokenomics more deeply: the notion of token unit efficiency. That was an interesting notion, and it still is. It's still a foundational way to approach this. There are no standards for measuring this yet, so Vernon, maybe it's something we can work on. Tokens are such a transparent metric in the industry, differently on inference than on training, obviously. With training, it's weights and checkpoints, and that's a little too opaque for my liking. But inference, which dominates AI spend nowadays, is super transparent around input token costs, output token costs, and particularly cached read and cached input token costs as a hyper-efficient version of that. So that's one way to take the conversation. But in hindsight, given the rapid advances we've seen with agents and the mainstreaming just last week of Meta coming out with a very popular Muse agent, now number one in the App Store, at least in the US, that is a clear step-function increase in data center utilization for AI inference, for this very heavy agentic workload that is now persistent. It's 24/7. It's no longer just event-driven. And I think the engineering concept of goodput, making sure that we're actually generating useful output for the energy and other resources invested, is more and more important right now. One of the trendsetting indicators is OpenRouter. They're no longer just measuring, very well, as they do in their leaderboards, cost per token, KV cache hit rate and so on. They're now talking about cost per session, and they're showing leaderboards for cost per session. Other folks outside of the economic side are pointing toward newer kinds of benchmarks focused on cost per task, and these new Pareto curves: how are these new models doing in terms of quality and performance, and how are they doing in terms of cost per task? Are they burning thousands of tokens per task, or merely hundreds or tens? That's where I'd love to see the conversation going. Not just token usage efficiency, which I think is the next step forward, but AI session usage efficiency, and ultimately outcome-based, task-based usage efficiency. That's where we should be taking this conversation. ### [08:20] Is Every AI Seat Worth the Same? **Vernon:** I think you really hit something on the head there, and that's the idea of the session. I know you've talked about it a little bit in the past: first of all, is the PUE model broken, but is the SaaS-based model broken as well? When I think of what you just said there, I've got to believe that the SaaS model... Not that I'm trying to put SaaS out of business, but what I'm saying is, how do you as a customer look at a model like that to measure your AI? Because is every AI seat the same? Are you getting the same value out of each AI seat? Is one person more valuable than another, depending on what they're doing? I'm not necessarily saying let's go down this path here, but it ties in with the whole framework of what we need to do with AI and how to measure the value that it brings. Let's start with this: we looked at the infrastructure ever so slightly with the data center PUE. But now we have to look at the SaaS part of it as well, to say, how do you charge me? I'm procurement, or I'm finance, or worse than that, I have to do chargeback. Heaven forbid cost recovery comes into this model. I would ask anyone in the audience, how many of you implemented a chargeback system that works? And then you'll hear crickets, or people will duck, because it's fundamental. Val and I probably remember the day when, for you to be able to provide new hardware, you had to cost-recover the use of that hardware over time, and therefore you had to be able to charge back the cost of that to the business unit. I think it's the same thing here. That's where this discussion with AI and tokens needs to move forward. And I'm not sure the software model is ready for that type of thing. So again, probably an opportunity for a lot of good work to be done there. **Serena:** You just pointed out something that I don't think we've quite talked about on the show as far as measurement. We've talked a lot about measurement, but I think we have stayed at the organization or system level, in aggregate. What you're pointing out, and I think we all know this intuitively, is that everyone consumes and uses AI in different ways, and therefore produces different output. There's good output and there's less good output. But the measurement, the pricing model, at least in the SaaS world, treats them all as identical, and that creates, to your point, probably a bigger problem now than in traditional SaaS. ### [11:15] The SaaSpocalypse and the Costs Nobody's Counting **Val:** I'd like to really pull on this thread, so I'm glad you brought it up, Vernon. The SaaSpocalypse has been discussed most of this year, and it's funny because the conversation goes from these popular memes of "it's so over" to "we're so back" to "it's so over." There have been a lot of ups and downs, just looking at Salesforce as a proxy for the SaaS industry. I still think we're only talking about half the conversation, because most of the industry and other podcasters and influencers are happy to talk about whether there's been product-market fit for agent-enhanced SaaS, and whether these companies can charge for agentic features. And in fairness, they are starting to find product-market fit for extra value, presumably at extra revenue, for these AI-powered features, Agentforce and all these names. But everyone keeps ignoring the costs. Everyone keeps ignoring the unsustainable, inefficient inference that's being used right now. Most of the inference is not leveraging cache reads. Most of the inference is doing redundant prefills, which is the biggest sin of inference right now, and almost every inference provider is committing it. So we're still having these very incomplete conversations, and there's even a very loose proxy with the highly anticipated Anthropic IPO and the S-1 filing. As people are digging into that, they're realizing Anthropic's revenues are breathtaking. Their revenue growth is outstanding. But their training costs are even bigger. And it seems like everyone wants to ignore these massive, unpredictable, lumpy training costs when you factor in the overall profitability of a high-profile business like Anthropic. So we need a much more complete view of what's happening in AI. We need a much more granular view, to your point, Serena. We're seeing quite a bit of diversity, not just across different kinds of agents, but within the same agent swarms. There are background agents, there are latency-sensitive foreground agents. There are all sorts of other parallel agents in an agent hierarchy. It's a research-rich field, it's young, and it's evolving quite dynamically. We need that complete view of it all to begin to measure it, understand it and optimize it. ### [13:36] Cloud Inference and Broken ROI Promises **Vernon:** I think one of the unintended consequences of AI running on a cloud service provider is exactly that: where you do your inferencing from. If you use the old models of simply saying, "We're just going to lift and shift my infrastructure hardware," then you're caught in a really difficult situation, where the ROI you thought you were going to get out of migrating to the cloud really starts to come under strain and pressure. There's a part of me that thinks it would be nice to just leave some of the infrastructure at home, where I could do the training, and then do the inferencing up in the cloud. And again, maybe that's part of the software model we see in the future for someone like Salesforce, whether you have a hybrid model or an on-prem model, and how that all fits together with AI. But that brings a lot of complexity, and people's heads start to spin and hurt when you go down that path. I think we're going to have to face that reality, otherwise we're going to have a situation of broken promises. Whether it's a financial promise or a value promise, or even an infrastructure promise that said, "This was going to bring me all these new products and all these new savings in efficiency." Whereas it might bring a whole generation of new IT professionals that could help us with the situation. ### [15:25] A New Framework: Tokens per Watt to Tokens per Workload **Serena:** Great conversation so far. Loving all of this. Now I want to make sure we give our listeners something concrete to take away. If PUE is not the right path forward, what is? Vernon, can you get us started, and maybe paint a picture if you have something more concrete inside a company? Help us visualize what the better metric is. **Vernon:** I have a framework that I've played with. My point is that PUE is a foundational part of this whole journey that I drew a framework for. Right above that is the idea that if you know how much energy you're using, then tokens per watt would be the next logical step. Then you can begin to understand what type of configuration you're running. Val, to your point about caching, about low IO, about the whole idea of server, storage and network efficiency and capacity. Then we can go a step higher, where you start to look at tokens per person, and that dovetails into the whole software model of shifting from a consumption model to a subscription model, and vice versa. And then at that point you could start to look at workloads. The reason I bring in workloads is that at IDC I helped run the workloads research arena. In the old days, if you had file and print, or transaction, or batch, or real time, remote and so on, you could bucket those into workloads. You can do the same nowadays, whether it's retail or finance or whatever applications are put together to make a viable product or workload. At that point, we could do the same with tokens per workload. And once you get to that area, I think you're on the golden journey of saying, "Now I see where the costs are going. Is it all in one particular vertical that I support? Is it all going to one particular application or workload?" At that point, I can join together the lower stack and the middle to upper stack and calculate the cost. That's where I think we end up. So instead of just looking at PUE and somebody saying, "I'm really efficient. Look at me. I should get a prize from the LEED consortium," you can stand up and say, "I can look at my business units and say, 'This is how much your application's taking on AI,'" because now I know the token costs per workload. And once you get to that, you can move to the whole idea of tokens for the output, because then you can start to mix in the business outcomes part of it. What am I actually selling? What am I generating? And that's when the sales and finance teams come in. So that's the arc I would take. If I just made the statement that PUE is dead, I would say no, but here are the metrics that are built off it. So Val, is there anything you see where it's, "I got you, Vernon. There's a big, massive hole in that framework"? ### [19:21] Why Neoclouds Can't Build for Inference **Val:** No, again, there's... **Vernon:** Sorry, trying to do Serena's job. Sorry. **Serena:** That's okay. I like it. **Val:** Instead of a hole, these are all important layers that we're adding together. What I'm seeing is that we need these kinds of metrics almost as financial instruments, because one of the problems I'm seeing right now is what I call the golden handcuffs of the training revenue in this industry. Financially, most investors are much more comfortable investing in a stable three-year GPU offtake contract. So what are data center providers building? What are the GPU providers and neoclouds building? Very training-optimized AI factories that are not inference-optimized. It's fundamentally a different reference architecture, a different network design. So we're really hamstringing the opportunity to do AI more efficiently, given that the majority of AI is inference right now. We need to create metrics, such as the ones we're discussing here, for financial instruments that let investors underwrite inference-optimized clouds. That's not happening right now. I deal with most of the top neoclouds around the world on a weekly basis, sometimes daily, and they all literally can't implement the new inference-optimized reference architectures because they've sold out their capacity to training GPU offtakers. So aligning these financial incentives better, via better metrics and better instruments, is critical right now. Otherwise, we're going to be in this uncomfortable, maybe even trough-of-disillusionment phase for a while, because we're not getting anywhere near the level of token, session and task-based usage efficiency that's possible. **Serena:** That's a great point, and I think the connection back to workload is something I see missing in a lot of conversations right now. We are measuring at the beginning of the arc that you mentioned, Vernon, frankly because it's easy. We know how to do that, and the dashboards are ready. When we want to talk about workload, oh my gosh, suddenly you're talking about which apps, which teams, and we don't have that ready. Now someone has to go do engineering work just to have that metric. ### [21:51] Where to Start Measuring AI Workloads **Serena:** And no one loves to do that. So I'm curious, Vernon, if you have something concrete for companies that do want to do this work. How do they even get started? If I'm starting from zero, and I was just measuring the easy stuff, how do I go from that to something more intelligent, like workload? **Vernon:** That's a very good question. Let me use an analogy, then I'll get to how it overlaps into workloads. I once did a project for a very large beverage company that makes a cola product. They wanted to move their workloads into the cloud, and they just thought they would lift and shift everything and hope that everything would work. My argument there was, "Have you done any prioritization of the workloads at all? Have you looked at which applications are right for it, fit for it?" And then go down the path of saying, "Let's put some smarts around the workloads and applications that will give you the most benefit first." You would be really surprised at how few people have done a rank scaling of applications based on consumption, whether it's hardware, software or what have you. The same thing has to happen again with the AI workload. You want to have a rough handle on which AI applications are consuming the most. And the important part here, Val talked about training and inference: what's important to you? Is it better to be well done on the training piece and be able to easily collect the information? The difference between AI workloads and traditional workloads is that traditional workloads were bounded. They were generally stuck in their own world, whereas AI workloads are like crabgrass. That's the only way I can describe it. They go out to everything, and they can impact a lot more areas that you didn't have to worry about before when you were trying to measure a particular application or workload. Because what the AI is trying to do, through agents and prompts and even multi-agent systems, is going to really complicate that sizing effort for you. So my take on it, and it sounds like motherhood and apple pie, is that you have to just pick one or two to get your hands around how to measure the token count. But be prepared for the complexity curve to really ramp up, because that's one of the things I see. The second part is the whole governance model for, I'll call it, AI workloads. There has to be a governance agenda built around who can write the application, who can run the application, and who sees the output of the application. And under AI rules, I think that all changes again, for all the right reasons. Having a financial services background at Fidelity and State Street, I know a lot about auditing. If you're a nickel short, somebody wants to know why, because a lot of nickels over time make a lot of money. It's the same thing here. I don't see enough governance being brought in. And the reason I bring it up, Serena, is that it's all part of that workload measurement where, as I say, the complexity curve for AI is going to be a really difficult row to hoe. And I think you said it earlier: it's hard work. And to be honest with you, Val may appreciate this, but not many people do: I look at myself as a plumber. It's a pretty boring job, but God help you when you need a plumber. You want a good one, you don't want a bad one. And I see people looking at all the stuff we might be talking about today as, "Oh, that's really boring plumbing work." But it isn't, because it's all to do with the financial output at the end of the day. That's where I think, when you're starting with AI workloads, you need procurement, you need finance, you need security, you need ops, you need the business all to be there to understand what that AI workload looks like, as opposed to just having a capacity planner, which is how we used to do it in the banking industry. The capacity planner would simply say, "Oh, file and print. I can measure the number of pages printed, the number of printers," and so on. It was pretty easy. And I'm showing my age. ### [27:22] Getting Finance, Procurement, and Ops to the Table **Serena:** I like the focus on the interdisciplinary team that you just mentioned, because I don't see that happening in most companies. It's a very small team having those conversations, or they have separate conversations. They don't talk to each other, and they really need to. Even if they don't speak the same language, they should be at the table, whether it's finance, procurement, tech... Actually, I would say the people side should be there as well, because there's a lot of employee impact too. I think measuring workloads is hard, and that's getting in the way for a lot of organizations. I'm hopeful that from today's conversation, our listeners see this is not boring and tedious work. It's meaningful, and it will get us to that outcome, that intelligence at the end, which is what CEOs and the board care about. When they talk about the ROI of huge AI investments, that's what they want to see. **Vernon:** For my sins, I was one of the industry analysts who said the IoT market was going to be many trillions of dollars, many billions of sensors. I can defend it, but that's water under the bridge. What I think you scratched the surface on there, Serena, is that IoT has actually helped bridge that gap. For example, with AI in manufacturing, we've seen this conversation before between OT operations people and IT people, who did not want to talk and who think differently, even though they're looking at the same problem. That's very loosely put. But I'm hopeful that the scars from IoT implementations in the early days for CIOs can help this discussion of bringing together people who talk different languages but all have the same goal. That small benefit from IoT is where I'm hopeful. ### [29:38] Trillions of Tokens and the AI Payback Question **Val:** I love the penny, the five-cent auditing angle, because we're talking about volumes right now of quadrillions of tokens per day being inferenced that we know of. The ones that are easier to measure transparently, for example on OpenRouter: the top models individually are at 15 to 20 trillion tokens per day. Any individual inference provider, whether it's the model hosts themselves or third-party inference providers, has a healthy chunk of that. I know many of the top inference providers, per top model and customer, are doing over five trillion tokens a day per customer. So these pennies, these nickels, really add up. If you don't get the plumbing right, you get the very legitimate questions a lot of Wall Street analysts are asking right now: is this enormous AI CapEx, a trillion-plus right now, ever going to be paid back? Is there ever a revenue model that pays back these unprecedented capital investments? Because we're not unifying a lot of these groups, because we're not connecting the dots well enough or directly enough, these remain open questions. We're not going to boil the ocean here, but the more we can educate the market, the more we can get different functions and teams collaborating and coordinating on transparent metrics, on observability, on understanding where the trillions of pennies are disappearing (quadrillions, actually, today), the more that will make all the difference. And at the very least from a numerical perspective, that will answer a lot of legitimate financial questions, which affect government policy and so forth, about whether this is all really worth it. It's a disturbingly open question right now, and it can be a lot less vague. **Vernon:** But how long can I keep going without somebody saying, "Stop. I want to see my ROI on my trillion-dollar data center investment"? ### [31:54] Bubble Wrap, Not a Bubble **Val:** Everyone's watching for the bubble to burst, and I always like to say there's never going to be a bubble bursting here, because if nothing else, the unprecedented growth of demand, which is still unending right now, shows that there's some kind of consumer backstop. People, organizations, companies want this. The question, of course, is what are they willing to pay and what does it cost? So instead of a bubble, I like to say we're dealing with bubble wrap. There are going to be some bubbles that will pop. We'll pop some ourselves. But there isn't one giant bubble that's going to burst here. There are just some bad businesses, some bad services, maybe bad model providers and infrastructure providers that are not sustainable, and many others that will be. And the Uber and Airbnb analogies always apply. The use cases we haven't even dreamed of are the ones that are going to make the most money in the future. ### [32:44] What Outcomes per Token Really Means **Serena:** All right, let's move on to a couple of other important topics before we wrap today. One is the metric that you land on, Vernon. Let's talk about the magical KPI called outcomes per token. I think every CEO would love to know: is there a magic wand I can wave and just have this pop up on my dashboard, so I can see trend lines? What does outcomes per token really mean in the business world? Where does it come from, and how can someone get started? **Vernon:** Good question. I think there's an opportunity here for a startup to come up with that metric. But let me describe it a little more. Let me put some clarity around it. If we follow that stack up and get to how many tokens it took in that workload, we can translate that to a product. I mentioned AI commerce earlier. So on one hand, call it the left-hand side, we know how much it cost in terms of AI to get to that output product. And on the other hand, we can measure the AI journey that commerce took to fulfill either buying or selling that product. I'm being very commercial here. I've gone away from the enterprise view inside the CIO world to the SVP of sales, who would stand back and say, "Look, this is how much we took in using AI, as a prompt or an agent or multiple agents, that steered somebody on a website, for example, through to completing the purchase of a product because of AI." So on one hand, you've got the cost, and on the other hand, you've got the revenue. That's where the output really starts to come in, because then that dashboard would hopefully be able to show somebody the tokens used to create it and the tokens used to sell it. And at that point, I think, nirvana. Then we might be able to stand back and say, "Was this worth it? Was it worth all this on the left-hand side to get this on the right-hand side?" **Serena:** If I could push you a little bit, Vernon. It sounds like you want us to connect it to customer outcomes, something that ultimately matters for customers. Does that mean the outcomes per token metric wouldn't matter for back-office functions, I'll call them? The finance, IT, HR and marketing of the world. **Vernon:** No. It would be almost the same, in the sense that you'd be able to measure how many tokens per person have been used and how many people you have in a department, and measure it that way. So you'd have an organizational structure that lets you see, for example, is HR doing lots of AI work, looking for candidates through résumé search and then parsing and whatnot? Is it worth it? Is that worth doing under AI? The same goes for any of the back office, and finance as well. It is very possible. I hate to say it. I don't want to say it's like an ERP project, because then you'll cut me off. But I think it's a project for life, if we have to go down that path. On a serious note, I think it's a restructuring of how you organize your information flow across the company. We've tried for years and years to have a common framework. And Val, this is where you'll say, "Yes, Vernon. Finally, in the last two minutes, we're talking about storage platforms." At that point, you have to have the ability to see transparently across the company. We're trying really hard to get there, but it's a CIO nirvana to be able to get there. So we talk about the outside, the commerce piece, but it's very much applicable inside as well, as long as we measure and tweak the data flow. ### [37:34] Saturated Benchmarks and the Search for Better Evals **Serena:** That's very helpful. Thank you. Val, what would you add to that? **Val:** My mind usually goes to the very provocative Alex Karp at Palantir, because they've had a very good track record of taking a risk, getting into an organization's actual processes and workflows, capturing what's in people's heads, which is often how work gets done, and creating context graphs and decision traces. I think the brand Palantir uses is the ontology. Once you have that, you're able to actually benchmark what your company does, what your divisions, departments and individuals do. And when you get to that level of maturity, which is rare today but will become more and more commonplace, you can actually measure it. You can have agents take action, take correct action. With these workflows, you know what good and bad are, what a successful outcome is, and, as importantly or more importantly, what a failed outcome is, so you don't repeat it. Those are the kinds of things we need to do. I don't know if they're startups or consider themselves that, but there's Artificial Analysis, which is measuring more and more task-based things, or domain-specific benchmarks for legal, marketing, sales and manufacturing. There's a startup called Arena AI, which is measuring more and more high-level tasks and work. There's one I like because of the name, the Vals AI index, which is all about evals. Vals for short: evaluations. The whole point is that we know the industry now, especially the frontier labs and even the open-weights models, does too much of what we call benchmaxxing. These benchmarks are saturated. Take MMLU, the Massive Multitask Language Understanding benchmark. It was really useful about a year, a year and a half ago, when the models couldn't saturate it. Today every model saturates it, so it's literally a meaningless benchmark. GPQA, the graduate-level Google-proof Q&A, is also very heavily saturated. So we're always on the lookout, and that's why I think there's an opportunity for startups here. We're always on the lookout for better evaluations, probably more domain-specific custom evaluations, more real-world evaluations that are either hard to saturate or will at least have a good half-life and provide value before they're inevitably saturated. That's really important going forward. And the good news is that, because nature hates a vacuum, there are definitely some smart researchers and scientists working on this. There's a controversial one right now that's trying to audit all these rogue agents that are escaping their sandboxes and seemingly attacking everyone. METR, in Berkeley. That's another one trying to evaluate how these agents are working and, to your point, Vernon, how they're breaking whatever governance there was. Maybe there wasn't any governance, so it wasn't really known that they were misbehaving. We're at the forefront. We're at the frontier of this right now. We can call it the jagged, or as I like to call it, the bleeding edge of some of these technology advances, and we need to rein them in. ### [40:59] Where We'll Be in Six Months **Serena:** I wish we had more time to talk about governance. Maybe that's the follow-up episode, because it's so important. As we get close to the end today, I'm curious: if we were to meet again in six months, the three of us, what are you hoping the conversation will be like between us? What do you hope we'll be talking about that's different from now? **Val:** Humans are bad, myself included, at predicting exponential progress correctly. Even the best in the world, the ones most deeply involved in this exact research, continue to be bad at predicting exponential progress. Look at the models right now. We're at Opus 5.5 this week, which is getting rave reviews for its efficiency and intelligence. We're at GPT-6 Astra. We're at these really cost-effective Chinese open-weight models that are progressing exponentially. Some people don't like this term, but they're progressing at machine speed, without a lot of humans in the loop, without a lot of human intervention. We're at the end of September, so by Q4, by Christmastime, we're going to have models whose power I don't think we can appreciate. So I hope that in six months we actually reach the next step function of governance and AI maturity overall, because we have to get much better at understanding what these models do. We have to get much better at controlling the negative externalities as well as the positive outcomes we all want. I'm not catastrophizing at all. I always look at it from the builder's perspective. We are at an inflection point right now where these models are getting supremely powerful, and we can control them. This is just like adding seat belts and airbags and anti-lock brakes to cars. Of course, we're building these models. We're calling them AI factories. We're manufacturing intelligence. It's not artificial at all. It's manufactured intelligence. We can control it. So it comes down to the incentives for creating observable, governable, safe AI. All we have to do is align those incentives, because it's not beyond our grasp. We're builders. We're building these things. What are we incentivized to build? So what I hope in six months is that we've put in at least the beginnings of the right incentives, and we're controlling these superintelligent beings that we've manufactured, and manufacturing them correctly. **Serena:** Love that. Okay, Vernon. **Vernon:** I'll be really quick. I think what's going to happen is that the economic climate is going to put so much pressure on companies to make a decision about AI that they won't have done their homework, and they're just going to go down the road and hope to heck that nothing falls off between now and then. Not the trough of disillusionment. I think the line of stupidity is going to creep in, where people make stupid decisions because they're under economic pressure. Some of our clients are seeing the pressure to use AI to reduce costs, and I don't know if that's the right business model to look at as the first point of reference. As long as they keep going down that path, they're going to make the same mistakes we've seen so far. Hopefully, like all teenagers, they'll come out of that dark hole safe at the end. Having three adult children, I know that really well. So I really think in six months' time we're still going to be in this turmoil of panic about what to do with AI, and how fast they can make a decision that's not well thought out. That's where I think we'll be in six months. **Serena:** All right. I think we should regroup in six months and see where we are, see whose predictions come true. Even though humans are not good at it, we do try. I hope this episode brings everyone something new to consider about measurement. Stay tuned. Thank you both so much for joining us today, Vernon and Val. Thank you so much for your time. **Val:** Thank you. Pleasure, as always. Thanks. **Serena:** Thanks for joining us on Deep Geeks, and thank you to Vernon and Val. The takeaway: start with one or two AI workloads, measure what they cost and what they produce, and bring finance, procurement, security and ops in from the start. That's how you get outcomes per token. Find Deep Geeks on YouTube, Spotify, Apple Podcasts, or wherever you get your podcasts. Until next time. ## Related Podcasts - [One Size Fits None. What Retail Reveals About AI.](/podcast/one-size-fits-none-what-retail-reveals-about-ai) - [Nobody Has Done This Job](/podcast/nobody-has-done-this-job) - [Why Investors Today Count on Kilowatts](/podcast/why-investors-today-count-on-kilowatts)