What It Actually Costs to Run Intelligence at Scale

Anand Iyer: Well, thank you everyone for being here. Thank you for joining us for this panel. My name is Anand. I’m the founder and managing partner of a firm called Canonical, based in San Francisco. Lucky and privileged to be here with these amazing folks. Maybe we’ll just go around real quick — would love to hear who you are, what you do, and maybe some context on how you got here. Evan, you want to kick it off?

Evan Conrad: Hey, I’m Evan. I run a company called San Francisco Compute. We build data centers, we build GPU clusters, we operate them, and then we sell them on an order book. We do this for other people, meaning we partner with people at each point. So if you are looking for a partner that will do GPU as a service with you, if that is us, please come find me.

Carmen Li: Hi, my name is Carmen Li, CEO for Silicon Data as well as Compute Exchange. Silicon Data is the market intelligence layer for the GPU compute ecosystem. We’re known for our indices on Bloomberg, Refinitiv, and a CME contract, which we’ll talk about later. I'm also CEO of Compute, a GPU marketplace for reserve contracts, forward contracts, and refurbished GPUs. I’m glad to be home — I was in Singapore for middle school and high school, so it’s lovely to be back.

Val Bercovici: I’m Val Bercovici, in charge of AI product strategy and also in charge of not having to convince people anymore. That was last year’s job: Getting people to use AI internally at the company and optimize it. Before that, I was CTO at NetApp, and I also helped Google open-source Kubernetes as a board member of the Cloud Native Computing Foundation (CNCF) under the Linux Foundation.

Anand: That’s cool. Very cool. All right, we were joking backstage that we’re going to switch this panel up and talk about how Fable is going to eat all our jobs, but maybe we’ll stick with the talk track for this one.

Anand: So to kick it off, this panel is about the cost of intelligence. I was wondering if you can break that down for us, maybe a high-altitude view. What does “cost of intelligence” mean to each of you? Evan, maybe you can kick it off from your vantage point. How would you define that?

Evan: Are you asking what goes into the cost of intelligence?

Anand: I’m asking what goes into the cost of intelligence.

Evan: So the price of an on-demand GPU — one where you have a guarantee that you will get that on-demand node — can be $10 an hour, or upwards of $10 an hour. And the price of that same GPU sold as preemptible, sold as “you’re not sure if you’re going to get it,” can be 50 cents. So I would say that the majority of the cost of a GPU hour is risk. Beyond that, there’s all sorts of other things we can optimize toward, but mostly it’s risk.

Anand: Okay. Yeah.

Carmen: So, I definitely echo that. It depends on the commercial contract — the cost can be very different regionally, and different GPUs, even the same H100 family or B200 family, can cost differently depending on the CPU, the RAM, and other spec differences. Right now, hyperscalers obviously charge a premium over the clouds overall — probably 2x to 3x on-demand instances around the world. But we see GPU costs not only being high, but also volatile. High is one thing — if it doesn’t move, I guess that can actually be a good thing, since you know what you’re going to pay the next hour. The problem is it moves all the time.

The daily volatility we observe for GPU on-demand pricing around the world is probably between 20% and 30%. So if you rely 100% on on-demand GPU instances, your margin can have a huge swing depending on how much you use GPUs. And when people mention cost of intelligence, some will also think about token cost — open source, open weights versus closed models, depending on who’s providing the inference service. And that can also be a big range, based on input/output tokens, how you’re caching them, batch orders, and a bunch of other things. It varies a lot.

Our token index is currently tracking around $180 per million tokens as a combination of pricing and usage-based weights on Bloomberg.

Anand: Got it. So we got this point of view from the GPU perspective and I think you leveled it up to tokens, which is also helpful framing. Val, what do you think? How should we define this?

Val: I love the question, because we work with about 70% of the world’s neoclouds and GPU providers, including the hyperscalers, and this cost of intelligence really breaks down very tangibly. As Evan and Carmen were saying, there’s definitely a marketplace for GPU hours — reserved, spot, and so on. But we also provide a lot of storage solutions to those customers, and memory solutions, memory software, which we’ll talk about later if there’s interest. And that’s very much a token-cost story.

I spent a lot of time studying OpenRouter, looking at all the top models and providers, and inspecting price per million input tokens and price per million output tokens. Last year that was a really important set of metrics — almost irrelevant this year. Cache-read tokens are now something like 80% of what agents consume, and it’s a wild marketplace in terms of cache-token pricing. But the general trend is that for every 100x reduction in unit cost of inference and unit token cost, we’re seeing something like a 10,000x increase in demand and consumption of those tokens, particularly cache-read tokens.

So the cost of intelligence, I hate to say it — tokenmaxxing was a thing for a while — has effectively gone up about 100x in the past year. That’s why, over the last week or two, we’re seeing a lot of CIOs and CFOs come to the table and say, “No, we’ve got to cap these out-of-control token costs. We’ve got to be wiser. We’ve got to start model routing. We’ve got to write smarter agent harnesses that realize budgets are finite, and make this work within those constraints.”

Anand: Yeah, sounds like a great plug for Jevons paradox, right? Exactly what you defined.

Anand: Interesting. I want to stay on memory for a second and zoom out. Help us understand memory and the constraints that exist today. What are the challenges we’re seeing? I think from a high level we understand the buzziness around memory, but would love to hear how you’d put it in terms of where the constraints are today, where the puck is going, and what we should expect in this industry over the next 12 months.

Val: Yeah, it’s a really important topic. There are actually two big sides to the memory coin. If you’re an AI engineer writing harnesses or writing AI apps, memory to you is context, it’s state. It’s really what supersedes the old idea of prompting and prompt engineering. It’s often markdown files, SQLite databases, context graphs.

To infrastructure operators, to supply-chain people, memory is literally the high-bandwidth memory (HBM) on the GPU package, the DDR5 on the motherboards, the NVMe extensions into storage — that’s the memory we play in. And you have this really unfortunate atomic packaging of compute and memory right now that you can’t really decouple.

So the reason we have these rate limits, the reason pricing is spiking right now, is that when you need — for an agent swarm, say, running a slash-go command for hours or days — when you need more of that memory reprocessed, you’re paying the compute tax over and over again. You’re prefilling the KV cache repeatedly because you have to evict slightly stale KV data as new prompts and a multi-turn agent come along. What if you didn’t have to evict? What if your problem went from KV cache evictions to KV cache garbage collection after days or weeks?

That’s the revolution we’ve introduced into the marketplace. I was a bit ahead of the curve when we brought this product out about a year ago, because Claude Code wasn’t a thing yet, Manus wasn’t a thing yet. Now we’re in the thick of it — the market’s white hot. So both sides of the memory coin are very topical today.

Anand: One more question and then we’ll come back to these folks. Break it down for us in terms of the workflow: How does the usage of memory change from training a model or fine-tuning it versus inferencing? I think you referenced agentic use cases and inferencing, but how does that look? How does that change?

Val: So training, historically, has not been memory-bound; it’s been very compute-bound. But the notion of pre-training is really evolving right now. Reinforcement learning is really the heart of a lot of general pre-training now, not just narrowly specified pre-training, and reinforcement learning is thousands of little loops, including inference in the middle and on the critical path of that.

Memory is becoming a big part of that. Batch processing, batch inference are becoming a really big part of pre-training and what some people might call post-training right now. So I’d say it’s still largely storage-centric: We still have to pull data sets out of storage, load models into memory from storage. Checkpointing is really relevant in terms of persisting all of the loss function and gradient descent into something useful that may or may not become an actual model.

Inference is fundamentally different, and there’s no better example than Google. At Google I/O this year, they said, “You know what, we’re splitting the whole TPU product line” — a traditional TPU8 for training, and a TPU8i for inference — because the balance between compute and memory in a TPU is fundamentally different for the two. NVIDIA acquired Groq for exactly that reason. Inference architecture isn’t just a GPU anymore; you need acceleration for feed-forward networks, you need static RAM to accelerate output tokens per second. Why was the Cerebras IPO so hot? Because it’s an inference processor, not really a training processor anymore, and that market is a fundamentally different supply chain and infrastructure.

Anand: Yeah. Cool. Thank you.

Anand: Compute, data, algorithms — maybe we’ll come to compute, Evan. You’ve been in this for quite some time, awesome background with San Francisco Compute. Tell us a little about the evolution of compute: Is it trending toward becoming a commodity?

Evan: I mean, it depends what you mean by “commodity.” The variability in price depending on where you get compute from changes quite significantly — different providers are not the same. However, if you hold everything constant, most people care about price assuming reliability and security are the same, you’re in the same type of chip, same type of potential services around it. Once you hit that table-stakes layer, then people care about price, because people are price-sensitive in a way that they weren’t in web software. In web software, almost the defining feature is that you have really high margins, so your cloud cost doesn’t matter as much as it does in AI. So in that sense, yes. People are price-sensitive and are mostly comparing on cost.

Carmen: I want to push back if I may. I think it’s a chicken-and-egg problem. You might not know you’re getting a different thing. If you think, “Hey, there’s no way to fully verify the server was that server,” your only comparison is specs and cost — you have to trust the specs, you have to trust the cost.

But if we can surface the discrepancies or the performance variance early on, when you make a decision or a purchase — by testing, whether that’s DGX, MLPerf, whatever — you might find, “This cluster performs one standard deviation better on FLOPS, memory bandwidth, or token throughput than the other servers.” Even the same chips — you might lean toward stability or other factors and want to lean toward the pricier option initially.

So by surfacing that benchmarking, the client might have a different opinion. You’re saying that if you surface all the details, people will price in the premium. They might say, “I’m willing to pay 20 more cents on the server because I trust the SLAs, I trust the stability, I know the token throughput is higher, the latency is better.”

Evan: I think I agree with that — that’s basically what I’m saying, that if you hold things constant, if you hold reliability constant, hold the SLA constant, hold security constant, hold the services around it constant, then functionally people are competing on price. That’s what we see, people competing on price.

Carmen: Yeah, I think it’s hard to have two very identical things. What we see is that the newer architectures actually have more consistent output than older architectures. Especially among the A100 and H100 families, performance can vary up to 38%, split between intra-provider, inter-provider, and the chips themselves — we published a paper on that, which is fully public, everyone can search for it. So we thought that’s fascinating because you’d think that if the market is efficient, people would trade the better-performing server at a higher price rather than the worse ones. But it’s not 100% reflected because no one really knows unless you test it.

Evan: Yeah, we’ve seen pretty big variability between different clusters. Even things like, if in an H100 cycle you got the early H100 chips, those were maybe not the greatest. Or those specialized A100s for hybrid accelerators, they can have a lot of problems too. The way we’ve handled this typically is we’ve just exposed this to customers, which is similar to what you all are doing — I think that’s the right mentality.

But assuming all your variance is the same, then you are indeed just competing on price. I also think it depends on the particular type of customer. For some people — I don’t know if this is the best way to describe it — but they’re like goats, as in the animal. They’ll eat whatever’s available. In today’s environment, they’ll literally take whatever is available.

Anand: What kind of customer is that? How would you define that?

Evan: I think the biggest cases are where you don’t care about latency, you don’t have particular security requirements. If you’re willing to take preemptible, that means you’ve designed your system such that anything can blow up at any point. If you’re willing to take something that can blow up at any point, you might be willing to handle reliability that’s not the same, if the pricing is better for you.

So inference that doesn’t care about latency can do this. Running agents on open-source models is a non-terrible use case, as long as they’re not responding immediately to a customer. If they’re running a background process and need to complete a large volume but don’t necessarily need it right now, that’s a good example.

Anand: Switching gears. Carmen, you had this partnership with CME recently around futures. Tell us about the genesis and evolution of that, and what it’s led to so far.

Carmen: Yeah, it’s quite exciting. We really believe compute will be the largest human resource in the future, surpassing all energy combined — oil, natural gas, and electricity. If that premise is true, which we’ve sort of staked our position on, we’re globally spending $600 billion USD every year on compute. In the spot markets — derivative markets are usually 10x to 15x spot — you’re looking at potentially $6 trillion to $8 trillion in derivatives markets, which is gigantic.

We really believe that as financing progresses, the market needs benchmarking services, indices, basically a mechanism for people to hedge. You can argue, “Why don’t we just do longer-term contracts?” — Compute does that, Compute Exchange does that — but at the same time, look at oil markets: I’m sure OPEC and Shell do long-term contracts every single day, but that doesn’t mean they don’t also use WTI Brent, which lets them hedge futures any time during trading hours.

It’s more efficient, more transparent, more effective, and you don’t have to be in Manhattan or Singapore to hedge — you and I can hedge if we want to. That provides equal access to the whole market-participant ecosystem.

Anand: We’ll stay on this one more minute. Evan, you mentioned earlier that part of the value prop San Francisco Compute brings is that any excess compute a customer has can be resold through this order book. How does what you do work together with Carmen’s side? Do you rely on their price index? Do you price it yourself? How do you folks work together, if at all?

Carmen: You can correct me if I’m wrong, but the analogy I’d use is NASDAQ and NYSE. They’re trading stocks: You can buy stocks from those exchanges, or buy them from banks, Bank of America, brokers. Compute doesn’t have any stocks to sell you because we don’t have underlying resources. What we’re like is Bloomberg: You can see the indices, the benchmarking, the fair-market-value evaluation of any securities. So it’s a bit of a different ecosystem: One is the actual transaction layer, and another is a data and financial-products layer, which is supposed to normalize and aggregate all the information from millions of pieces of compute globally, across Compute and other platforms. What do you think?

Evan: We run the clusters. We also run an order book, and the value proposition we bring to our customers who buy from us is: If you bought a three-year contract on Blackwell, for example, and said, “That was a bad idea,” a few days later you can completely sell that contract to somebody else. This means you don’t take on a tremendous amount of risk.

The way we functionally do that is we built a continuous limit order book under the hood, and we built our orchestration layer around that order book. What this means in context here is we have a great index, and we also have a way to price compute into the future — a forward curve — and then we do physical delivery. We’re actually the people who service the compute for you, all the way down to the dirt. Right now, we’re doing a multi-gigawatt data center project in the United States, and we can do it at really big scale. I think the way we’d work together is the same way any neocloud might work with a compute exchange: It’s a way to price risk, not just for our customers, but for ourselves and for other folks.

Carmen: People just assume it’s actually the compute AI ecosystem. I think it’s very interesting. People just assume you compete with everybody. But actually people collaborate a lot more than everyone thinks.

Evan: Yeah, I think people realize this because it’s a very integrated ecosystem. When we started, we were an AI lab that had bought a GPU cluster, and we had bought it from one neocloud who had bought it from another neocloud, who had bought it from another neocloud. The note here is that behind the scenes, most of the industry, especially in compute, is much more interested in partnering than people might otherwise assume. It’s kind of a very weird industry relative to a lot of other spaces because of that, but also a cool one.

Carmen: So if fusion takes off, we’re supposed to have a lot more capital, have more efficient allocation, and more capital will feel comfortable coming into the industry because they can hedge, have other underwriters for volatility. So it’s supposed to help everybody, from hyperscale to neocloud to customers, you and me. That’s really the whole goal. So it’s a lot of collaboration across the ecosystem.

Anand: Cool. Understandable. It’s like inception, but for GPUs.

Anand: Val, coming back to you. We were chatting last night, and you alluded to the fact that the era of subsidized intelligence might be coming to an end this year. I don’t want to put words in your mouth, but I think that’s what you said. Tell us a little about that: Where do you think the trend line is going? Is the cost of intelligence going to go up, because are Frontier Labs just trying to buy traffic right now?

Val: Cliché as it may be, it’s unquestionably like the Uber story all over again. Lots of subsidized rides early on — I don’t know how many years ago, 10 years ago — we all got addicted to the service. Then market reality set in. I think investors reached their 10-year impatience window and wanted to see real returns on capital, and sure enough, surge pricing became a thing. But surge pricing is really just another term for actual market supply-and-demand pricing.

We’re very rapidly entering that phase for tokens right now, where all these $20 subscription plans — which interestingly became $100 and $200 max subscription plans — are getting more and more restricted in terms of what you can do with them. They’re becoming very homogeneous. The Anthropic example has taught us that a lot, relative to OpenClaw and things like that.

But there’s no question that the hard realities of energy cost, the hard realities of data-center-buildout latency not matching surging demand mean we’re paying more and more actual market pricing for tokens, not subsidized pricing. That’s why we’ve seen the headlines just this past week — The Wall Street Journal, CNBC — that tokenmaxxing is dead, token capping is in. That’s a perfect reflection of the fact that budgets are real now, and we’ve got to work within actual constraints.

Right now the subsidization era for hobbyists is over. I think Anthropic pretty much said there’s no such thing as unlimited now. You get $200 of credits, you don’t get $200 of unlimited usage for a month; there are rate limits and all.

Anand: I don’t want to say push back, but the question I’d have is: Uber is supply-constrained, right? Hence surge pricing. Our market was constrained and driven based on how much supply you had, which is the number of drivers. Same could be said for Airbnb and homes. This feels naturally different, very software-driven. We’ve got open source, open weights, open data emerging. How does that change the equation for you in terms of cost of intelligence?

Val: The ultimate constraint is energy. Energy is very finite, particularly here — well, all around the world. Germany isn’t having a great time. It’s finite on Earth, I guess that’s the point. There’s only so many FLOPS, and so many megabytes or terabytes per second you can move per unit of energy, per watt. That’s the ultimate constraint, and whether you’re able to manufacture more processors and accelerators, whether you’re able to manufacture more memory, or use software to apply that memory much more intelligently and efficiently — we have real, hard constraints here. I think the Uber analogy holds really well: We are supply-constrained.

Evan: Can I make a comment? I sort of agree that the constraint is energy, but I would suspect that the constraint behind that is credit. If you want to get a project off the ground, you’ve got a series of energy problems to solve. Can you get grid access? In some places you cannot easily get grid access; somebody needs to build a substation and do a study. Can you get behind the meter? Sorry, all the turbines are bought out. Can you do RECs (renewable energy certificates)? Sorry, all the RECs are bought out. Can you do a fuel cell? Maybe. Is there a nuclear power plant somewhere? Cool.

At each point, there are lots of ways to build energy, but to get it off the ground, you need somebody to finance the project at a reasonable enough scale that you can actually build your new power plant. The concern is that the hyperscalers are almost tapped out in terms of how much they’re spending. Your Frontier Labs aren’t considered investment grade — that’s when Moody’s gives you a really high rating, kind of like having a really high credit score — and then everyone who isn’t a Frontier Lab has basically no chance of ever becoming investment grade, to buy compute at the scale you’d need to build a new power plant. If you’re going to build a power plant next to it, you need a scale that’s big enough.

So you end up with a big question, which I think will be the constraint of the next few years: How is everyone else who isn’t a Frontier Lab going to get enough credit to actually buy compute at scale, such that you can then build the energy?

Val: Ultimately, though, people are going to get ahead of others in line through market pricing.

Anand: Carmen, I want to come back to something you said about the energy constraint on Earth — give us 45 seconds on that, and then I want to move to sovereign AI.

Carmen: Number one, energy is definitely a must-have for any GPU. Even though it’s a must-have, it’s actually a really small component of GPU cost — we’re looking at probably 10% to 15% operational cost for the energy piece, which translates to: I need air to breathe, but air doesn’t cost me much money if I know I’m not going to be dead in the next minute. So energy is important, but it’s not driving the price.

The second thing — Silicon Data is referenced in SpaceX’s S-1 filing; we were referenced a few times for price indices and price forecasting for SpaceX. I don’t know anything about building data centers in space, but it’s always exciting to see there are potential alternative futures for humanity as a whole.

Anand: Yeah. What’s your take on orbital data centers?

Evan: There are many countries in the world who don’t currently have their energy assets being used toward data centers, and I’d suspect before they put data centers in space, they might put them somewhere else — in Texas, for instance. The world has figured out they should land a data center in Texas, and a few other places in the United States. I think they’re going to go around the world next.

An interesting question is: There are a lot of countries with reasonable power assets but that are landlocked, so they don’t have fiber access — they can’t hook up to some undersea cable. So there are probably interesting geopolitical trade deals where a country with access to the undersea cable digs a fiber line through to a country with access to power, so it can reasonably deploy a data center.

But I suspect that’s more likely to happen than in space, though I could be totally wrong. I do think there are going to be data centers in space eventually, but it’ll be for low-latency access to robots in space — that’s the first use case. Adjacency makes sense.

Anand: I know we’re coming up on time, but I want to hear both your thoughts on sovereign AI — keep it tight, but I want to see how it challenges any of the things we’ve discussed so far. Val, you want to start?

Val: I mean, the simplest context is sovereign AI is literally not an issue in the United States, and it’s an issue everywhere else outside the United States.

Anand: So far, yeah.

Val: Yeah — all the topics we talked about, finite energy is a real issue in most of these sovereign environments. I’d add the element of cybersecurity — nation-state cybersecurity is really fundamental. I believe in this concept of “gross token product,” as opposed to just gross domestic product, as a reflection of an economy going forward. Sovereign AI is just a key catalyst for all of that.

Anand: Amazing. All right, I know we’re coming up here, but I want to get a rapid fire. You mentioned you’ve been here before, June 2025 — it was a different world back then, and we’re here now in 2026. If we’re sitting here on this panel in 2027, what’s the hot-button item we’re discussing? Evan, you want to kick it off?

Evan: I would think it’s data centers moving around the world. I do think a sovereign AI play is the correct move for most countries. They don’t currently have offtake for that — that’s what San Francisco Compute does. We bring San Francisco offtake to countries around the world and help them deploy data centers.

Anand: Amazing. Thank you. Carmen?

Carmen: I’ll echo that. I think more data centers will come online, not only with NVIDIA chips, but probably with other design houses, too, which will be great. I also think maybe next year we’re going to talk a lot more about exporting tokens as the next export product for different countries. You can’t export electricity or energy directly, but you can export tokens as a way to generate revenue for the country.

Anand: Val, I love that concept.

Val: We’re on the cusp — as Anthropic recently published — of recursive self-improvement, models autonomously improving themselves without human intervention. We’ve started an arms race that we’re at risk of not being able to control.

So just like in the nuclear era in the 1960s, we need non-proliferation treaties among companies that are almost as big as nation-states — the Frontier Labs — and among sovereign AI nations themselves. Otherwise, there are some dark scenarios here that I didn’t care about last year. I very much care about them now.

Anand: Sure, sure. Folks, thank you so much. Can we please give these panelists a round of applause? Thank you.