# The Cost of Intelligence Isn’t One Number. It’s Seven.

**Author:** WEKA

**Published:** October 26, 2026

![Monochrome image of a central metallic modular cube surrounded by a stone, robotic arm, silicon wafer, metal sphere, cable, stacked blocks, and clear droplets.](https://cdn.sanity.io/images/ult5g8gw/production/1def5eb9eeec6f23f841da1131b2859e6660e152-1584x672.jpg)

## TL;DR

Rethink AI budgeting by accounting for seven forces shaping infrastructure costs beyond flat GPU and token prices.

- Price workload flexibility to reduce GPU risk premiums.
- Benchmark provider performance before paying more.
- Plan for memory, energy, financing, and rate limits.

Every AI roadmap eventually runs into the same problem: a bill to pay that exceeds expectations. During a [recent panel discussion](/video/what-it-actually-costs-to-run-intelligence-at-scale) at [SuperAI](https://www.linkedin.com/company/superai-conference/) in Singapore on the true costs of running intelligence at scale, leaders from [San Francisco Compute](https://www.linkedin.com/company/sfcompute/about/), [Silicon Data](https://www.linkedin.com/company/silicon-data/), and WEKA compared notes on what’s really driving GPU and token pricing today, and what AI builders should do about it. Here are seven takeaways that matter for anyone budgeting AI infrastructure this year.

## 1️⃣ Most of a GPU hour is risk, not silicon

Spoiler alert: The true cost of a GPU hour isn’t the silicon. It’s the risk. As [Evan Conrad](https://www.linkedin.com/in/evan-conrad/) of San Francisco Compute put it, an on-demand GPU with a guaranteed node can be “upwards of $10 an hour” while that same chip when sold as preemptible (e.g., there’s no guarantee you’ll keep it) “can be 50 cents.” He concludes, “The majority of the cost of a GPU hour is risk.”

[Carmen Li](https://www.linkedin.com/in/carmenrli/) of Silicon Data added that volatility compounds the problem. Her company tracks GPU on-demand pricing globally and finds swings of “between 20% and 30%” day to day. The action for buyers? If your workload can tolerate interruption, price in that flexibility because it’s worth real money.

👉 WEKA previously spoke with Carmen on a panel, [check out our coverage](https://www.linkedin.com/pulse/how-leading-organizations-rethinking-ai-infrastructure-measurement-zu25c/?trackingId=OjdQtjP%2FQRWlWxYcAfzTpw%3D%3D)

## 2️⃣ Cheaper tokens have made intelligence more expensive

For years, we’ve optimized for input and output token cost. But this no longer paints the whole picture. As WEKA’s [Val Bercovici](https://www.linkedin.com/in/valentinbercovici/)explained, “Cache-read tokens are now something like 80% of what agents consume, and it’s a wild marketplace in terms of cache-read token pricing.” The pattern he's seeing is, “For every 100x reduction in unit cost of inference and unit token cost, we’re seeing something like a 10,000x increase in demand.”

The net effect is “Jevons paradox”, a counterintuitive economic principle stating that as a resource becomes more efficient and affordable to use, total consumption of this resource actually rises instead of falling. The real cost of intelligence “has gone up about 100x in the past year.” This is why CFOs are now demanding model routing and smarter agent harnesses instead of open-ended token spend.

Moderator [Anand Iyer](https://www.linkedin.com/in/anandiyer/) of [Canonical](https://www.linkedin.com/company/canonical/) summed up the dynamic simply: “Yeah, sounds like a great plug for Jevons paradox, right? Exactly what you defined.”

⭐ Recommended reading: [Greg Rosalsky](https://www.linkedin.com/in/rosalsky/)’s Planet Money newsletter for [NPR](https://www.linkedin.com/company/npr/), “[_Why the AI world is suddenly obsessed with a 160-year-old economics paradox_](https://www.npr.org/sections/planet-money/2025/02/04/g-s1-46018/ai-deepseek-economics-jevons-paradox),” which traces William Stanley Jevons’ paradox from 19th-century English coal mines to today's AI buildout.

## 3️⃣ Memory, not compute, is where AI runs out of room

Compute grabs headlines, but memory is where the true constraint to more efficient AI lives. Val frames it as “two big sides of the memory coin”: context and state for AI engineers, and physical HBM, DRAM, and NVMe for infrastructure architects. Val said the core problem is, “You have this really unfortunate atomic packaging of compute and memory right now that you can’t really decouple.” Every time an agent’s KV cache goes stale, users pay a compute tax to rebuild it. His challenge to the industry: “What if you didn’t have to evict? What if your problem went from KV cache evictions to KV cache garbage collection after days or weeks?” If you’re looking to run long-lived agents, Val’s reframe of treating memory as a persistence problem rather than a cache-eviction problem is worth building into infrastructure decisions now.

The distinction between training and [inference](https://www.weka.io/learn/ai-ml/how-ai-inference-works/) matters here, too, and it matters for procurement. Val noted that “training historically has not been memory-bound; it’s been very compute-bound,” while inference has become its own discipline entirely. He said this is evidenced by [Google Cloud](https://www.linkedin.com/showcase/google-cloud/)’s recent announcement to [split its tensor processing units (TPU) line](https://thenewstack.io/google-splits-tpu-line/) into dedicated chips for training and inference, and [NVIDIA](https://www.linkedin.com/company/nvidia/)’s recent [acquisition of assets from Groq](https://www.cnbc.com/2025/12/24/nvidia-buying-ai-chip-startup-groq-for-about-20-billion-biggest-deal.html). The lesson for buyers is don’t assume one chip strategy serves both training and inference workloads in the same ways.

📚 We’ve put together resources to help you navigate these bottlenecks. Check these out:

- [_What Is KV Cache? The Hidden Engine Behind Every AI Response_](/learn/ai-ml/what-is-kv-cache/)
- [_The AI Memory Wall: Why Inference Is Running Out of Memory (and What to Do About It)_](/learn/ai-ml/ai-memory-wall/)

## 4️⃣ Compute is commoditized. Performance isn’t.

Although compute is being commoditized, performance variance is what matters. Evan argued that once reliability, security, and SLAs are held constant, “People are price-sensitive and are mostly comparing on cost.” Carmen from Silicon Data had a different take: Her team’s published research found performance across A100 and H100 fleets “can vary up to 38%” between providers and even within the same chip family — a variance most buyers never test for. Her advice is to benchmark before you buy; surfacing real performance data lets you decide if a premium is truly justified.

## 5️⃣ The subsidized era is over. Plan for rate limits.

[Val thinks so](/video/ai-inference-agent-swarms-and-token-economics-val-bercovici-at-venturebeat-ai-impact-tour): “It’s unquestionably like the Uber story all over again,” meaning years of subsidized usage by Uber to attract consumers was followed by real market pricing once investors began expecting returns. The result? “$200 max subscription plans” that are “getting more and more restricted,” and a shift from unlimited usage to more concrete budgets. His advice to anyone building on frontier models is to plan for future rate limits, and to build in real energy and unit economics now, not later.

## 6️⃣ Energy and credit decide who builds next

Two assets ultimately gate AI capacity, and neither one is silicon: energy and credit. Val called energy “[the ultimate constraint](/article/gigawatt-ai-zero-waste).” Evan argued the deeper issue is financing: “How is everyone else who isn’t a frontier lab going to get enough credit to actually buy compute at scale, such that you can then build the energy?” The gap between hyperscaler balance sheets and everyone else’s will shape who builds capacity over the next several years.

📈 Check out our [“_AI Energy Metrics Cheatsheet_”](/resources/eguide/ai-infrastructure-efficiency-metrics-explained/) that breaks down six core AI energy metrics and seven efficiency plays that separate high-performing infrastructure from expensive noise.

That gap is what the new GPU derivatives market exists to close. Carmen’s team recently launched a futures contract with CME Group, built on the notion that “compute will be the largest human resource in the future, surpassing all energy combined.” With roughly $600 billion spent annually on compute and derivatives markets typically running at 10x that, she sees a $6 trillion-plus opportunity for any participant — not only hyperscalers — to manage price risk. As Anand put it while pressing the panel on how these markets connect, “It’s like inception, but for GPUs.”

If you can’t out-balance-sheet a hyperscaler, you can hedge like one. Buyers priced out of long-term capacity commitments now have a financial instrument for the same problem.

📝 Capacity planning can be the bane of people’s existence. We created a [field guide](/resources/eguide/rules-of-ai-capacity-planning-most-teams-learn-the-hard-way/) with tested frameworks for how to navigate it.

## 7️⃣ Tokens are becoming a national resource

Outside the United States, energy and cybersecurity are shaping the sovereign AI conversation. Val’s framing is that nations should start measuring “gross token product” alongside the longstanding metric of GDP, and Carmen predicted countries will increasingly treat tokens as exportable resources because “you can export tokens as a way to generate revenue for the country.”

The panel closed on a shared warning: As models edge toward recursive self-improvement, the AI industry and its governments will need to adopt policies akin to non-proliferation frameworks. As Val put it, “There are some dark scenarios here that I didn’t care about last year and I very much care about this year.”

The throughline for AI builders is this: The cost of intelligence isn’t one number. It’s risk, it’s volatility, it’s cache economics, it’s memory architecture, it’s energy, it’s financing — with all of them moving in parallel. If you price each of these separately, you can out-compete your peers who still think in flat, per-token rates.