# The Difference Between Throughput and Goodput? Your Entire AI Budget 😀

**Author:** WEKA

**Published:** September 22, 2026

![Monochrome image of a pipe pouring glowing capsules into a "WASTE" box, connected by a glowing line to an "AI" chip.](https://cdn.sanity.io/images/ult5g8gw/production/a0f4f6632700207f58c4be864b0d9d53d7d50847-1584x672.jpg)

## TL;DR

Measure useful AI output instead of raw token activity to control costs and improve infrastructure efficiency.

- Prioritize goodput that meets quality and latency targets.
- Track tokens per watt and energy per workload.
- Decouple memory from compute to reduce GPU waste.

Sundar Pichai stood on stage at [Google I/O](https://blog.google/innovation-and-ai/sundar-pichai-io-2026/) last month and [basically confessed](https://www.theregister.com/ai-ml/2026/05/19/google-touts-tokenmaxxing-huge-capex-and-ai-agents-at-i/o/5242983).

Not dramatically. Just as a passing acknowledgment, the kind you make when you know the joke is already out there. Google's infrastructure now processes 3.2 quadrillion tokens per month — up from 480 trillion the year before — and Pichai got ahead of the obvious read: _"Now some out there might call this tokenmaxxing, and there's probably some truth to it."_

If the CEO of Google is making tokenmaxxing jokes at his own keynote, the concept has officially arrived.

## How we ended up “tokenmaxxing”

Tokenmaxxing, or “optimizing for token consumption as a proxy for AI productivity,” didn't emerge from bad intentions. It emerged from a measurement vacuum.

When your CFO wants proof that the AI investment is working, you reach for whatever's visible. Tokens are visible. They're countable, trackable, dashboardable.

So companies built leaderboards around them.

Some organizations experimented with token consumption as a productivity signal — internal leaderboards, usage tracking, team-level benchmarks. It was a reasonable instinct in the absence of better metrics.

[Amazon built usage leaderboards](https://fortune.com/2026/05/12/amazon-tokenmaxxing-claude-ai-capex-meta-gil-luria/). [Meta's employees built their own](https://theoutpost.ai/news-story/amazon-employees-are-tokenmaxxing-to-inflate-usage-scores-amid-pressure-to-add-ai-tools-26182/) — a grassroots leaderboard called "Claudeonomics" that ranked all 85,000 workers by token consumption. [Microsoft sent a memo](https://www.metaintro.com/blog/amazon-ai-usage-targets-inflated) making AI usage "no longer optional."

The lesson came quickly: measure throughput, and you get throughput. You get employees running trivial tasks at midnight to climb the scoreboard. Predictably (or not, in these unprecedented times!) things got a little weird — and most of the programs were quietly walked back.

What everyone actually wanted was goodput. They just didn't have the word for it yet.

[TheTigerGraph framed it up in a way that resonates](https://www.tigergraph.com/blog/tokenmaxxing-is-a-phase-inference-yield-is-the-strategy/): Tokenmaxxing is the AI equivalent of measuring software developer productivity by lines of code written. It drives behavior. Just not necessarily the right behavior.

👉 Get a different take: [How to Get Signalmaxxing out of Tokenmaxxing](/video/how-to-get-signalmaxxing-out-of-tokenmaxxing) by Val Bercovici, Chief AI Officer at WEKA.

## The paradox hiding in your AI budget

There is a number that deserves more attention than it gets.

The cost of a single output token has fallen roughly 280x over the last two years. But the average enterprise AI budget grew from $1.2 million per year in 2024 to $7 million in 2026. Some Fortune 500 companies now report monthly AI bills in the tens of millions. (Vishal Rajput's piece over at AIGuys — [Everyone Is Scaling AI. Nobody Is Solving Inference. That's the Real Problem](https://medium.com/aiguys/everyone-is-scaling-ai-nobody-is-solving-inference-thats-the-real-problem-2e2ce4fd4288) — breaks this paradox down in painful detail and is worth your time.)

Intelligence got 280x cheaper. Bills tripled.

That is not a pricing problem. That is a utilization problem — and not in the way most infrastructure teams frame it. Your GPUs might be running at 90% utilization. The dashboard is green. And you're still burning capital at a rate that doesn't match the output your users are actually receiving.

This is the difference between **throughput **and **goodput. **This is becoming the most important distinction in AI infrastructure right now. **Throughput measures activity. Goodput measures value.**

Throughput is the volume of tokens your system processes. It tells you your hardware is busy. It does not tell you whether that busyness produced anything a user actually received — at a quality and latency they found useful.

Goodput does. NVIDIA's own inference economics team [defines it cleanly](https://blogs.nvidia.com/blog/revenue-potential-ai-factories/): the amount of useful output a system delivers while hitting key latency targets. A system may process 100 requests per second — but if only 30% meet your latency SLOs, your goodput is effectively 30 r/s. The other 70% is throughput that went nowhere.

The gap between the two is where your AI budget disappears.

A GPU can register high utilization while stalled on KV-cache movement, waiting on memory pressure, processing pipeline overhead that never produces a single usable token. The dashboard reads healthy. The output rate tells a different story. As Dave Salvator, Director of Accelerated Computing Products at NVIDIA, [told The Register](https://www.theregister.com/2026/03/07/ai_inference_economics/): _"It's not one size fits all — there are SLAs, there are different application types."_The real question is how many tokens per second per dollar per watt you can generate for a given goodput target. Everything else is noise.

👉 This concept was covered on Episode 1 of WEKA’s new podcast, DeepGeeks, in conversation with Daria Mukhortova and Val Bercovici _—_ get more insights in this article, [Watt’s Holding AI Back](/article/can-ai-survive-its-own-energy-appetite-or).

## Why this gets exponentially worse from here

If you think the throughput/goodput gap is a problem today, wait until agents arrive at scale.

Chat inference is relatively lightweight. Reasoning models — the kind powering extended chain-of-thought — consume roughly an order of magnitude more compute than chat. Agentic workflows, where models orchestrate tools, retrieve context, and iterate autonomously, add another 10x on top of that. Deloitte estimated in a November report that [inference workloads will account for two-thirds of all AI compute in 2026](https://www.computerworld.com/article/4114579/ces-2026-ai-compute-sees-a-shift-from-training-to-inference.html), up from half in 2025. And that inference load is increasingly long-context, multi-step, and memory-hungry.

Here's what that means in practice. In January 2026, Turing Award winner David A. Patterson co-authored a paper that opened with three words: _"LLM inference is a crisis." _Not a challenge. Not a bottleneck. A crisis. His diagnosis — that memory, not compute, is the fundamental constraint — [is explored in depth over at Tech Unpacked](https://techunpacked.substack.com/p/28-exploring-new-directions-in-llm) if you want the full picture.

And then there's this detail, which Shayle Kann surfaced on [Latitude Media's Catalyst podcast](https://www.latitudemedia.com/news/catalyst-will-inference-move-to-the-edge/) and which deserves to be repeated more widely: data centers are running _fake _compute workloads — burning real energy on nothing — just to keep their power draw flat and avoid grid spikes. That is the logical endpoint of a system optimized for throughput at the expense of everything else. Busy hardware. Zero goodput. Real megawatts.

## The metric shift is already happening at the top

At GTC 2026, Jensen Huang reframed the AI infrastructure conversation around tokens per watt — not FLOPS per dollar, not GPU utilization, not raw throughput. How much useful AI work do you extract from every watt consumed?

SambaNova called this one early. Their [2025 predictions recap](https://sambanova.ai/blog/ai-trends-2026-insights) is worth a read: _"Power really did beat compute; inference really did swallow the bill; efficiency really did decide who could scale. "_KPIs like "intelligence per joule" and "tokens per watt" aren't aspirational anymore — they're operating conditions.

The organizations that still think in terms of GPU utilization percentages are measuring last decade's problem.

## Inference is a memory problem. Not a compute problem.

This is the part that doesn't get nearly enough attention in infrastructure conversations.

In most current architectures, GPU memory is physically coupled to the GPU. Need more memory for long-context or agentic workloads? You provision more GPUs — not because you need the compute, but because that's the only way to get the memory. Those GPUs sit largely idle during inference, drawing power, consuming capital, and contributing to your throughput numbers while adding nothing to your goodput.

Decouple memory from compute, and the economics flip entirely. You provision each resource where you actually need it. You stop paying the GPU tax on every inference job that runs long. The goodput gap closes — not because you bought more hardware, but because your architecture stopped wasting what you already had.

This is why inference optimization isn't a software patch. It's a fundamental mismatch between how modern AI models are architected and how the hardware we use to run them actually works. Getting the infrastructure layer right is what separates organizations that scale AI sustainably from those triangulating between their GPU dashboard and their AWS bill and wondering where the money went.

## What to actually measure

If goodput is the target, here's how to start:

- **Goodput over throughput. **Track the tokens users actually received at acceptable quality and latency — not total system activity. If you can't distinguish between the two today, that's the first thing to fix.
- **Tokens per watt. **Make energy efficiency a first-class metric alongside performance. This is where competitive differentiation lives in the inference economy.
- **Energy per completed workload. **Total energy per AI task: compute, cooling, and overhead. Track the full cost, not just the GPU line item.
- **Idle and overhead draw. **Provisioned-but-idle GPUs still consume 12–25% of active power. That's not a rounding error — it's a recurring tax on your efficiency.

👉 Check out this [AI energy metrics cheatsheet](/resource/ai-infrastructure-efficiency-metrics-explained) outlining the measurements that matter along with tips for designing efficient infrastructure.

## The bottom line

Tokenmaxxing was a symptom of a measurement problem. The industry grabbed the most visible metric and optimized around it — predictably, hilariously, and at significant cost.

The fix isn't to stop measuring. It's to measure the right thing. Goodput aligns your infrastructure investment with actual user outcomes. It connects the GPU cluster to the business result. And it exposes where throughput has been masquerading as value.

Sundar's tokenmaxxing joke landed because everyone in that room already knew it was true. The question is who builds infrastructure that makes it irrelevant.

_Want to go deeper on the concept of Goodput? [Deep Geeks Episode 1](/resources/podcast/can-ai-survive-its-own-energy-appetite-deepgeeks-ep-1/) breaks down what goodput-first infrastructure actually looks like in practice — with the people building it._