# Inference Shouldn’t Be Graded on a Training-Era Scorecard. Now It Has Its Own.

**Author:** Liran Zvibel

**Published:** November 16, 2026

![A stylized 3D brain divided into two hemispheres against a numbered grid background, featuring a translucent, frosted left side and a glossy black right side.](https://cdn.sanity.io/images/ult5g8gw/production/74d4831125136cb6a0ba24f272ae0c7ec6aa770f-2400x1350.png)

## TL;DR

Measure inference by useful work, not GPU activity, with a new scorecard and open-source testing tools.

- Compare output per watt, rack, and dollar.
- Test real agent workloads over eight hours.
- See how shared context reduced recomputation.

In July, I wrote that [inference needs a scorecard](/article/inference-economy-ai-factory-infrastructure) of its own, one that judges infrastructure by how much useful work it gets from every watt, every rack, and every dollar rather than by how busy the chips look.

Today it has one: NAND Research, an independent analyst firm, has published the Inference Economics Scorecard, and alongside it we are [open-sourcing](https://github.com/weka/kvlab), the tool our engineers built to test against it.

## A busy GPU isn’t a productive GPU.

For most of the past three years, the industry graded AI infrastructure on one metric: GPU utilization, specifically measured as MFU (Model FLOPS Utilization). That logic held during training, because training is a fixed amount of work with a beginning and an end. A busy GPU was a productive GPU, and when the job finished, you could see what you got for it.

Inference doesn’t work that way, and agentic inference breaks the logic completely. An agent doesn’t answer a prompt. It completes a task, and a task has dozens or hundreds of requests. At every step, the agent re-reads everything it has already done, so the prompt for turn forty is the entire session so far. The inference engine only skips that recompute while the session’s context stays resident in GPU memory, and GPU memory is shared with the model weights and every other session the node serves. When context falls out of memory, the system has to rebuild. During this recomputation, the GPU stays busy, and utilization reads high, but much of that effort is work the cluster already did about an hour earlier, and a complete waste.

Utilization measures effort, and effort was a fine proxy when the job had a predetermined scale. It tells you nothing about whether the work got finished, got repeated, or got thrown away. It never sees the request that timed out, the call that had to be retried, or the user who gave up because the first token took too long to arrive. All of that shows up in your token and energy costs, and none of it shows up on the dashboard, which is why so many teams see a healthy utilization chart and an unhealthy bill at the same time. In a world where a lot of work gets repeated and wasted, it’s actually easier to also see a higher utilization number.

Inference never had its own measurement discipline, and I think that’s because training was a capital project and inference is an operating cost. You measure a capital project by whether it finished and what it cost to build, and GPU utilization answered that well enough. You measure an operating cost by what it produces every hour it runs, and nobody built the tools for that because, until agents, the question wasn’t expensive enough to ask. Short benchmarks survived for the same reason. A vendor sells on peak numbers, a buyer evaluates on the vendor’s numbers, and a ten-minute run is cheaper for both. The incentive to measure steady-state longer sessions only arrives when the steady state is what’s costing you money, which is where the industry is now.

## The new math of inference.

The bill has changed shape too. Token demand is growing faster than anyone can build the infrastructure to serve it. OpenRouter’s weekly token volume has grown more than 250x since the start of 2025. A new study by S&P Global Market Intelligence projects that North American token production will grow nearly tenfold by 2028, with agents going from roughly a third of those tokens to nearly three-quarters. Some of that growth is waste. Agents that re-read and recompute the same context burn tokens and GPU hours without getting more work done, which is why counting tokens isn’t enough either. Not long ago the thing that held you back was capital, and the question was how many GPUs you could get your hands on.

Today the GPUs are the easy part. What runs out is power, rack space, and memory, none of which scale as fast as demand, and the question has become how much finished work you can get from the power and footprint you already have. That’s why the measures that matter now are useful work per watt, per rack, and per dollar, delivered at the response times your users will actually tolerate.

NAND’s scorecard puts that math on paper: nine measures that run from tokens per watt to context retention, and the full list is in the report. The part I care about most is the four rules for how you are to test against them, because the rules are what turn a list of metrics into a test you can fail:

- The same workload and the same service targets on every system you compare
- Measurement of the whole serving stack, not just one component in isolation
- Runs of at least eight hours
- Only successful output counted

Peak throughput tells you what a node can do for ten minutes, and nobody runs agents for ten minutes. The industry is arriving at the same conclusion from several directions, with SemiAnalysis bringing multi-turn agent workloads into the AgentX part of InferenceX, and MLCommons adding KV cache testing to MLPerf Storage. The scoreboard is changing because the workload already has.

## Nobody had a test that behaves like an agent, so we built one.

Rules like these are easy to write and hard to run because you need traffic that behaves like agents. Public benchmarks still use synthetic single-turn prompts, which is exactly the traffic shape that makes utilization look healthy. Real agent traffic is hard to come by because it includes source code and proprietary context, and summary statistics aren’t a substitute, since cache behavior depends on which context repeats, when agents branch, and how long they pause between turns.

So our engineers built kvlab. A router in front of the coding agents they use every day records the structure of every request and never its content so that sessions can be replayed on any stack with the same timing, the same context growth, and the same sub-agents spawning and rejoining, and measured over hours rather than minutes. We captured more than 5,000 real sessions and over half a million requests this way without keeping a word of code or conversation, and then we published the corpus, the capture router, and the replay harness.

This wasn’t a marketing-led decision. Our engineers needed a benchmark they could trust, and they understood that nobody else would trust it unless anyone could run it. A benchmark only its author can run is a worthless claim, not a valuable measurement. The trace format has since been picked up outside WEKA: NVIDIA’s AIPerf now ships a loader for it, and SemiAnalysis’s AgentX benchmark builds on the same corpus. That’s the outcome we hoped for. A scorecard becomes a standard only when the people being scored didn’t write it.

## What eight hours on Nebius AI Cloud Platform showed.

With a real test in hand, we ran it with our partners at Nebius on standard NVIDIA HGX B300 instances in Nebius AI Cloud, under the same conditions the scorecard calls for, because neoclouds are where inference challenges compound the most. They sell tokens and pay for the power, and a bad hour eats into their margin.

The setup was deliberately plain: one eight-GPU node, the same serving stack in both runs, 512 concurrent sessions, eight hours per configuration, and every failed and timed-out request counted. The only variable was where evicted context could go. In one run, nowhere: when it fell out of HBM, the node recomputed it. In the other, it went to a shared tier that WEKA Augmented Memory Grid built across the cluster, so any machine could read back context another had already written.

On HBM alone, 60% of the prefill work over the run was spent recomputing context the cluster had already processed earlier in the session; with the shared tier, it was 6.8%. The same node produced 3.7x the output tokens per GPU-hour, and 97% of requests reached a first token within five seconds, against 44%. At the measured rate, the HBM-only node would have needed about 30 hours to finish what the other configuration finished in eight. Same hardware, same workload, same cloud. That’s the difference between keeping a GPU busy and making it productive. Nebius and WEKA have published the full results, including the model, the context window, and the configuration.

## Where the scorecard can’t help you.

I want to be transparent about the limits. If your sessions are short, your context is small, or your working set fits in HBM with room to spare, a shared context tier won’t change your numbers much, and you shouldn’t buy one based on this post. The gain appears when sessions are long - over many turns, concurrency is real, and the run is measured over hours rather than minutes. That’s where agents live, and that’s where utilization may have been misleading you. Capture a few hours of your own traffic with kvlab, and it will show you which side of that line you are on.

## Measure us too.

When we sponsored the scorecard research and supplied the workload it was tested on, we knew we’d have to be measured by the scorecard’s rules like everyone else. The traces are public, the tooling is public, and the test conditions are published. Run it on your own stack, run it on ours, run it on another architecture, and bring your numbers. The scorecard was designed to put the question back in the hands of the people paying the bill, and the question is no longer how busy their chips are. It’s how much finished work you got for the power, the rack, and the dollar. Hold every vendor to those rules, including us.

Intelligence stopped being scarce some time ago. What’s scarce now is the power, memory, and floor space needed to serve it. The inference economy will reward whoever wastes the least of it, because every token that has to be recomputed is margin that someone else will keep.

—

Read the [Inference Economics Scorecard report](/resource/the-inference-economics-scorecard) from NAND Research. The kvlab code, traces, and getting-started guide are now available on [GitHub](https://github.com/weka/kvlab).
