# Agents Don’t Benchmark Like Chatbots. Meet WEKA kvlab

**Author:** Ford Shaper

**Published:** November 16, 2026

![Black and white photo of a linear guide rail mechanism demonstrating recirculating ball bearings along tracks and curved loops on a dark surface.](https://cdn.sanity.io/images/ult5g8gw/production/c23268badc118c25ecbb37cc1d193474bf8ce522-1376x768.jpg)

## TL;DR

WEKA kvlab is an open source toolkit that measures how many real agent sessions your inference infrastructure can sustain, and how much of its GPU time goes to rebuilding context it already computed. It securely captures agent traffic as fingerprints, never text; replays it with the original context, pauses, and sub-agents; adds sessions until latency crosses a limit; and shows where every token came from: computed, served from GPU memory or loaded from an external KV tier.

Your AI agents don't ask one question and leave. Each one works in a loop. It sends the model a prompt, and the model answers with a tool call: read a file, run the tests, search the code. The tool runs, the agent appends its output to the conversation, and the agent sends the whole conversation back for the next step. A single task can take hundreds of these turns. Agents also hand work to sub-agents, which start from a copy of the parent’s context and run in parallel. Between turns, an agent may wait seconds for a file read or many minutes for a build or a human.

Every turn resends everything that came before it. The model’s working memory for that conversation, its key-value (KV) cache, is what lets the GPU skip reprocessing it. We call the KV that all live sessions need to keep warm the **working set**. Whether that working set is still in memory when an agent returns decides how much useful work your GPUs do. We built kvlab to measure exactly that.

## Why GPU Utilization Is the Wrong Scorecard

Today’s inference benchmarks measure how busy the infrastructure is: tokens per second, requests per second, and utilization at a chosen load. [MLPerf Inference’s](https://mlcommons.org/benchmarks/inference-datacenter/) Llama 2 70B test uses question-and-answer prompts of up to about 1,000 tokens in and out. SemiAnalysis’s [InferenceX](https://inferencex.semianalysis.com/) sweeps fixed input and output lengths, such as 1,000 in and 8,000 out, across set concurrency levels.[ vLLM’s serving benchmark](https://github.com/vllm-project/vllm/tree/main/benchmarks) commonly replays ShareGPT chat conversations. These benchmarks do their job well: they compare chips and serving software on independent requests.

For agents, a busy GPU isn’t the same as a productive one. In the traces we publish, the median agent request carries about 198,000 input tokens, and one in ten carries more than 530,000. Almost all of it is context the model has already processed. If its KV is still in memory, the GPU computes only the new tokens. If the engine evicted it, the GPU runs the full model over every token of that context again. What should have been a small prefill becomes a large, unexpected burst of compute, and every request queued behind it waits. Utilization stays high either way. The work is a repeat.

Working sets grow fast and vary widely: a session can reach hundreds of thousands of tokens in an afternoon, and hundreds of sessions share a node. GPU memory (HBM) holds a few dozen long sessions. Host DRAM holds more, but only on the server that computed it. Once the working set outgrows the memory it can reach, the engine evicts context, and every returning session pays to rebuild it.

Chat benchmarks rarely reach that point. Their prompts are short and independent, and the load is chosen up front. If it fits in HBM, every configuration looks the same and recompute never shows up. Synthetic generators can match each statistic of agent traffic and still miss how they combine.

## The New Benchmark Standard: Sessions Sustained

For agentic inference, the measure that matters is how many concurrent agent sessions a system can sustain while every user still gets a timely response, and how much of its GPU time goes to new work rather than rebuilt context. Sessions per server converts directly into cost: GPU servers per thousand agents. Measuring it takes three things chat benchmarks don’t provide:

- **Real traffic, not synthetic prompts:** sessions with their actual context growth, pauses and sub-agents.
- **Load that rises until latency breaks, not a load picked in advance,** so the result is a capacity, not an input.
- **An account of where every token came from,** so recompute is visible instead of hidden inside a utilization number.

kvlab does all three.

## Real Traffic, Captured Safely

What stresses a cache is a session that has grown to hundreds of thousands of tokens, pauses for four minutes, then spawns six sub-agents that share its prefix, all while hundreds of other sessions compete for the same memory. You only get that from real traffic, and real agent traffic is full of source code and proprietary context. kvlab replays real sessions faithfully while keeping their content private, in three steps:

- **Capture structure, not content.** A router in front of the AI coding agents WEKA engineers use every day records every block of context as a SHA-256 fingerprint, a one-way hash that shows when text repeats without revealing it, plus size, timing, think time, and which agent spawned which. The router stores no prompt or response text.
- **Rebuild sessions and their sub-agents.** A preparation step groups requests into sessions and links every sub-agent back to the turn that spawned it, so each tree of agents and sub-agents replays in the same bursts it ran in.
- **Regenerate deterministically.** At replay, kvlab generates stand-in content from each fingerprint. The same fingerprint always produces the same bytes, so context repeats exactly where it repeated in real work, and the server’s cache hits and misses where it did originally.

Because the router stores nothing sensitive, we can publish the captures. kvlab ships with two corpora of WEKA engineering traffic, both free to download without an account:

- **40 days:** 3,190 sessions and 370,658 requests. It’s built into the release and is the default, sized for a single-node, eight-hour run.
- **160 days:** 14,305 sessions and more than 1.7 million requests, for larger fleets and longer runs that need more distinct sessions.

You can also put the router in front of your own endpoints, capture your own agents’ traffic the same way, and replay it on any system.

## Let the Fleet Find Its Own Limit

kvlab replays every session in real time, with its own pauses intact. It keeps adding sessions while two limits hold: time to first token and decode speed. When either limit is crossed, it stops adding. The number of sessions a configuration sustains is its capacity, and every run records which limit set it.

The defaults, 10 seconds to first token and 20 tokens per second, are deliberately loose: a one-second limit would cap most systems at a few dozen sessions, where everything fits in HBM. They need no tuning and keep results comparable from one fleet to the next; the guide covers how admission works and when a fixed-concurrency mode fits.

## What a Run Tells You

- **Where every prompt token came from.** Read from vLLM or SGLang’s own accounting: computed, hit in local cache, or loaded from an external KV tier. Latency alone can’t show recompute.
- **How big the working set got,** sampled throughout, so you can see whether the run actually outgrew GPU memory.
- **Latency where users feel it.** Time to first token and response time at the median and the tail, plotted over time.
- **Every session on a timeline.** One row per session, each request split into queueing, prefill, and decode, with think pauses between. You can watch it live.
- **Every failure.** kvlab counts requests that hit the client’s timeout instead of dropping them. Without a cache tier, most are turns stuck behind a long rebuild.
- **Whether the answers stayed right.** An optional verify mode plants markers in replayed turns and flags any response that loses its own context or carries another session’s.

Every run starts cold: a unique stamp on every prompt means no run can borrow KV left behind by the last one.

## What It Has Shown So Far

We ran kvlab with Nebius on NVIDIA HGX B300 instances in Nebius AI Cloud: one eight-GPU node serving DeepSeek-V4-Pro to 512 active sessions for eight hours per configuration. With HBM alone, 60% of prefill was recomputed context. With WEKA Augmented Memory Grid™, it was 6.8%. The cache hit rate rose from 40% to 93%, and output tokens per GPU-hour rose 3.7x. 97% of requests got a first token within five seconds, against 44%. Same hardware, same workload, same eight hours. The full results are published by [Nebius](https://nebius.com/blog/posts/nebius-weka-shared-kv-cache-benchmark-hgx-b300) and [WEKA](/article/benchmarking-weka-on-nebius-ai-cloud).

## Where It Helps, and Where It Won’t

kvlab shows where a shared, pooled KV cache tier would pay off in your AI infrastructure, and where it wouldn’t. If your sessions are short, or your working set fits in HBM or host DRAM with room to spare, a shared tier won’t change your numbers much. Capture a few hours of your own agents’ traffic, and kvlab shows you which side of that line you’re on before you spend a GPU-hour.

## Built in the Open

kvlab, its capture router, and both corpora are open source under Apache 2.0, and they run against any infrastructure, not just WEKA® NeuralMesh™. NVIDIA’s AIPerf already ships a loader for the traces, and the methodology follows [NAND Research’s Inference Economics Scorecard](/resource/the-inference-economics-scorecard).

## Get Started

Point the Helm chart at your model server and it replays the embedded corpus for eight hours under the default limits:

`helm upgrade --install my-replay oci://quay.io/weka.io/helm/kvlab --version <vX> \`

`--set endpoint=http://YOUR-LLM-HOST:8000`



Or run it from the command line against a downloaded corpus:

`kvlab benchmark auto \`

`--router-replay-file replay.jsonl \`

`--models http://YOUR-LLM-HOST:8000 \`

`--timeout 8h --save-request-data ./results`



To test your own workload, put the kvlab router in front of your endpoint with `--capture redacted`, let it record a working day, and replay the result.

Our [**Getting Started with WEKA kvlab** guide ](https://weka.github.io/kvlab/)walks through the concepts, design decisions, and methodologies. The code and corpora are on [GitHub](https://github.com/weka/kvlab).
