Benchmarking WEKA’s Augmented Memory Grid on Nebius AI Cloud

This blog was co-written by Ford Shaper, Product Manager at WEKA; Anton Bykov, Senior Software Architect at WEKA; Alexey Averin, Senior Product Manager, Storage at Nebius; and Hooman Ramezani, ML Specialist Solutions Architect at Nebius.

Long-context and agentic inference workloads are largely bound by KV cache and memory. Every multi-turn session involves an agent re-reading a 100K+ token context, which can be accelerated by keeping that session’s KV cache resident instead of paying full prefill cost again. The HBM memory holds only a few dozen long-context sessions before evicting; DRAM offload extends this but is capped by per-node memory.

Augmented Memory Grid™ is a persistent, distributed KV cache storage layer for AI inference that integrates with NVIDIA HGX B300 instances on Nebius. Part of the WEKA® NeuralMesh® platform, it pools each node’s local NVMe over the most performant networking path available (RDMA over the NVIDIA InfiniBand fabric), turning siloed local storage into a fast, low-latency third tier for KV cache that sits between GPU/CPU memory and object/file storage.

The question this benchmark answers: under a realistic, sustained, multi-hour production load pattern, does Augmented Memory Grid measurably change the throughput, latency, and reliability envelope of long-context inference, and does it hold up, or degrade, over time?

The Dataset

The sessions are real captured agentic coding sessions, replayed, not synthetic prompts. They were recorded from live agent traffic through wekai’s capture router and are open-sourced alongside the benchmark: 5,000+ sessions, over half a million requests. Each replays a real multi-turn session, growing as the agent re-reads its accumulated context on every step. Turn increments and session depth come from the original traffic rather than being fixed, ranging from tens of tokens per turn up to contexts approaching 900K.

The Workload

WEKA replayed sessions drawn from a corpus of 5,000+ captured sessions, keeping 512 active at any moment, at a concurrency of 32: 28 steady-state sessions plus 4 "hot" sessions issuing back-to-back, sub-agent-style requests. Sessions that ran to completion were replaced, so the active set stayed at 512 while the run worked through the corpus. We settled on a concurrency of 32 after testing higher settings. At 64, TTFT rose without a corresponding gain in useful throughput, which is the behavior you’d expect once the scheduler is queuing rather than serving. Holding 512 sessions active is what makes this a cache test rather than a latency test: the working set grows past what HBM can hold, which is the condition the cache tier exists to handle.

Benchmarking Goals

The run was built to answer four questions:

1. Does it increase useful GPU output? Every token of context recomputed is GPU compute capacity not spent generating output. We want to measure how much of a fleet’s prefill work is redundant, and how much a distributed cache tier eliminates.
2. Does it hold interactive performance under load? We measured p90 and p99 on a busy cluster, not the median on an idle one. The tail determines how much hardware you need to meet an SLO.
3. Does it hold over time? Cache tiers can look really good for ten minutes. The failure mode that matters appears once the working set outgrows what memory can hold. This can present itself hours into the test.
4. Does the cache tier have room to grow into? Decode can be the bottleneck in today’s serving, so the cache tier is only asked for a fraction of what it could deliver. If it were already near its limit at that demand level, it would stop helping the moment decode got faster. We measured how much headroom it actually has.

Why HBM-only is the baseline

At the time of testing, no DRAM offload implementation could handle DeepSeek V4, so HBM-only was the sole baseline available on this model. Availability aside, DRAM offload carries a structural limit: it extends the cache only within a node. Capacity stops at the memory in the box, and a session that lands on a different node starts cold. Other solutions have matured since this testing. Look out for an upcoming report from WEKA comparing Augmented Memory Grid directly against them.

Time-to-First-Token (TTFT)

This is the metric users feel most directly in an interactive or agentic session. It’s the latency between sending a prompt and receiving the first token.

Tail spread matters as much as the percentiles. HBM’s p99 (48s) is 8.5× its own p50. This means a meaningful share of users are hitting long cold-prefill waits. WEKA Augmented Memory Grid’s p99 (11.7s) is 11× its p50, but it starts from a p50 that is already sub-second. WEKA’s p99 lands below HBM-only’s p90 of 17.2s.

This is where capacity planning happens. Against a 5-second TTFT bar, HBM-only clears it for 44% of requests and carries a p95 of 23 seconds, so you either add GPUs to buy down the tail or accept that most requests miss. Augmented Memory Grid’s p95 is 3.8 seconds, well under the bar without additional hardware.

Fraction of requests under common SLO thresholds:

Shares in this table are over successfully completed requests, excluding those that failed the 300-second client deadline. The throughput table below reports the same TTFT bar over all requests attempted, which is why the figures there read 43.3% and 96.3%.

These thresholds mark where latency distribution becomes a design constraint. Agentic workloads compound it: a session that makes ten model calls pays TTFT ten times, so per-request latency accumulates across the chain rather than being absorbed once.

At the p50 figures above, a ten-step chain spends 57 seconds waiting for first tokens on HBM-only versus 11 seconds on Augmented Memory Grid. Against a 5-second bar, HBM-only clears 44% of requests, so roughly 5.6 of those ten steps land over the line; Augmented Memory Grid clears 97%, or about 0.3 steps. The 2-second bar separates them further: 16.2% versus 82.5%.

Throughput

What does a GPU-hour buy you? Both HBM and WEKA Augmented Memory Grid runs consumed the same 64 GPU-hours: eight GPUs at TP=8, eight hours of wall clock, identical hardware, the same workload generator, and the same size 512-session pool. What differs is how much of that capacity went to producing useful output versus recomputing context the system had already seen.

For HBM-only, 60% of all context volume processed was freshly computed, and in most cases that was context the cluster had already computed earlier in the same session. With so many sessions competing for a few dozen HBM slots, the GPU’s prefix cache is overwritten faster than it can be reused, so the fleet pays full prefill repeatedly for the same tokens. On Augmented Memory Grid, that drops to under 7%. Per request, the GPU computes 4.4K fresh tokens instead of 33.9K.
That recovered capacity shows up directly as output. Same silicon, same clock, same eight hours: 46K output tokens per GPU-hour on HBM-only, 169K on Augmented Memory Grid. Put another way: the HBM-only node needs roughly 30 hours to produce what Augmented Memory Grid produced in eight.

Augmented Memory Grid served 2.4X the load at roughly a quarter of the error rate, which matters because the usual way to buy throughput is to accept more failures at the margin. Here the additional headroom came from eliminating work, not from pushing the system to its limit. (See the stability section for what those errors are and why the two columns aren’t the same failure mode.)

End-to-end response time (TTLT)

TTFT tells you how fast the first token lands; TTLT tells you how long the whole request takes, including decode.

The gap narrows at the extreme tail - both tiers are dealing with some long agentic sessions with large output - but WEKA Augmented Memory Grid is uniformly faster at every percentile, and the gap is largest exactly where it matters most for interactivity (p50/p90).

The p99 figures deserve a note, because the two tiers arrive there for different reasons. We looked at the longest requests on each side. On WEKA Augmented Memory Grid, they are decode-bound by output volume: one request returned its first token in 1.1 seconds, then spent 296 seconds generating 4,682 output tokens. Across the p99 tail, median output is 2,896 tokens against an overall median of 99. On HBM-only, the longest requests take a comparable amount of time but emit four times fewer tokens (994 to 1,417) because compute recomputes rather than generates. WEKA’s tail is the shape of the workload. HBM’s tail is the cost of recomputing context.

Stability over an 8-hour window

A single-point average can hide degradation. Bucketing both runs by wall-clock hour shows whether each tier holds up as the active dataset grows over the full soak. Error rate here is the share of requests in that hour where no output token arrived for 300 seconds, and the client gave up. Every failure in both runs was this stall, and neither produced OOM, eviction, or connection errors, so the two columns are the same failure mode under the same threshold. HBM-only’s rising rate is recompute: as sessions get deeper, more of each prompt has already been evicted, and falling back to full prefill produces stalls long enough to trip the deadline. On the WEKA path, the workload does not fall back to recompute, so the rate stays flat.

HBM-only trends worse across the run: TTFT p50 climbs from 6.7s to 7.9s, and error rate climbs from 1% to 5% as the active dataset grows past what HBM can hold. WEKA Augmented Memory Grid stays flat: TTFT p50 sits in a tight 0.98–1.14s band, and error rate never exceeds 1.3% over the full 8-hour window, while serving more than double the request load.

Cache efficiency

Long-context inference is expensive because the model re-reads context it has already processed. A cache hit means those tokens are fetched instead of recomputed. Augmented Memory Grid holds a high hit rate under sustained load. HBM-only doesn’t. That difference produces the results above.

HBM-only serves 40% of its context volume from cache. The other 60% is recomputed, most of it context the cluster had already processed earlier in the same session. There’s only so much HBM, and with hundreds of long sessions in flight, a session’s context is evicted before its next turn arrives. Augmented Memory Grid serves 93%. It fetched 6.6× more tokens from cache and recomputed 3.1× fewer.

The third row matters most. Augmented Memory Grid did not simply add a tier beneath HBM; it made HBM useful. The WEKA arm served 1,528.6M prefill tokens from GPU HBM, nearly three times what the HBM-only arm served from cache in total. On HBM-only, the GPU’s cache sits largely idle while the GPU recomputes evicted context. Remove the recompute, and that cache finally has something to do.

That’s the whole story. WEKA isn’t making GPUs faster; rather, it’s helping the system do less pointless work. Both configurations ran the same hardware for the same eight hours under the same load. Augmented Memory Grid completed 2.4X the requests.

Bottom line

For Nebius customers running agentic or long-context workloads, this is the difference between a platform that slows as usage grows and one that doesn’t. For Nebius, WEKA NeuralMesh and Augmented Memory Grid significantly increase the amount of useful work the same hardware can do, which greatly increases revenue extractable from the deployment with relatively lower software and storage costs.

We set out to answer four questions. Here’s where each landed.

1. Does it increase useful GPU output? Yes, by over 2.4 times. Augmented Memory Grid served 2.4x the request volume on identical hardware in the same window because it recomputed 3.1x fewer tokens. Serving that load from an HBM-only fleet takes substantially more GPUs. That is capex going into served requests instead of repeated prefill.
2. Does it hold interactive performance under load? Yes, and the tail is where it shows. HBM-only returns a first token within 5 seconds for 44% of requests, with a p95 of 23 seconds. Augmented Memory Grid does it for 97%, with a p95 of 3.8 seconds. At 97%, a 5-second first-token SLA is something you can commit to. At 44%, it isn’t. Augmented Memory Grid closed the gap without adding hardware.
3. Does it hold over time? Yes, across the full eight hours. Augmented Memory Grid’s median TTFT stayed inside a 0.98–1.14s band, and its timeout rate never passed 1.30%. HBM-only degraded steadily: median TTFT climbed, and timeouts rose from 1.08% to 5.21% as the working set outgrew what HBM could hold. A decaying tier forces you to provision for its worst hour. That’s hardware you’re buying to cover degradation, not to serve load.
4. Does the cache tier have room to grow into? Yes. Prefill ingest swung between 66K and 158K tokens/sec across the run, and WEKA’s share alone peaked near 115K, while median time to first token never left a 980-1,138 ms band. A tier near its limit shows up as latency tracking load, and this one did not move. Synthetic KV-load testing puts the ceiling well above anything this workload asked for, so as decode gets faster, the tier has room to absorb it.

Underneath all four goals is one number: a 93% cache hit rate against HBM-only’s 40%. Every result above follows from it. If your working set outgrows HBM, and for agentic workloads it will, that gap is the difference between a platform that degrades as your usage grows and one that doesn’t.

To go deeper, explore Augmented Memory Grid and Nebius AI Cloud.