# Want 10x Token Throughput? Check Out Our Latest Results with OCI. **Author:** WEKA **Published:** August 20, 2026 ![A server rack with light lines in a dark data center.](https://cdn.sanity.io/images/ult5g8gw/production/1d2b2846629ca84e431161947b9c8e3ef778448c-1408x768.jpg) ## TL;DR Discover how WEKA and OCI deliver 10x token throughput and concurrency without adding GPUs. - Boost concurrent users and token throughput on existing OCI H100 clusters. - Break through DRAM limits using NeuralMesh with Augmented Memory Grid. - Cut cost per token by reducing recomputation at high concurrency. [Oracle Cloud Infrastructure (OCI)](https://www.oracle.com/cloud/) and WEKA released [production-scale results](/company/weka-newsroom/press-releases/weka-and-oracle-cloud-infrastructure-validate-10x-throughput-gains/) that change the math on long-context AI inference. [NeuralMesh™](/product/neuralmesh/) with [Augmented Memory Grid™](/product/augmented-memory-grid/) on OCI delivered 10x more concurrent users, 10x higher token throughput, and 7x more tokens served compared to a DRAM-only architecture without adding a single GPU. OCI validated and published the full methodology on its [AI & Data Science blog](https://blogs.oracle.com/ai-and-datascience/scaling-long-context-inference-on-oci-with-wekas-augmented-memory-grid) using a nine-node OCI bare-metal H100 cluster 72 GPUs, 100,000-token context windows, and thousands of concurrent users. This isn’t a lab result. It is production infrastructure, tested at scale, with numbers OCI stands behind publicly. ## Why benchmarking token throughput at scale matters right now Every conversation about AI capacity eventually arrives at the same conclusion: You need more GPUs, more chips, more infrastructure. The supply chain for H100s and H200s has been constrained for two years running [and is expected to drag into 2027](/resources/eguide/surviving-the-nand-flash-shortage/), with companies that can’t get hardware fast enough being told to wait. 👉 Read more: WEKA’s take on [AI capacity planning](https://www.linkedin.com/pulse/everyones-talking-ai-capacity-nobodys-why-its-breaking-weka-io-dx9bc/) These benchmarks say something different. The constraint isn’t always supply. It’s also architecture. Specifically, it’s the AI [memory wall](/article/how-ai-s-memory-wall-is-reshaping-infrastructure-strategy-beyond-gpus): what happens to your GPU cluster when [key-value (KV) cache](/learn/ai-ml/what-is-kv-cache/) exhausts available HBM and DRAM. As context windows grow, the KV cache required to process each request grows with them. A standard cluster configuration holds roughly 8-10 TiB of DRAM. Once the DRAM fills up, the GPU can’t retrieve prior context. Instead, it evicts and recomputes older queries from scratch. Every eviction burns GPU cycles, adds latency, and inflates the cost of every token you serve. At 100,000-token context windows running thousands of concurrent sessions, this recomputation overhead isn’t a rounding error. It’s the reason your infrastructure underperforms relative to what you paid for it. ## What the OCI numbers actually show The new benchmarks position NeuralMesh with Augmented Memory Grid against a DRAM-only baseline on identical OCI hardware. The results: - **10x more concurrent users.**Testing surpassed 5,000 concurrent users vs. roughly 600 users on DRAM-only infrastructure. The same nine nodes, the same 72 GPUs. The difference is the active cache working set growing from 8.64 TiB of DRAM to 287 TiB of usable NVMe, eliminating the saturation cliff that DRAM-only configurations hit long before business targets are achieved. - **10x higher token throughput.**Approximately 2 million tokens per second on OCI vs. under 200,000 for the DRAM-only baseline. For teams running real-time AI features — search, summarization, code assist, multi-turn agents — this throughput ceiling determines how many users you can serve, how fast features respond, and how much revenue the infrastructure can support. - **7x more tokens served.**Five billion tokens versus 700 million in a single one-hour, 2,400-user test. For agentic workflows where sessions are long and context is everything, DRAM saturation quietly converts GPU capacity into recomputation overhead — a direct hit on cost per token that won’t appear as a line item anywhere in your infrastructure budget. The architecture behind this is Augmented Memory Grid, a capability of NeuralMesh. It decouples KV cache from local GPU memory and stores it in a high-performance token warehouse accessible across the cluster. Any host can serve any session with cache hits intact — no session stickiness, no recomputation penalty, clean horizontal scaling as concurrency grows. ## Why the efficiency story lands differently in 2026 There’s a broader reason these results matter beyond the benchmark numbers themselves. AI infrastructure spending is under mounting scrutiny as boards and CFOs evaluate ROI on AI spending. [Gartner recently surveyed 700+ industry leaders](https://www.gartner.com/en/newsroom/press-releases/2026-04-07-gartner-says-artificial-intelligence-projects-in-infrastructure-and-operations-stall-ahead-of-meaningful-roi-returns) in late 2025 and found 28% of AI use cases in infrastructure and operations fully succeed and meet ROI expectations, while 20% fail entirely. The reality is executive and investor tolerance for open-ended infrastructure investment without a clear payoff is shrinking fast. Coming out ahead in the AI race won’t necessarily be dependent on who can build the largest GPU fleets. The decisive advantage will belong to organizations that can extract the most value from the hardware they already have — breaking through the AI memory wall [and meeting exponential token demand without investing in more infrastructure](/article/token-demand-went-exponential-your-infrastructure-didn-t-get-the-memo). A 10x concurrency gain and a 7x cost-per-token improvement on existing infrastructure isn’t just a technical achievement; it’s a credible answer to the question every CFO is asking. Can you scale AI without costs scaling just as fast? ## Three numbers to pressure-test in your own environment If you’re running long-context or agentic inference in production, these data points are worth knowing before your next infrastructure conversation: **1️⃣ KV cache hit rate. **Cache misses at scale aren’t a tuning problem, but an architecture problem. If you’re running context windows of 100,000-plus tokens on DRAM-only infrastructure, your hit rate is likely worse than you think, and the cycles lost to recomputation aren’t visible anywhere in standard observability tooling. Ultimately, the higher your KV cache hit rate, the less time your GPU is spending on compute-intensive operations that aren’t necessary. **2️⃣ Concurrency ceiling. **At roughly 600 concurrent users, the DRAM-only configuration tested in the benchmarks hit a failure cliff. Understanding exactly where this cliff lies — and what it costs in user capacity per GPU dollar — changes the ROI calculation on your current footprint. **3️⃣ Actual cost per token at peak load. **Benchmarked cost per token on a quiet cluster looks very different from cost per token when demand spikes and cache saturation kicks in. The OCI results reveal a gap between the two architectures where a realistic sustained load delivers 7x more tokens with Augmented Memory Grid on OCI. This gap equates to real money that can be recovered. ## The efficiency era isn’t approaching. It’s here. The organizations that move first to maximize existing inference architecture — before the next GPU procurement cycle, before the next infrastructure contract — are the ones that will have the most to show for the hardware they already own. The benchmarks exist. The architecture is proven. The question is how much GPU capacity you’re leaving on the table while you wait. Make sure to head over to Oracle’s blog to read about the latest benchmark: [Scaling Long-Context Inference on OCI with WEKA’s Augmented Memory Grid](https://blogs.oracle.com/ai-and-datascience/scaling-long-context-inference-on-oci-with-wekas-augmented-memory-grid) written by Amar Gowda, Dennis Kennetz, Betsy Chernoff, and Anton Bykov. 👉 Want to learn more about how to increase AI’s ROI through better efficiency? [Watch this video](/resources/video/the-real-cost-of-ai-how-smart-companies-are-maximizing-token-roi/) of WEKA’s Val Bercovici during a panel discussion at the [HumanX](https://www.humanx.co/us) conference.