# What Top-Performing AI Clouds Built Differently with WEKA This isn't a best-practices guide. It's a brief based on direct feedback, anecdotes, and learnings from the AI clouds running inference workloads in production; including what they built, what they measured, and what changed. **Type:** White Paper ## Introduction WEKA's infrastructure runs inside some of the world's leading AI clouds and hyperscalers: CoreWeave, Cohere, Firmus, and Oracle Cloud Infrastructure. Over the past two years, we've worked alongside these teams, managing, testing, and benchmarking what happens when AI inference workloads hit real scale: context windows stretch to 100,000 tokens and beyond, concurrent users multiply, and multi-agent workflows replace single-shot queries. And we keep finding the same thing: the infrastructure decision that looked fine at 10 petabytes starts failing at 50. The GPU utilization number stays high. The token economics quietly deteriorate. This isn't a best-practices guide. It's a brief based on direct feedback, anecdotes, and learnings from the AI clouds running these workloads in production; including what they built, what they measured, and what changed. We've added what we think comes next, because the pattern we've seen across these deployments points somewhere specific. ## The pattern we kept finding inside the best AI clusters Every large language model inference request generates KV cache — the record of key-value pairs computed during a session. Retaining it means the model doesn't have to re-compute context it's already seen. The problem is that KV cache grows with context length, and GPU high-bandwidth memory (HBM) runs out. Even the newest Blackwell GPUs don't close that gap: a B200 carries 192GB of HBM, a B300 288GB — up from 80GB on the H100. Models are also getting more KV-cache efficient. But neither trend will keep pace with what production inference demands: 100,000-token context windows, thousands of concurrent sessions, and agentic workflows that keep context alive far longer than a single query. Run enough of those sessions at once, and even Blackwell's added headroom disappears. When HBM fills up, the cache evicts. The system re-prefills from scratch. Latency spikes. Throughput drops. The GPU utilization number stays high because the GPU is busy redoing work it has already done. Every eviction is a tax: on GPU cycles, on latency, on user experience, and on the cost of every token served. For long-context and agentic workloads, where inputs routinely run to 100,000 tokens or more, that tax isn't a rounding error. It's the real ceiling on what the infrastructure can do. DRAM was not built to serve as a KV-cache tier. It's designed for high-speed, sequential, transactional work, not overflow storage. DRAM in a standard GPU tops out around 8–9 TiB, and at $5–10 per GB, it's too expensive to scale to petabyte-class context volumes. Worse, DRAM is strictly node-local. It can't share KV state across the cluster, so each inference instance becomes an isolated cache island. Offloading overflow duty to WEKA's NVMe tier removes that ceiling and frees DRAM to do the job it was designed for. ```json { "_key": "d69ee0d99982", "_type": "textCallout", "content": [ { "_key": "rp00022", "_type": "block", "children": [ { "_key": "rp00021", "_type": "span", "marks": [], "text": "Below DRAM saturation, systems look fine. Above it, cache misses compound, TTFT climbs, and throughput falls all while GPU utilization stays high." } ], "markDefs": [], "style": "normal" }, { "_key": "rp00024", "_type": "block", "children": [ { "_key": "rp00023", "_type": "span", "marks": [], "text": "Cache evictions force re-prefill and GPU cycles spent redoing work already done." } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "rp00026", "_type": "block", "children": [ { "_key": "rp00025", "_type": "span", "marks": [], "text": "TTFT becomes unpredictable under load, making SLO commitments harder to keep." } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "rp00028", "_type": "block", "children": [ { "_key": "rp00027", "_type": "span", "marks": [], "text": "Adding DRAM delays the ceiling, it doesn't remove it." } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" } ], "markDefs": null, "title": "The failure cliff is invisible until you hit it." } ``` ## CoreWeave: 4.2x more tokens from the same 64 GPUs CoreWeave, ranked #1 on SemiAnalysis's ClusterMAX AI cloud rating, was running production inference workloads on 64 H100 GPUs when we measured what was actually happening to token economics as context length increased. The results quantified the cost of the memory wall precisely. With DRAM-only caching, once working sets exceeded DRAM capacity, throughput dropped to 55 tokens per second and TTFT climbed to 10.9 seconds. With **WEKA Augmented Memory Grid**, which extends KV cache storage 1,000x beyond HBM and DRAM using NVMe over RDMA via NVIDIA GPUDirect Storage, throughput reached 231 tokens per second and TTFT held at 1.8 seconds. Same 64 GPUs. Same cluster. 4.2x more usable output and 6x faster response. ```json { "_key": "rp00038", "_type": "statsBlock", "markDefs": null, "stats": [ { "_key": "rp00035", "_type": "statItem", "label": "More token output per GPU — CoreWeave H100 production", "value": "4.2x" }, { "_key": "rp00036", "_type": "statItem", "label": "Faster time-to-first-token — same CoreWeave cluster", "value": "6x" }, { "_key": "rp00037", "_type": "statItem", "label": "Tokens/sec with AMG vs. 55 DRAM-only — CoreWeave", "value": "231" } ] } ``` **NeuralMesh Axon** — which fuses WEKA's distributed file system directly into GPU servers using the NVMe drives already on compute nodes — is what makes the data path possible. No separate storage infrastructure. No additional rack space or power draw. The storage is already in the cluster. NeuralMesh Axon activates it. On CoreWeave's production infrastructure, NeuralMesh Axon delivers more than 30 GB/s read and 1 million IOPS to a single GPU server at microsecond latency. Learn more at [NeuralMesh Axon](/product/neuralmesh-axon). ```json { "_key": "2b7b5aa4f75e", "_type": "customerQuoteBlock", "markDefs": null, "quote": { "anonymizedAttribution": null, "anonymous": false, "authorName": "Marc Hamilton", "companyLogo": { "_type": "image", "asset": { "_id": "image-90168664a1fab04e62ecbb3fe444e6fb6afa4b08-300x56-svg", "metadata": { "dimensions": { "_type": "sanity.imageDimensions", "aspectRatio": 5.357142857142857, "height": 56, "width": 300 }, "lqip": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAECAYAAACOXx+WAAAACXBIWXMAAAsTAAALEwEAmpwYAAAA1UlEQVR4nC3QTStEYQDF8d8n8Bk0RcTIS5pBZCOysWDBwlsNU5Rp7nMnMogVmZ0GsaQs7WaeYeeb6U539V+c06n/cfZrKIlWw4/bEHXS6K/65X5ywzrmMYcSxjGFZaxhEUWMYBpjGBC6tpOokvQ069FLiL4P3rQLJTWc4BpXOMYhLvGEB5xiH+d5d1jasdmIKqHrph61kp7XvWePhbIqdpCgmXMXDbTRQshH75CirBYNJh0rSddFiN7Tno+jT82Zzb7aKGaxgCVM5Nyif0mmnWUZs2uK/3U5Oo1LH6dRAAAAAElFTkSuQmCC" }, "url": "https://cdn.sanity.io/images/ult5g8gw/production/90168664a1fab04e62ecbb3fe444e6fb6afa4b08-300x56.svg" }, "crop": null, "hotspot": null }, "companyLogoMono": { "_type": "image", "asset": { "_id": "image-466d566ddd8027bf1a60d56b2e0d9680103246af-300x56-svg", "metadata": { "dimensions": { "_type": "sanity.imageDimensions", "aspectRatio": 5.357142857142857, "height": 56, "width": 300 }, "lqip": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAECAYAAACOXx+WAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAqUlEQVR4nC3Qv0oCcBTF8c8T9AwuiuIfSJAyURqtOYeCCJcERQL/IGgGgaLkJg4NufQCrs0+WfzgTudy7uF7uQeyuMcafzhjE94N6rhCCZe4RRtNlJFHFUVcwBP6WOEHJxzwFv4S73hFF3PsscUAL5hFNpeAHQzxFcHvmHt4xAiL0GdM4+AO44B+YoLrBMzgDh844jcA6bUCamighUroQ1SS3k67pKma8j8mcx3pOyXAqAAAAABJRU5ErkJggg==" }, "url": "https://cdn.sanity.io/images/ult5g8gw/production/466d566ddd8027bf1a60d56b2e0d9680103246af-300x56.svg" }, "crop": null, "hotspot": null }, "companyName": "NVIDIA", "jobTitle": "VP, Solutions Architecture & Engineering", "text": "Partner solutions like WEKA's NeuralMesh Axon deployed with\nCoreWeave provide a critical foundation for accelerated\ninferencing while enabling next-generation AI services with\nexceptional performance and cost efficiency. Embedding\nultra-low latency NVMe storage close to the GPUs unlocks more bandwidth and extends available on-GPU memory." } } ``` ## Oracle Cloud Infrastructure: 10x concurrent users, 7x more tokens — same hardware In May 2026, Oracle published production-scale benchmark results on its AI & Data Science blog validating WEKA Augmented Memory Grid on OCI bare-metal H100 infrastructure — nine nodes, 72 GPUs, 100,000-token context windows, thousands of concurrent users. The test was designed specifically to expose how systems behave when pushed past steady-state and the conditions that reveal the real production ceiling. DRAM-only configurations hit a hard ceiling at approximately 600 concurrent users. Augmented Memory Grid scaled past 5,000. In the same one-hour, 2,400-user test, DRAM-only served 700 million tokens. Augmented Memory Grid served 5 billion. Token throughput: under 200,000 tokens per second for DRAM versus approximately 2 million tokens per second with Augmented Memory Grid — a 10x gain. The cache pool: 8.64 TiB of DRAM versus 287 TiB of usable NVMe. The hardware was identical. What changed was the architecture. ```json { "_key": "rp00057", "_type": "statsBlock", "markDefs": null, "stats": [ { "_key": "rp00054", "_type": "statItem", "label": "More concurrent users — OCI H100, 100K-token context", "value": "10x" }, { "_key": "rp00055", "_type": "statItem", "label": "More tokens served — same 1-hour test window on OCI", "value": "7x" }, { "_key": "rp00056", "_type": "statItem", "label": "Tokens served with AMG vs. 700M DRAM-only — OCI", "value": "5B" } ] } ``` Oracle's team noted the pattern clearly in their published results: the performance gap between DRAM-only caching and Augmented Memory Grid "does not appear gradually. It appears at the point where DRAM cache saturates." That's the operational reality: below the ceiling, the difference can be invisible. Above it, one configuration keeps scaling and one doesn't. Oracle has positioned the results as a validated reference architecture for long-context inference on OCI, and NeuralMesh with Augmented Memory Grid is now available on the Oracle Cloud Marketplace — with OCI as WEKA's exclusive cloud launch partner. ```json { "_key": "7c408064edf4", "_type": "customerQuoteBlock", "markDefs": null, "quote": { "anonymizedAttribution": null, "anonymous": false, "authorName": "Liran Zvibel", "companyLogo": { "_type": "image", "asset": { "_id": "image-c8453def11a637f3d8d029d796a8991ecce5675d-3499x1086-png", "metadata": { "dimensions": { "_type": "sanity.imageDimensions", "aspectRatio": 3.221915285451197, "height": 1086, "width": 3499 }, "lqip": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAYAAADDl76dAAAACXBIWXMAAC4jAAAuIwF4pT92AAAA4klEQVR4nF3RyyuFURQF8O9yP67yikKIlNdVrmRI8hgYKELKyEBIKCkZyMDQX71/OrXV7Z5a7dPZ56x19loVBjCGkazTETGPCdRoJuqIGEQr39Q9/SYaFaawjwMc4goX2MQkFrGR2MI21hNLmMUyZopAlZsbfOMLv/jADhZwFhHveMVL4haPOMcJ7vJTw4VwKFXf8JOkn9hLwtMkfMY9nroIryPiMiIecFQsK4SNJO2k2i6Oc7zi41yOu5boREQ7R17FSp6Xe63qf6XZ4yWciBjNkIrRfd3B9IRSakHp9ZdQ/gBni3VBZxrGXQAAAABJRU5ErkJggg==" }, "url": "https://cdn.sanity.io/images/ult5g8gw/production/c8453def11a637f3d8d029d796a8991ecce5675d-3499x1086.png" }, "crop": null, "hotspot": null }, "companyLogoMono": null, "companyName": "WEKA", "jobTitle": "Co-Founder & CEO", "text": "NeuralMesh with Augmented Memory Grid running on OCI brings orders of magnitude more tokens to customers in an extremely cost-efficient way." } } ``` ## Cohere: when the infrastructure change shows up in operations The CoreWeave and OCI results measure efficiency at the infrastructure layer — tokens per GPU, users per cluster. Cohere's experience with **NeuralMesh Axon** shows what the same architecture change looks like in day-to-day AI cloud operations. Inference deployments that previously took five minutes complete in 15 seconds. Checkpointing — the operation that saves model state during training and protects against costly restarts — accelerated by 10x. Both outcomes come from the same structural change: WEKA's file system running directly on the NVMe already inside the GPU servers, removing the data path overhead of a separate storage tier. ```json { "_key": "rp00068", "_type": "customerQuoteBlock", "markDefs": null, "quote": { "anonymizedAttribution": null, "anonymous": false, "authorName": "Autumn Moulder", "companyLogo": { "_type": "image", "asset": { "_id": "image-2fcf1744017ced2e5e07d87135c07cd24ee3d23e-300x51-svg", "metadata": { "dimensions": { "_type": "sanity.imageDimensions", "aspectRatio": 5.882352941176471, "height": 51, "width": 300 }, "lqip": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAADCAYAAACTWi8uAAAACXBIWXMAAAsTAAALEwEAmpwYAAAA9klEQVR4nGOwCfJWsgn3bbaLDLzoFB22wjEqzM4w1E3FINRJyzTcU800wl3bJNzLyjTUK8w03MfZNNTLwCTQw9Qs0E3f1N/VwCjAzdgozMPCONJNw9jHh4uhOilIPj8psig3vWhVXX5/Z0FOs5N5hKerWYSXN8gAkzB3F7Nw71CzUM9sizCfFLNQr1jzMO8I8zDPYPNQz3DTUO9AszDPOJB6m1APUYZHhaEql0rzig/VLp5/uP1c77SKZX7WEb5h5hFeMWahXt5mYZ4BZqFekSahHklm4V7JZqHeSaZh3tHmoV4xpqGeGSA58zDPRPMwby/7SB8RADjSTFUsuKY4AAAAAElFTkSuQmCC" }, "url": "https://cdn.sanity.io/images/ult5g8gw/production/2fcf1744017ced2e5e07d87135c07cd24ee3d23e-300x51.svg" }, "crop": null, "hotspot": null }, "companyLogoMono": { "_type": "image", "asset": { "_id": "image-143a388ffb79f85b240b29cdd7c4b7a53a71df9f-300x51-svg", "metadata": { "dimensions": { "_type": "sanity.imageDimensions", "aspectRatio": 5.882352941176471, "height": 51, "width": 300 }, "lqip": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAADCAYAAACTWi8uAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAgUlEQVR4nC3OvQqBARyF8d9sZlaSxEBJyWB4sVAi4ZWP8jEoG5dgMrgGq9yNO9Jb/+mcek5PB0p44IcPeiijhgrq6GKJPppooxG9hQ6qyEERN3zxRIIhxiEYYIErztghxRwrzLCPfSETZm/ueOOFSbzZxmiKNY44RW6CX4IdMEL+D5XXEvRvMUYhAAAAAElFTkSuQmCC" }, "url": "https://cdn.sanity.io/images/ult5g8gw/production/143a388ffb79f85b240b29cdd7c4b7a53a71df9f-300x51.svg" }, "crop": null, "hotspot": null }, "companyName": "Cohere", "jobTitle": "Director of Infrastructure & Security", "text": "Embedding WEKA's NeuralMesh Axon into our GPU servers enabled us to maximize utilization and accelerate every step of our AI pipelines. The performance gains have been game-changing: Inference deployments that used to take five minutes can occur in 15 seconds, with 10 times faster checkpointing. Our team can now iterate on and bring revolutionary new AI models, like North, to market with unprecedented speed." } } ``` ## Firmus: when infrastructure efficiency becomes a financing thesis [Firmus](/customer/firmus) built its entire AI cloud on the premise that energy efficiency is a business model, not an engineering goal. In production, WEKA delivered 6.5x higher tokens per second and 4x faster TTFT at scale. Firmus then raised $10 billion in financing — anchored, in material part, by its ability to demonstrate infrastructure economics that competing neoclouds couldn't match. The efficiency story became the investor story. The investor story became the growth story. Infrastructure efficiency compounds: tokens-per-watt, sustained performance under multi-tenant load, and SLO stability at scale attract different capital, different customers, and different contract structures. ## What CoreWeave, OCI, Cohere, and Firmus have in common The infrastructure architecture in each case is the same: extend KV cache storage beyond local GPU memory into a high-performance token warehouse — persistent, petabyte-scale, accessible across the cluster at memory-class speed. Decoupling cache from GPU-coupled DRAM removes session stickiness, enables safe GPU memory oversubscription, and increases tenant density without adding hardware. The workloads are different. The business models are different. The architecture decision is the same. There's also a supply chain dimension worth noting. Because NeuralMesh Axon uses NVMe drives already in GPU compute nodes, the storage capacity upgrade doesn't require a separate hardware procurement cycle. For neoclouds building fast — or responding to demand faster than purpose-built storage can arrive — WEKA deploys on existing bare metal or Kubernetes infrastructure from any hardware vendor. WEKApod™ provides a pre-integrated appliance path for providers who need turnkey deployment with predictable lead times, particularly relevant for EMEA neoclouds navigating sovereign AI mandates and long procurement cycles. ```json { "_key": "88f5a2e1858d", "_type": "textCallout", "content": [ { "_key": "5efab2e1c4ec", "_type": "block", "children": [ { "_key": "8e39514da4f2", "_type": "span", "marks": [], "text": "Every deployment extended KV cache beyond GPU-coupled DRAM into a persistent, cluster-wide token warehouse. The workloads differed. The decision didn't." } ], "markDefs": [], "style": "normal" }, { "_key": "rp00084", "_type": "block", "children": [ { "_key": "rp00083", "_type": "span", "marks": [], "text": "Decoupled KV cache allows safe GPU memory oversubscription and higher tenant density." } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "rp00086", "_type": "block", "children": [ { "_key": "rp00085", "_type": "span", "marks": [], "text": "Persistent cache eliminates session stickiness: any host can serve any session with cache hits intact." } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "rp00088", "_type": "block", "children": [ { "_key": "rp00087", "_type": "span", "marks": [], "text": "Storage activates inside existing GPU servers requiring no separate infrastructure or procurement." } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" } ], "markDefs": null, "title": "The architecture decision that runs through all four" } ``` ## Where we think this goes next Based on what we've seen across these deployments, here's where AI cloud infrastructure is heading — and why the decisions made now compound in one direction or the other. ### The DRAM ceiling will become visible for more AI clouds Most AI clouds today haven't hit the DRAM saturation point yet. They will. Context windows are expanding. Agentic workloads requiring multi-turn, multi-step, persistent context are moving from pilot to production. Each step multiplies the KV cache pressure: a reasoning model needs roughly 10x more context memory than a chat model; an agentic workflow needs 10x more than a reasoning model. The AI clouds building infrastructure that handles 100,000-token context windows today will have a meaningful head start when 1-million-token contexts become standard. ### The re-architecture trap will claim its first high-profile casualties We've already seen one leading AI cloud that is now operating more than 140 petabytes go through a full storage re-architecture after an early infrastructure choice hit its performance ceiling at scale. The migration cost exceeded the original storage investment. That pattern will repeat as more AI clouds approach the 50–100 petabyte threshold. The providers who locked in the right architecture at day one won't face that event. The ones who optimized for initial cost will. ### Sovereignty will split the market EMEA enterprise buyers are moving from preferring data locality to requiring it. National AI frameworks, government procurement requirements, and sovereign cloud mandates are becoming selection criteria before technical evaluation. AI cloud providers who can demonstrate jurisdictional control of AI workloads — where model weights are stored, where KV cache lives, which legal systems govern access — will be shortlisted for contracts that others won't even see. ### Token economics will become a public metric Tokens per watt, tokens per dollar, and concurrency density per GPU will move from internal benchmarks to published differentiators — the way PUE moved from engineering metric to sales asset over the past decade in hyperscale data centers. The AI clouds that measure these numbers now, build infrastructure that optimizes for them, and can document the methodology will have a credibility advantage when buyers start asking. ## Want to see these results in your environment? Talk to WEKA about running a production benchmark on your infrastructure. Visit [contact us](/company/contact-us) to get started. Benchmark data reflects production testing environments. CoreWeave results: 64-GPU H100 cluster. OCI results: 72-GPU bare-metal H100 cluster, 100K-token context, published Oracle AI & Data Science blog, May 2026. Firmus results: production environment. Performance varies by workload configuration. [Download PDF](/api/resource-pdf?slug=what-top-performing-ai-clouds-built-differently-with-weka) ## What's next - [AI Cloud](/solutions/ai-cloud-providers) - [Eight Plays to Turn GPU Capacity Into Durable AI Cloud Margin](/resources/eguide/eight-plays-to-turn-gpu-capacity-into-durable-ai-cloud-margin) - [Built With AI Clouds: Multitenancy That Scales and Economics That Finally Work](/article/built-with-ai-clouds-multitenancy-that-scales-and-economics-that-finally-work) - [How Firmus Extended Long-Context AI Inference Beyond DRAM with WEKA](/resources/case-study/how-firmus-extended-long-context-ai-inference-beyond-dram-with-weka)