Engineering Out the Ceiling
Singapore’s pioneering sustainable AI infrastructure provider used Augmented Memory Grid™ on WEKA® NeuralMesh™ to break through the DRAM ceiling, delivering up to 6.5x higher tokens per second at scale while maximizing power efficiency across every watt.
How Firmus Extended Long-Context AI Inference Beyond DRAM with WEKA
Firmus Technologies operates SMC Cloud in Singapore, run on their proprietary HyperCube liquid cooling platform. Built on proprietary immersion-based cooling technology, HyperCube cuts power and costs for large-scale GPU clusters by up to 50% while saving 3,800 tons of CO2 per GPU annually in its Singapore H100 Region.
Firmus extends its sustainability mandate to every layer of the infrastructure stack.
“We look at every single possible touchpoint where energy is consumed, right through from the compute itself, the thermal management, the power and the integration of the grid. We really want to make sure every watt counts.”
The Challenge
Delivering Long-Context Inference at Scale
GPU HBM is fast but scarce, and most inference stacks extend it with host-local DRAM for KV cache. Both are finite and bound to each host, a hard memory ceiling that limits context reuse and constrains the long-context and agentic workloads Firmus can support.
At scale, the challenge becomes both operational and architectural. Host-local KV cache forces a tradeoff: preserve cache locality through session affinity, or maximize elasticity and load balancing across the cluster. Neither is free. The result is hotspot hosts, uneven GPU utilization, lower cache hit rates, and mounting redundant prefill costs.
The Foundation
WEKA NeuralMesh
Firmus had already selected WEKA NeuralMesh as its storage foundation for training and inference workloads, chosen for its software-defined architecture, protocol flexibility across file and S3 per tenant, and up to 11x more IOPS per kilowatt than alternatives. Augmented Memory Grid on NeuralMesh extends this further by providing a shared memory tier that pools capacity across hosts to overcome the per-node cache ceiling.
With NeuralMesh and Augmented Memory Grid in place, Firmus was positioned to evaluate how shared memory at scale could transform the inference workloads its customers depend on.
“WEKA was pretty striking in that it’s performant and also software-defined, so we can just deploy it. As long as we have certified hardware, we can bring it up really fast.”
The defining result came under sustained concurrent load. When working sets approached and exceeded DRAM limits, performance diverged sharply: DRAM throughput collapsed as eviction pressure drove hit rates down, while WEKA’s Augmented Memory Grid remained stable, delivering up to 6.5x higher input token throughput at scale.
For Firmus, that result is not just a token throughput story; it is a tokens-per-watt story. In a Sustainable AI Factory, power, cooling, and rack capacity are first-order constraints. By enabling cache reuse beyond local DRAM limits, Augmented Memory Grid helps each powered GPU host deliver more long-context inference work before Firmus has to add more servers, increase power draw, or increase cooling load.
Mr. Aung describes the team’s reaction: “We pushed all the limits of how the GPU scales, how performant the storage is, and how fast the network is. The results are exciting. When I shared them with my network and internal team, the reaction was: This is really awesome. It’s something a lot of people are interested in.”
Augmented Memory Grid also eliminates the cold-cache penalty triggered by node restarts, maintenance, or cluster expansion. Because the shared tier persists independently of individual hosts, new or returning nodes serve traffic immediately without rebuilding local state.
The Results
Redefining What Sustainable AI Inference Can Deliver
The results validated every dimension of the model-to-grid thesis: higher throughput, lower latency, and greater efficiency without additional hardware.
Augmented Memory Grid performance:
- Throughput at 2x DRAM working set: 6.5x higher sustained input token throughput vs. DRAM across 11 hosts
- Agentic coding-agent responsiveness: 34% lower TTFT in replayed multi-turn coding-agent traces.
- Cluster read/write bandwidth: 815 GB/s read, and 431 GB/s write sustained under synthetic workloads
Infrastructure and sustainability:
- Power efficiency: 400% more power efficient than alternatives; 11x more IOPS per kilowatt
- CO2 savings: 3,800 tons per GPU annually in the Singapore H100 Region
- Cooling cost reduction: Up to 50% lower power and associated costs vs. conventional GPU clusters
“Highly performant data infrastructure is essential to ensuring our customers’ demanding AI workloads operate at peak efficiency. Within our Sustainable AI Factories, a steady, frictionless data pipeline is also essential to further ensure ongoing energy savings. Because of this, WEKA’s NeuralMesh solution was a natural choice. Their high-performance platform, combined with our energy-efficient GPU clusters, embodies our mutual commitment to not just innovate with AI but to do so responsibly.”
Real-World Impact
Commercial outcomes from removing the DRAM ceiling
Serving larger context windows without additional GPU investment, extending the productive life of existing hardware, and reducing energy per token delivered: these are the commercial outcomes that flow from removing the DRAM ceiling. As Dr. Kearney explains, the same architectural logic applies to hardware generations: “With this capability and working with the WEKA team, we can engineer out obsolescence. We can take an existing GPU-based system and bring it to market ready for the next generation of workloads, without having to redeploy, throw out the old, and bring in a whole new system.”
Firmus is extending Augmented Memory Grid across its APAC GPU Availability Zones in Singapore, with further markets to follow.
“What I love about WEKA is the engineer engagement. They’re very much a technical entity, just like ourselves. We’re out to solve problems and bring value to customers.”
Maximize Power Efficiency Across Every Watt
Learn more about Firmus and WEKA
Dive a little deeper
