# How Firmus Extended Long-Context AI Inference Beyond DRAM with WEKA

Firmus extended long-context inference beyond DRAM with Augmented Memory Grid on NeuralMesh, reaching up to 6.5x higher tokens per second while making every watt count.

**Type:** Case Study

## Introduction

**Firmus Technologies** operates SMC Cloud in Singapore, run on their proprietary HyperCube liquid cooling platform. Built on proprietary immersion-based cooling technology, HyperCube **cuts power and costs for large-scale GPU clusters by up to 50% while saving 3,800 tons of CO2 per GPU** annually in its Singapore H100 Region.

Firmus extends its sustainability mandate to every layer of the infrastructure stack. As Dr. Daniel Kearney, CTO at Firmus, puts it, “We look at every single possible touchpoint where energy is consumed, right through from the compute itself, the thermal management, the power and the integration of the grid. We really want to make sure every watt counts.”


![Firmus engineer in a high-visibility vest walking through the HyperCube liquid-cooling facility, with piping and rack infrastructure](https://cdn.sanity.io/images/ult5g8gw/production/566eebe1736b4130ec242e23c3cee167c9ae5129-729x381.png)


## The Challenge: Delivering Long-Context Inference at Scale

GPU HBM is fast but scarce, and most inference stacks extend it with host-local DRAM for KV cache. Both are finite and bound to each host, **a hard memory ceiling** that limits context reuse and constrains the long-context and agentic workloads Firmus can support.

At scale, the challenge becomes both operational and architectural. Host-local KV cache forces a **tradeoff: preserve cache locality through session affinity, or maximize elasticity and load balancing across the cluster**. Neither is free. The result is **hotspot hosts, uneven GPU utilization, lower cache hit rates, and mounting redundant prefill costs**.

## The Foundation: WEKA NeuralMesh

Firmus had already selected WEKA NeuralMesh as its storage foundation for training and inference workloads, chosen for its **software-defined architecture, protocol flexibility across file and S3 per tenant, and up to 11x more IOPS per kilowatt** than alternatives. Augmented Memory Grid on NeuralMesh extends this further by providing a **shared memory tier that pools capacity across hosts** to overcome the per-node cache ceiling.

Swe Win Aung, Senior Manager of HPC & Cloud, Software-Defined Infrastructure at Firmus, notes that deployment speed was equally decisive: "WEKA was pretty striking in that it's performant and also software-defined, so we can just deploy it. As long as we have certified hardware, we can bring it up really fast." With NeuralMesh and Augmented Memory Grid in place, Firmus was positioned to evaluate how shared memory at scale could transform the inference workloads its customers depend on.

## Augmented Memory Grid on NeuralMesh: Maximum Cache Capacity

Mr. Aung led the evaluation in a 16-server NVIDIA H200 environment, backed by a 9-node NeuralMesh cluster, to answer one practical question: **how much more real-world long-context and agentic inference work can Firmus sustain with the same GPU footprint when the KV cache is no longer constrained by local DRAM?**

After validating basic cache-hit behavior, the team focused on the workloads that better reflect production demand: **sustained concurrent inference and replayed multi-turn coding-agent traces.**

The defining result came under sustained concurrent load. When working sets approached and exceeded DRAM limits, performance diverged sharply: **DRAM throughput collapsed as eviction pressure drove hit rates down, while WEKA’s Augmented Memory Grid remained stable**, delivering **up to 6.5x higher input token throughput** at scale.

For Firmus, that result is not just a token throughput story; it is a **tokens-per-watt story.** In a Sustainable AI Factory, power, cooling, and rack capacity are first-order constraints. By enabling cache reuse beyond local DRAM limits, Augmented Memory Grid helps each powered GPU host **deliver more long-context inference work before Firmus has to add more servers, increase power draw, or increase cooling load.**

Mr. Aung describes the team's reaction: "We pushed all the limits of how the GPU scales, how performant the storage is, and how fast the network is. The results are exciting. When I shared them with my network and internal team, the reaction was: This is really awesome. It's something a lot of people are interested in."

Augmented Memory Grid also eliminates the **cold-cache penalty** triggered by node restarts, maintenance, or cluster expansion. Because the shared tier persists independently of individual hosts, **new or returning nodes serve traffic immediately without rebuilding local state.**

## The Results: Redefining What Sustainable AI Inference Can Deliver

The results validated every dimension of the model-to-grid thesis: **higher throughput, lower latency, and greater efficiency** without additional hardware.

### Augmented Memory Grid performance:

- **Throughput at 2x DRAM working set: **6.5x higher sustained input token throughput vs. DRAM across 11 hosts
- **Agentic coding-agent responsiveness: **34% lower TTFT in replayed multi-turn coding-agent traces.
- **Cluster read/write bandwidth: **815 GB/s read, and 431 GB/s write sustained under synthetic workloads

### Infrastructure and sustainability:

- **Power efficiency: **400% more power efficient than alternatives; 11x more IOPS per kilowatt
- **CO2 savings: **3,800 tons per GPU annually in the Singapore H100 Region
- **Cooling cost reduction: **Up to 50% lower power and associated costs vs. conventional GPU clusters

Tim Rosenfield, Co-CEO and Co-Founder at Firmus, frames what that foundation enables for their customers: “Highly performant data infrastructure is essential to ensuring our customers’ demanding AI workloads operate at peak efficiency. Within our Sustainable AI Factories, a steady, frictionless data pipeline is also essential to further ensure ongoing energy savings. Because of this, WEKA’s NeuralMesh solution was a natural choice. **Their high-performance platform, combined with our energy-efficient GPU clusters, embodies our mutual commitment to not just innovate with AI but to do so responsibly.**”

## Real-World Impact

Serving larger context windows without additional GPU investment, extending the productive life of existing hardware, and reducing energy per token delivered: these are the commercial outcomes that flow from removing the DRAM ceiling. As Dr. Kearney explains, the same architectural logic applies to hardware generations: “With this capability and working with the WEKA team, we can **engineer out obsolescence**. We can take an existing GPU-based system and bring it to market ready for the next generation of workloads, without having to redeploy, throw out the old, and bring in a whole new system.”

Dr. Kearney sums up the partnership: “What I love about WEKA is the engineer engagement. They’re very much a technical entity, just like ourselves. We’re out to solve problems and bring value to customers.” Firmus is extending Augmented Memory Grid across its APAC GPU Availability Zones in Singapore, with further markets to follow.

[Download PDF](/api/resource-pdf?slug=how-firmus-extended-long-context-ai-inference-beyond-dram-with-weka)