# HTX, Singtel RE:AI and WEKA Run Sovereign Physical AI at Full Speed

**Author:** Ronald Pereira

**Published:** October 30, 2026

![Futuristic grayscale server room with glowing data streams.](https://cdn.sanity.io/images/ult5g8gw/production/f2dbc8eecdc99a4bb036225a3057234af01186c5-1584x672.jpg)

## TL;DR

Explore how RE:AI and WEKA engineered a cluster to keep GPUs productive for sovereign physical AI.

- Achieve 99.49% accelerator utilisation with multitenancy enabled.
- Deliver 7.7 GB/s read and 6.0 GB/s write per GPU.
- Scale tenants and racks with predictable performance.

_By RE:AI and WEKA_

## Clusters built for hard problems

HTX is building an AI-enabled, sovereign Home Team. Under the five-year Strategic Partnership for Innovation that HTX renewed with Singtel in February 2026, RE:AI, Singtel Digital InfraCo’s sovereign AI cloud business, provides the AI infrastructure behind HTX’s growing requirements, keeping latency-sensitive applications inside trusted local environments. RE:AI designs and runs the platform end to end, from facility and fabric through GPUs, Paragon platform powered orchestration, tenancy and managed service. NeuralMesh is the data layer inside it.

HTX's NGINE platform is organised into purpose-built engines that run on RE:AI infrastructure: Physical AI tier, is where robotics and real-time vision models are trained and refined.

The Physical AI clusters are an initial build, designed to scale. Physical AI training streams large multimodal datasets, writes large checkpoints often, and exposes any part of the system that can’t keep up. So the cluster was engineered as a system, not a collection of parts: NVIDIA GB300 NVL72 compute, a ConnectX-8 InfiniBand fabric for GPU-to-GPU traffic, a separate BlueField-3 RoCE fabric for storage, and NeuralMesh as the data layer.

Validated together, that stack delivered 7.7 GB/s of read and 6.0 GB/s of write bandwidth to every GPU with the full fleet running, 155 µs read latency, 29.7 million random read IOPS and 99.49% accelerator utilisation.

## Why GPU utilization is the number that matters

GPUs are the most expensive and most constrained resource in any AI platform, and they only create value while they’re working. Every minute a GPU spends waiting on data is training that doesn’t happen, a model that ships later and an insight that arrives late. That is why the useful question isn’t what a storage system peaks at on a datasheet. It’s what storage delivers to every node, and every GPU in that node, when the whole fleet is busy at once. Utilisation is uptime, uptime is work completed, and work completed is the return on the platform. A data layer that keeps GPUs fed turns more of the same compute investment into results. One that stalls charges the cost to every GPU in the job.

## Test parameters

The system under test was one of RE:AI’s clusters: multiple racks of NVIDIA GB300 NVL72, served by a single NeuralMesh cluster. Every run was made from inside a tenant network space with NeuralMesh native multitenancy enabled, which is the configuration the platform runs in production.

```json
{
  "_key": "efabc0108d54",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "975cdb9821b2",
      "_type": "row",
      "cells": [
        {
          "_key": "1ce82cdf7364",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "08e1283a09f6",
              "_type": "block",
              "children": [
                {
                  "_key": "012a18e42b63",
                  "_type": "span",
                  "marks": [],
                  "text": "Benchmark"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "728ebb33337b",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "54d90b820812",
              "_type": "block",
              "children": [
                {
                  "_key": "0b23dcd19c29",
                  "_type": "span",
                  "marks": [],
                  "text": "Configuration"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "9e5f07748e52",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "fc8a8320d729",
              "_type": "block",
              "children": [
                {
                  "_key": "900f58704ab4",
                  "_type": "span",
                  "marks": [],
                  "text": "What it shows"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "e714b8574cce",
      "_type": "row",
      "cells": [
        {
          "_key": "2390472f52c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "063eb1b4d766",
              "_type": "block",
              "children": [
                {
                  "_key": "04dd9635f769",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Data layer (FIO)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "173d95a8f19b",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "391b25be4517",
              "_type": "block",
              "children": [
                {
                  "_key": "4517a95f09a1",
                  "_type": "span",
                  "marks": [],
                  "text": "FIO ran directly against the filesystem on the GB300 compute nodes, with 16 CPU cores per node assigned to the NeuralMesh client."
                }
              ],
              "markDefs": [],
              "style": "normal"
            },
            {
              "_key": "2f5a25512754",
              "_type": "block",
              "children": [
                {
                  "_key": "a3a9d8a3a0e4",
                  "_type": "span",
                  "marks": [],
                  "text": "Each job used libaio with direct I/O, read or wrote 10 GB and ran for 30 seconds. Jobs were spread across the selected nodes, and results were aggregated per run."
                }
              ],
              "markDefs": [],
              "style": "normal"
            },
            {
              "_key": "46004beaf47a",
              "_type": "block",
              "children": [
                {
                  "_key": "9ee4b5e3e970",
                  "_type": "span",
                  "marks": [],
                  "text": "Bandwidth tests used 1 MiB sequential I/O with 32 jobs per node. IOPS and latency tests used 4 KB random reads."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "40631b85e032",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "330171b2e9ab",
              "_type": "block",
              "children": [
                {
                  "_key": "3472b696f9e6",
                  "_type": "span",
                  "marks": [],
                  "text": "It shows what the data layer can deliver to a node. No model, framework or data loader was in the path, so these are not figures for any single training job."
                }
              ],
              "markDefs": [],
              "style": "normal"
            },
            {
              "_key": "db59181beb9e",
              "_type": "block",
              "children": [
                {
                  "_key": "b65964b7fb79",
                  "_type": "span",
                  "marks": [],
                  "text": "Per-GPU figures are the per-node result divided by the four GPUs in a GB300 compute node."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "bf57a7db235e",
      "_type": "row",
      "cells": [
        {
          "_key": "88cedff9344b",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "951461f7afbb",
              "_type": "block",
              "children": [
                {
                  "_key": "4bbc1dba4628",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Accelerator utilization (MLPerf Storage)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "251a75c70ba2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "fc8a3b5fbdf3",
              "_type": "block",
              "children": [
                {
                  "_key": "7674c607f35d",
                  "_type": "span",
                  "marks": [],
                  "text": "Singtel RE:AI ran MLPerf Storage using the MLCommons DLIO harness, which replays a real training job’s I/O pattern without running GPU compute."
                }
              ],
              "markDefs": [],
              "style": "normal"
            },
            {
              "_key": "8a1eb4452d8f",
              "_type": "block",
              "children": [
                {
                  "_key": "8ea3433c9093",
                  "_type": "span",
                  "marks": [],
                  "text": "The test replayed a Unet3D training workload for four simulated accelerators over five epochs on a single node."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "c9e0de859c50",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "57e6e7072de4",
              "_type": "block",
              "children": [
                {
                  "_key": "8d5b7c084925",
                  "_type": "span",
                  "marks": [],
                  "text": "It shows whether the simulated accelerators ever wait on storage."
                }
              ],
              "markDefs": [],
              "style": "normal"
            },
            {
              "_key": "f0fb84780dd7",
              "_type": "block",
              "children": [
                {
                  "_key": "a5739076b63a",
                  "_type": "span",
                  "marks": [],
                  "text": "This is an unverified MLPerf Storage result that MLCommons has not reviewed."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

_All runs are recorded in RE:AI’s HTX Cluster 1 GB300 NVL72 Handover Report v1.0, 17 August 2026._

## Results at a glance

```json
{
  "_key": "08bd1a1834ce",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "4a56da05f5d7",
      "_type": "row",
      "cells": [
        {
          "_key": "101d65031ede",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "90867f2303db",
              "_type": "block",
              "children": [
                {
                  "_key": "e2d96a880b46",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Measure"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "63fb6ae8c216",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "f07e3a434dc4",
              "_type": "block",
              "children": [
                {
                  "_key": "348df6732e92",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Single node (4 GPUs)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "b210f0660c63",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "19125b3799bc",
              "_type": "block",
              "children": [
                {
                  "_key": "41bf13e76969",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "One rack"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "d89b6ee5e63a",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "954bf8e65854",
              "_type": "block",
              "children": [
                {
                  "_key": "fe601c8e4159",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Full cluster"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "7abdda76c3ec",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "f409ff1daad7",
              "_type": "block",
              "children": [
                {
                  "_key": "f690df99bfaf",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Against the standard"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "2ccff01f616f",
      "_type": "row",
      "cells": [
        {
          "_key": "6564fc77bdce",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "ff1640ac4054",
              "_type": "block",
              "children": [
                {
                  "_key": "58a678c29b10",
                  "_type": "span",
                  "marks": [],
                  "text": "Sequential read"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "48cbb3134e3c",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "4c0743943d09",
              "_type": "block",
              "children": [
                {
                  "_key": "ac6d7a67f708",
                  "_type": "span",
                  "marks": [],
                  "text": "46.35 GB/s"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "55136750266a",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "7e8efd2281cb",
              "_type": "block",
              "children": [
                {
                  "_key": "8d177cc81bd8",
                  "_type": "span",
                  "marks": [],
                  "text": "706–714 GB/s"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "b1ecb92158fc",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "7f1c5d2d3cf7",
              "_type": "block",
              "children": [
                {
                  "_key": "e7c743f42a2c",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "1.11 TB/s"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "7273880510eb",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "5a90595e0a41",
              "_type": "block",
              "children": [
                {
                  "_key": "7f31003e65f7",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "1.9x"
                },
                {
                  "_key": "0cf1b9046fd2",
                  "_type": "span",
                  "marks": [],
                  "text": " the guideline: 7.7 GB/s per GPU vs. 4 GB/s, with every node running."
                }
              ],
              "markDefs": [],
              "style": "normal"
            },
            {
              "_key": "d48cbd9e87ef",
              "_type": "block",
              "children": [
                {
                  "_key": "15f4ee0e919f",
                  "_type": "span",
                  "marks": [],
                  "text": "A single node reaches 93% of its 400 Gb/s storage link."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "2b7da0fa617c",
      "_type": "row",
      "cells": [
        {
          "_key": "751b995c7738",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "c535266a2aff",
              "_type": "block",
              "children": [
                {
                  "_key": "4519d8685c59",
                  "_type": "span",
                  "marks": [],
                  "text": "Sequential write"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "e24be43aca3e",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "9da7c1d88dcc",
              "_type": "block",
              "children": [
                {
                  "_key": "2c455b5447b1",
                  "_type": "span",
                  "marks": [],
                  "text": "28.60 GB/s"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "a8e37f260f04",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "d48e65fef41c",
              "_type": "block",
              "children": [
                {
                  "_key": "4fa013d8b84e",
                  "_type": "span",
                  "marks": [],
                  "text": "531–533 GB/s"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "8791a01f3000",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "78dfc3bc70b7",
              "_type": "block",
              "children": [
                {
                  "_key": "c845822876ba",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "864 GB/s"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "8a1c8f0083d2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "7f00b8615fd1",
              "_type": "block",
              "children": [
                {
                  "_key": "4758c66287fd",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "3x"
                },
                {
                  "_key": "bde6b080969b",
                  "_type": "span",
                  "marks": [],
                  "text": " the guideline: 6.0 GB/s per GPU vs. 2 GB/s, with every node running."
                }
              ],
              "markDefs": [],
              "style": "normal"
            },
            {
              "_key": "177805a03f9b",
              "_type": "block",
              "children": [
                {
                  "_key": "626517acf0d8",
                  "_type": "span",
                  "marks": [],
                  "text": "Checkpoints finish in a fraction of the time, so GPUs get back to training sooner."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "c7e8a6b91a66",
      "_type": "row",
      "cells": [
        {
          "_key": "20286209a32e",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "06178ec764e5",
              "_type": "block",
              "children": [
                {
                  "_key": "8e53505273d7",
                  "_type": "span",
                  "marks": [],
                  "text": "Read latency, 4 KB random"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "e776342b2e1e",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "ec49e42410fa",
              "_type": "block",
              "children": [
                {
                  "_key": "00d6a1477744",
                  "_type": "span",
                  "marks": [],
                  "text": "195 µs"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "c99796eca9db",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "6a47e08aefd8",
              "_type": "block",
              "children": [
                {
                  "_key": "9fbc68ddb9e7",
                  "_type": "span",
                  "marks": [],
                  "text": "156–157 µs"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "8e630165cb75",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "461368aa5a9c",
              "_type": "block",
              "children": [
                {
                  "_key": "83dc1cc71147",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "155 µs"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "173f3b364ac8",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "2a8667d25857",
              "_type": "block",
              "children": [
                {
                  "_key": "8f647abdd720",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Flat"
                },
                {
                  "_key": "bd75cc0b2a88",
                  "_type": "span",
                  "marks": [],
                  "text": " from one rack to the full cluster."
                }
              ],
              "markDefs": [],
              "style": "normal"
            },
            {
              "_key": "da580c4a6a16",
              "_type": "block",
              "children": [
                {
                  "_key": "abef64ed7448",
                  "_type": "span",
                  "marks": [],
                  "text": "Latency doesn’t rise as client count grows, so shared load doesn’t slow jobs."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "3f59a31ed484",
      "_type": "row",
      "cells": [
        {
          "_key": "2c98513b912e",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "45445b2a5065",
              "_type": "block",
              "children": [
                {
                  "_key": "f45098488e34",
                  "_type": "span",
                  "marks": [],
                  "text": "Random read IOPS, 4 KB"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "4843bd075bd3",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "2f508a5cc749",
              "_type": "block",
              "children": [
                {
                  "_key": "4316e3452695",
                  "_type": "span",
                  "marks": [],
                  "text": "1.31 M"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "ee366ee0bd48",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "64797c0baebe",
              "_type": "block",
              "children": [
                {
                  "_key": "9a1db3c542e4",
                  "_type": "span",
                  "marks": [],
                  "text": "19.9–22.5 M"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "f7fbefc10e14",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "2c3bfdbc8386",
              "_type": "block",
              "children": [
                {
                  "_key": "4ba43bf7bd46",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "29.7 M"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "73904d4f9150",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "8d88b7c489a7",
              "_type": "block",
              "children": [
                {
                  "_key": "62484b583ecb",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "About 1.6 M"
                },
                {
                  "_key": "d78e99cac541",
                  "_type": "span",
                  "marks": [],
                  "text": " per NeuralMesh node."
                }
              ],
              "markDefs": [],
              "style": "normal"
            },
            {
              "_key": "28b844ae46e0",
              "_type": "block",
              "children": [
                {
                  "_key": "75109251207f",
                  "_type": "span",
                  "marks": [],
                  "text": "Ample headroom for data shuffles and metadata-heavy pipelines."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "d88796d36445",
      "_type": "row",
      "cells": [
        {
          "_key": "aaef2b6ca7ac",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "78a788ab5a6e",
              "_type": "block",
              "children": [
                {
                  "_key": "e95f2a9d9dc8",
                  "_type": "span",
                  "marks": [],
                  "text": "Accelerator utilization, MLPerf Storage (Unet3D)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "c2569f4ddaf0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "d07207a2bacb",
              "_type": "block",
              "children": [
                {
                  "_key": "bfc4505b8507",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "99.49%"
                },
                {
                  "_key": "663c5898a073",
                  "_type": "span",
                  "marks": [],
                  "text": " at 24 GB/s sustained"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "d7c57614a120",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "c19d4425f16a",
              "_type": "block",
              "children": [
                {
                  "_key": "60703b6201a8",
                  "_type": "span",
                  "marks": [],
                  "text": "—"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "296dd684bb54",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "e1d27e0375ea",
              "_type": "block",
              "children": [
                {
                  "_key": "608bed94e483",
                  "_type": "span",
                  "marks": [],
                  "text": "—"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "3594722fae01",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "ca2388778e4d",
              "_type": "block",
              "children": [
                {
                  "_key": "7c5317cc9e76",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "99.49%"
                },
                {
                  "_key": "a31754eb5ddc",
                  "_type": "span",
                  "marks": [],
                  "text": " vs. the MLCommons pass threshold of 90%."
                }
              ],
              "markDefs": [],
              "style": "normal"
            },
            {
              "_key": "9da9d72f1a87",
              "_type": "block",
              "children": [
                {
                  "_key": "424b42cd14c5",
                  "_type": "span",
                  "marks": [],
                  "text": "The accelerators effectively never waited on storage."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

_Rack figures show the range across racks, each tested separately. Per-GPU figures are the full-cluster result divided by the cluster’s total GPU count. Backend ceiling at full load: 58.4 GB/s read and 45.5 GB/s write per NeuralMesh storage node._

## Interpreting the results

Each result answers a practical question an operator has to get right when sizing such a platform.

1. **Every GPU gets fed with the whole fleet running**
   1. **What we tested: **sequential read and write from every compute node at the same time.
   2. **Result: **7.7 GB/s read and 6.0 GB/s write per GPU. NVIDIA’s DGX SuperPOD reference architecture sets 4 GB/s per GPU of read bandwidth for the most I/O-intensive training, with write at half of read (2 GB/s) or better where checkpoints matter. The cluster cleared the read guideline by almost 2x and the write guideline by 3x.
   3. **What it means: **headroom for data loading and checkpointing, and room for larger models and datasets without re-architecting storage.
2. **Peak performance is available to any single node**
   1. **What we tested: **one node running alone, to see how much a single job can pull when it needs a burst.
   2. **Result: **46.35 GB/s read, about 93% of the node’s 400 Gb/s storage port. (Note: the node's two 400 Gb/s ports share a PCIe link, so ~400 Gb/s is the practical ceiling).
   3. **What it means: **storage isn’t the ceiling for an individual job. When a node needs to load a dataset or restore a checkpoint, it gets close to line rate, and time spent waiting is cut to the minimum the network allows.
3. **Performance scales predictably**
   1. **What we tested: **scaling from a single node to nearly the full cluster, and each rack on its own.
   2. **Result: **throughput rose almost linearly with client count, then levelled off at the ceiling of the 19-node NeuralMesh backend: 58.4 GB/s read and 45.5 GB/s write per NeuralMesh node. Tested separately, the racks came within about 1% of each other (706 and 714 GB/s read).
   3. **What it means: **fleet-wide throughput is set by the number of NeuralMesh nodes, not by how many clients mount the filesystem, so every new GPU rack comes with a clear number of storage nodes to match.
4. **Latency stays flat under load**
   1. **What we tested: **4 KB random reads from one rack up to the full cluster.
   2. **Result: **read latency held between 155 and 157 µs from one rack up to the full cluster, with 29.7 million random read IOPS at full load.
   3. **What it means: **heavier shared use doesn’t turn into slower jobs. Adding clients doesn’t add latency, which matters for small-file and random-access work like data shuffles and metadata-heavy pipelines.
5. **The GPUs stayed busy**
   1. **What we tested: **MLPerf Storage replayed the I/O of a Unet3D training job for four accelerators over five epochs on a single node.
   2. **Result: **accelerator utilisation averaged 99.49% (standard deviation 0.13%) at a sustained 24 GB/s, well above the MLCommons pass threshold.
   3. **What it means: **the accelerators never waited on storage. Nearly every GPU-hour the platform pays for goes to training, not to waiting on data.

## Full performance with multitenancy switched on

Shared AI platforms usually force a trade-off. Isolate tenants with dedicated hardware and accept idle capacity, or share infrastructure and accept weaker boundaries and noisy neighbours. For a sovereign platform serving government workloads, neither is good enough.

NeuralMesh runs multitenancy natively inside the cluster. Each tenant is a first-class entity with its own lifecycle, capacity quota, security policies and administrator. Tenants share one elastic resource pool, so capacity one tenant isn’t using is available to others instead of sitting idle, and removing a tenant returns its resources to the pool immediately.

Tenant networks are built as network spaces on WEKA’s Virtualized RDMA Data Fabric (VRDF), which enforces isolation inside the RDMA data plane rather than relying on host-level segmentation alone. In practice, that gives each tenant:

- **A private network context.** Dedicated VLANs and IP ranges, including overlapping address spaces across tenants, so onboarding a new tenant doesn’t mean redesigning the network.
- **Enforcement in the data path.** The origin of every mount request is validated. Requests from outside a tenant’s network space are rejected, even with valid credentials, which limits the damage a misconfiguration can do.
- **Independent security controls.** Each tenant can use its own identity provider (LDAP) and encryption keys (KMS), with its own administrators who have no visibility into other tenants.
- **Predictable performance.** Per-tenant quality-of-service limits keep any one tenant from crowding out the others on the shared cluster.
- On the deployed 5.1 release, network-space isolation covers POSIX and S3.

For HTX, the key point is that every benchmark ran with multitenancy enabled. Isolation didn’t cost performance. NGINE Physical AI GPU platform is designed to add tenants over time, and each one can come online in the same cluster without a new storage silo and without sacrificing performance.

## Sizing for peak, not average

AI I/O is bursty. Checkpoints, epoch boundaries, dataset shuffles and context loads all arrive as short bursts, synchronised across the fleet. A platform sized for the average handles the average, then stalls at every burst. That stall is paid in GPU time on every node in the job, not only the one that’s waiting.

So size for the burst. Calculate how much data the largest job writes at a checkpoint, and how long that job can afford to pause while it does. Here’s what that looks like for one rack. A full-rack job spans 18 nodes and 72 GPUs. Say it writes a 5 TB checkpoint, and every GPU in the job waits until the write finishes.

```json
{
  "_key": "62feee33e712",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "4803b1ea3db9",
      "_type": "row",
      "cells": [
        {
          "_key": "c597a7beb31c",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "e28f07aa0f05",
              "_type": "block",
              "children": [
                {
                  "_key": "2b17689e6922",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Storage write bandwidth for the rack"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "be701ad27bcb",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "3732949c51b1",
              "_type": "block",
              "children": [
                {
                  "_key": "1fd59723fba3",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Time to write a 5 TB checkpoint"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "07df29457f02",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "1c69c1b30a46",
              "_type": "block",
              "children": [
                {
                  "_key": "672bea57a34f",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "GPUs waiting"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "d501cb9afd8f",
      "_type": "row",
      "cells": [
        {
          "_key": "01c7f8d761fa",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "b5a747dfa371",
              "_type": "block",
              "children": [
                {
                  "_key": "9daf41df471d",
                  "_type": "span",
                  "marks": [],
                  "text": "531 GB/s, as measured on this cluster"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "b30e238223df",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "0ac2d6560cf2",
              "_type": "block",
              "children": [
                {
                  "_key": "dfef4aa69a01",
                  "_type": "span",
                  "marks": [],
                  "text": "About 9 seconds (5,000 GB ÷ 531 GB/s)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "22628699f1ff",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "1bae9ee20732",
              "_type": "block",
              "children": [
                {
                  "_key": "25e488851c15",
                  "_type": "span",
                  "marks": [],
                  "text": "72"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "c26a14b098a6",
      "_type": "row",
      "cells": [
        {
          "_key": "e58079d38623",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "dc284c41c513",
              "_type": "block",
              "children": [
                {
                  "_key": "1ea9da2ba96e",
                  "_type": "span",
                  "marks": [],
                  "text": "144 GB/s, the reference-architecture minimum (72 GPUs × 2 GB/s)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "bb0f356b8150",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "fad23933387a",
              "_type": "block",
              "children": [
                {
                  "_key": "4f4e6d704b50",
                  "_type": "span",
                  "marks": [],
                  "text": "About 35 seconds (5,000 GB ÷ 144 GB/s)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "d9aca33ad356",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "1bbc5079623b",
              "_type": "block",
              "children": [
                {
                  "_key": "f56407c94d17",
                  "_type": "span",
                  "marks": [],
                  "text": "72"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

The difference is about 25 seconds for 72 GPUs, or roughly 30 GPU-minutes lost at every checkpoint. If the job checkpoints once an hour, that adds up to about 12 GPU-hours a day, per rack, spent waiting instead of training. Sizing for peak gets that time back.

## How WEKA and RE:AI engineered it

Storage performance at this level is a property of the whole system. A data layer can’t deliver line rate into a node that isn’t set up to receive it, so the two teams divided the work clearly and agreed one approach from the start: design the storage path end to end for maximum throughput per node, then prove every part of it by measurement before handover.

RE:AI owned the compute side: host networking, fabric layout and the configuration of every GPU node. WEKA owned the data layer: deploying and configuring NeuralMesh, mapping its tenant network spaces onto RE:AI’s VLAN design, and running and diagnosing the benchmark programme. These were the decisions that made the difference.

```json
{
  "_key": "f4d099bed387",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "fb7809650040",
      "_type": "row",
      "cells": [
        {
          "_key": "83abc80174cb",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "50a1c0c70b54",
              "_type": "block",
              "children": [
                {
                  "_key": "22988050bf3a",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Decision"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "28364c478d6c",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "a11a7ba8dc07",
              "_type": "block",
              "children": [
                {
                  "_key": "5ee981d2636d",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "What was done"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "f4fefc434087",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "caef87354420",
              "_type": "block",
              "children": [
                {
                  "_key": "0497b5ea68df",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Outcome"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "81c701b20b2f",
      "_type": "row",
      "cells": [
        {
          "_key": "0c496bc51e59",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "c56e9673adbc",
              "_type": "block",
              "children": [
                {
                  "_key": "37f998398806",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Separate storage fabric"
                },
                {
                  "_key": "8e8f082014ce",
                  "_type": "span",
                  "marks": [],
                  "text": " (RE:AI)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "493e9ff6b141",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "dd1c3a07e78b",
              "_type": "block",
              "children": [
                {
                  "_key": "81f890366d42",
                  "_type": "span",
                  "marks": [],
                  "text": "Storage traffic runs on each node’s BlueField-3 Ethernet port over RoCE, kept apart from the four ConnectX-8 InfiniBand rails that carry GPU-to-GPU (NCCL) traffic."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "229e9aea4728",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "3f39aa249195",
              "_type": "block",
              "children": [
                {
                  "_key": "85f576b730f0",
                  "_type": "span",
                  "marks": [],
                  "text": "Checkpoint bursts and GPU collectives never compete for the same links."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "d35b926534da",
      "_type": "row",
      "cells": [
        {
          "_key": "974432c209ec",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "cf540f4fe438",
              "_type": "block",
              "children": [
                {
                  "_key": "b809ea7fcf73",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Networking that matches the filesystem"
                },
                {
                  "_key": "aa8a9bb9b5f0",
                  "_type": "span",
                  "marks": [],
                  "text": " (RE:AI and WEKA)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "9345055d5b0b",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "30ba7bf4a05e",
              "_type": "block",
              "children": [
                {
                  "_key": "e88adf442f0b",
                  "_type": "span",
                  "marks": [],
                  "text": "Bonded ports with LACP on the storage path, and one VLAN for each NeuralMesh network space."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "edf772ee7e6c",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "ba2836c2438c",
              "_type": "block",
              "children": [
                {
                  "_key": "0a65c334af4c",
                  "_type": "span",
                  "marks": [],
                  "text": "Tenant isolation is the same object on the switch, the host and the filesystem, so there is nothing to reconcile and nothing to drift."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "85a7ae06c6ec",
      "_type": "row",
      "cells": [
        {
          "_key": "b4884bed36ac",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "e584665ceb03",
              "_type": "block",
              "children": [
                {
                  "_key": "bf84a003d6b5",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Dedicated, well-placed client cores"
                },
                {
                  "_key": "d92372c25790",
                  "_type": "span",
                  "marks": [],
                  "text": " (RE:AI and WEKA)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "e535b98944a7",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "adf0480f2e11",
              "_type": "block",
              "children": [
                {
                  "_key": "32a8575783e9",
                  "_type": "span",
                  "marks": [],
                  "text": "16 CPU cores on each node reserved for the NeuralMesh client, on the same memory domain (NUMA node) as the storage NIC. GB300 hosts expose 34 NUMA nodes, and data that crosses between them loses bandwidth and gains latency."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "9a81fbb491ba",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "3a7914981a6a",
              "_type": "block",
              "children": [
                {
                  "_key": "508f7ffa78b1",
                  "_type": "span",
                  "marks": [],
                  "text": "Storage I/O doesn’t compete with application CPU, and data takes the shortest path from network to memory."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "b577fc124050",
      "_type": "row",
      "cells": [
        {
          "_key": "bfad02c25bee",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "37a96ec48f75",
              "_type": "block",
              "children": [
                {
                  "_key": "155cb354181a",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Configuration built into the node image"
                },
                {
                  "_key": "e2fc8bedfb4a",
                  "_type": "span",
                  "marks": [],
                  "text": " (RE:AI)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "cca6cd64353a",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "ce035e204c47",
              "_type": "block",
              "children": [
                {
                  "_key": "2be2f9854bc6",
                  "_type": "span",
                  "marks": [],
                  "text": "NeuralMesh client settings ship in RE:AI’s stateless node image."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "2af962fc19f8",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "3d125618f47b",
              "_type": "block",
              "children": [
                {
                  "_key": "234d64612f5a",
                  "_type": "span",
                  "marks": [],
                  "text": "Every node boots with identical settings, and none can drift out of tune."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "a6c5450e8260",
      "_type": "row",
      "cells": [
        {
          "_key": "282fa989f3f4",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "c18fab0cb97d",
              "_type": "block",
              "children": [
                {
                  "_key": "52628329b9df",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Measure, fix, measure again"
                },
                {
                  "_key": "8b2c30435b3e",
                  "_type": "span",
                  "marks": [],
                  "text": " (joint)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "371c2c137795",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "e7f759e31621",
              "_type": "block",
              "children": [
                {
                  "_key": "85bc23d62054",
                  "_type": "span",
                  "marks": [],
                  "text": "The first single-node test returned 37 GB/s read. The teams traced the gap to storage-path configuration and corrected it."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "606d1d56c73f",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "196c563ac1ed",
              "_type": "block",
              "children": [
                {
                  "_key": "5776370301c5",
                  "_type": "span",
                  "marks": [],
                  "text": "The same node reached 46.35 GB/s, about 93% of its 400 Gb/s port, and the full cluster reached 1.11 TB/s."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

Before the service went live, the teams also swept concurrency from 4 to 16 jobs per node, tested each rack separately and exercised failover.

## What this means as NGINE grows

RE:AI NGINE starts compact, and it’s built to grow. The results from this validation testing indicate how that growth will behave.

1. **Adding tenants won’t cost performance.** Every result here was measured with multitenancy on. Because tenants share one pool rather than reserving fixed slices, a new tenant adds demand without stranding capacity.
2. **Adding load won’t slow existing work.** Read latency held at 155–157 µs from one rack to the full cluster, so heavier shared use doesn’t become slower jobs.
3. **Adding racks is simple math.** Each NeuralMesh node contributed about 58 GB/s read and 45 GB/s write, so every new GPU cluster comes with a clear number of storage nodes to match.

For HTX, that means the Physical AI tier can scale on a platform that keeps its most valuable resource, the GPU, doing useful work. For RE:AI, it’s a repeatable blueprint for a regulated, multitenant sovereign AI zone: isolation enforced in the data plane rather than only at the host, a host configuration RE:AI owns and can reproduce on every node, and performance measured on the production configuration rather than by datasheets.

## References

- [_HTX and Singtel Strategic Partnership for Innovation renewal, Feb. 2, 2026_](http://singtel.com/about-us/media-centre/news-releases/htx-and-singtel-group-deepen-partnership-to-strengthen-singapore)
- [_NGINE launch, HTX, August 2025_](http://htx.gov.sg/whats-happening/all-news%E2%80%94events/all-news/2025/the-ngine-behind-htx-ai-ambitions)
- [_Singtel RE:AI and WEKA sovereign AI infrastructure MOU, June 10, 2026_](/pressRelease/singtel-reai-and-weka-partner-to-deliver-sovereign-ai-infrastructure)
- [_NeuralMesh multitenancy technical brief_](/resource/neuralmesh-multitenancy)
- [_NVIDIA DGX SuperPOD Reference Architecture (NVIDIA DGX™ GB200), Storage Architecture_](http://docs.nvidia.com/dgx-superpod)