Feature Focus: Replication Reimagined for AI

GPU capacity is one of the most constrained and expensive resources in AI. Availability and pricing shift across providers and regions, so teams reserve capacity in advance, seize spot opportunities, burst into the cloud, and move workloads wherever the economics and availability are best.

But securing the GPUs is only half the challenge. The training datasets, model checkpoints, vector indexes, and other AI data assets those workloads depend on are often anchored somewhere else, and can span hundreds of terabytes or petabytes.

Before work can begin, that data must often be staged, mirrored, or copied to the new environment. By then, the meter is already running. Every minute spent waiting for data is a minute expensive GPUs are not training models, serving inference, or generating business value. The problem is not simply dataset size. It is the assumption that all data must become local before it becomes useful.

Traditional replication was built for a different era

Replication was designed to create complete secondary copies for disaster recovery and business continuity. It still serves that purpose well, but its assumptions do not fit the way distributed AI environments operate today:

• Every destination becomes a full mirror, even when the workload only needs part of the dataset.
• Work waits for the copy to finish, putting data movement directly in the path of expensive GPU capacity.
• Object storage often becomes an intermediary, adding extra infrastructure, copies, cost, and operational complexity.

For AI infrastructure, those assumptions create unnecessary cost, delay, and operational overhead. Distributed AI environments need data to follow workloads across clouds, regions, research sites, and GPU environments, often on short notice.

A new model for AI data mobility

NeuralMesh Replication breaks the dependency between where data lives and where AI work runs. Rather than treating namespace information and file contents as the same operation, it separates them. It synchronizes the metadata that describes the source file system (namespace) independently from file contents. The destination quickly receives a browsable view of the directory structure, file names, sizes, and permissions without waiting for every file to arrive.

That architectural separation is what makes remote data useful before it's fully local.

Teams can browse the namespace, understand what exists, and decide which file contents should become local. Most data can remain remote and hydrate on demand. Selected files or directories can be copied proactively. The entire filesystem can also be hydrated when a complete replica is required.

Better yet, these approaches are not mutually exclusive. A destination can cache most files on demand while pre-hydrating selected paths, or transition to a full copy as requirements change.

Instead of forcing workloads to wait for replication, NeuralMesh Replication lets workloads determine how and when data moves.

How NeuralMesh Replication works

The mechanics for this feature are straightforward. Walking through them step-by-step:

1. Pair two clusters. Connect a source WEKA cluster to a destination through a secure trust relationship.
2. The namespace becomes visible first. Metadata describing the directory structure, file names, sizes, and permissions arrives before any file contents, making the source namespace browsable at the destination almost immediately.
3. Choose what becomes local. Set the hydration policy independently from the replication schedule. Files can be retrieved and cached on demand, proactively hydrated for selected files or directories, or copied in full when the destination requires the complete filesystem.
4. Keep the destination current. At each scheduled update, the source captures a point-in-time snapshot and transfers the changes made since the previous snapshot. Teams choose how often updates run based on how current the destination needs to be and the recovery point their environment requires.
5. Resume without starting over. If connectivity between clusters is interrupted, synchronization continues from the latest snapshot both clusters share once the connection is restored.

How NeuralMesh is built differently

The key difference is flexibility. Traditional replication typically applies one data-locality model broadly: every destination filesystem becomes a complete copy. NeuralMesh Replication lets teams select the model that fits each filesystem.

Once the destination has a browsable view of the source filesystem, teams choose how its file contents become local:

• On-demand hydration retrieves and caches files when workloads first access them.
• Selective hydration proactively copies specified files or directories.
• Full hydration maintains a complete local copy of the filesystem.

These policies can coexist across a destination cluster. One filesystem might hydrate on demand for temporary GPU workloads, another might pre-hydrate selected training data, and another might remain fully replicated for recovery or migration.

Data also moves directly between WEKA clusters without requiring object storage in the replication path.

The result is one flexible replication foundation that lets teams choose the right level of data locality for each workload, from on-demand access to selective pre-hydration to a complete local copy.

The business impact of smarter data mobility

For AI infrastructure teams, the value is not simply that data moves differently. It is that expensive compute starts producing value sooner, storage no longer scales with every destination, and distributed teams can work from shared data without rebuilding the same staging workflows at every site.

• Put available GPU capacity to work sooner. When spot capacity opens in a new region, a cloudbursting window appears, or a partner environment needs to run a job, teams can begin without waiting for a full dataset copy. The namespace is already visible, so they can identify what the workload needs and start hydrating only that data.
• Scale storage with what each filesystem requires. Not every filesystem at a destination needs to be a full mirror. Teams can use on-demand or selective hydration where only part of the data is needed, while reserving full replicas for filesystems with recovery, migration, or operational requirements.
• Collaborate across sites without copying everything everywhere. Teams in different locations can browse the same source namespace and hydrate only the files, checkpoints, or directories their local work requires. The source remains authoritative, while remote sites access what they need without maintaining a complete replica at every location.
• Replace custom staging with native data mobility. NeuralMesh Replication removes the need to rely on scripts, manual transfers, or bespoke pipelines to move data between clusters and GPU environments, reducing engineering effort, operational risk, and infrastructure complexity.

Together, these capabilities help AI teams spend less time moving and managing data and more time putting infrastructure to productive use.

Go deeper

If you're running AI workloads across more than one site, cloud, or GPU environment, the technical details matter.

Read the NeuralMesh Replication tech brief to get into the architecture: how cluster pairing works, hydration policy configuration, RPO considerations, and how it fits into an existing WEKA deployment.

To learn more, download our solution brief that goes into more detail.