Feature Focus: S3 Object Storage That Never Makes GPUs Wait

Object storage sits in the middle of the AI pipeline, supporting training data sets, checkpoints, and massively concurrent inference sessions. Training reads millions of small files in unpredictable, metadata-heavy lookups. Checkpointing writes multi-terabyte sequential dumps every few minutes, often from thousands of GPUs at once. Inference reads under strict latency budgets, with thousands of concurrent sessions hitting the same objects at the same time. Most S3 object stores were built for one of those patterns, not all three at once.

Today, we announce WEKA® NeuralMesh™ Fast S3 Object Storage. It runs on the same distributed architecture as every other NeuralMesh protocol, with no centralized metadata and no single point of failure to break under load. S3 and POSIX operate on the same physical data, so a file written through one protocol reads instantly through the other. And an S3 Object Store covering performance, data management, security, and multitenancy targets the exact operations AI pipelines run continuously. On its own, this architecture already removes the ceilings that stall conventional S3 object stores. When combined with the new, next-generation WEKApod™ systems, announced earlier today, that same architecture reaches rack-scale numbers no general-purpose hardware platform can match.

Why conventional S3 object stores stall under GPU clusters

Most S3 object stores concentrate metadata in a small number of controller nodes and manage client connections through a shared pool, both sized for sequential access by a modest number of clients. GPU clusters running hundreds of simultaneous jobs for AI training, inference, or both, blow past that design point. Once the connection ceiling hits, typically around 1,000 connections per node, request queuing cascades through every job on the cluster, not just the one that caused it. The metadata controllers fail next: the millions of small-object operations that AI checkpointing and RAG pipelines generate overwhelm a design built for large sequential reads.

NeuralMesh removes the ceiling instead of raising it. Metadata and data are distributed across every node using deterministic hash-based placement, so there's no controller to saturate and no connection pool to share. NeuralMesh Fast S3 Object Storage supports 5x the conventional limit on concurrent connections per node¹, and performance scales linearly as nodes are added, from six to hundreds, because there's no centralized component for GPU cluster concurrency to overwhelm. Y

The same distribution is what makes the system harder to disrupt as it scales, not more fragile, which matters for S3 specifically because tenant SLA commitments are exactly what break when an object store degrades under load. More nodes means more independent failure domains, so a single node event stays a single node event. On the next-generation WEKApod systems announced today, a cable-based, backplane-free drive interconnect carries that isolation down to the physical drive level, and the density compounds too: WEKApod Nitro delivers 10.2 TB/s of throughput and 210 million IOPS per rack, 3 to 4x greater rack-scale S3 throughput than the nearest published alternative, and WEKApod Prime Max packs 1.1 exabytes of effective capacity into a single rack, the first system to cross 1 EB in one.

Why S3 object stores force redundant copies

A conventional AI pipeline, ingest, ETL, train, validate, infer, archive, copies the full dataset at multiple stage boundaries before a model ships, sometimes two or three, sometimes five or more, depending on the pipeline. Every copy is a sequencing dependency the next stage has to wait out, and it isn't a workflow problem. It's what happens when an S3 object store and a file system have no shared foundation.

NeuralMesh Fast S3 Object Storage runs S3 and POSIX on the same physical data blocks, so that requirement doesn't exist. New files are immediately readable through S3, with no gateway and no translation layer between them. Training, fine-tuning, and inference all read and write against the same S3 deployment. However many copies a pipeline needed before, it needs one now.

Governance follows the same path. S3 lifecycle rules, IAM policies, encryption, and audit logging set once apply automatically across POSIX, NFS, and SMB, which closes a gap specific to object storage: conventional S3 object stores that bolt on file access through a gateway can't guarantee a lifecycle rule written in S3 also governs data touched through POSIX. Here, it does. One compliance posture covers every protocol instead of one per protocol, which for organizations with data sovereignty requirements means every pipeline stage can run on-premises without giving anything up to get there.

What's built specifically for S3

S3 Performance Bucket. S3 over RDMA matters most: it moves S3 data directly into GPU memory over RDMA instead of through TCP and a shared connection pool, cutting CPU overhead and latency out of every S3 read and write. For inference operators serving weight loads under concurrent sessions, that shows up as time-to-first-token. For training and checkpointing, it's the difference in how fast a multi-terabyte dataset or checkpoint moves without stealing CPU cycles from the job. The same engineering also speeds up catalog scans and high-concurrency writes, the two other operations that saturate first under AI load. Enumeration alone runs about 4.5x faster than the nearest published alternative on large catalogs, and the full set runs 5 to 6x faster than standard S3 configuration on AI workload patterns.

S3 Data Management. Conventional S3 object stores make pipelines poll for changes, and enforce lifecycle rules that don't survive a protocol switch. Event notifications push to Kafka the moment an object changes, so preprocessing workers and indexers react in real time instead of waiting on a poll cycle. Lifecycle rules, versioning, and delete protection apply the same way no matter which protocol touched the object, so a training dataset or checkpoint stays governed and recoverable across the whole pipeline.

S3 Security and Identity. This closes the permission gap conventional S3 gateways can't: access set once, at the POSIX layer or in S3 IAM policies, holds no matter which protocol touches the data next. Temporary, zero-trust credentials and existing LDAP identities work directly against S3, so security teams don't need a separate identity system just for the object tier.

S3 Multitenancy and Operations. Slow S3 tenant provisioning delays revenue directly for AI cloud providers scaling their tenant base. New tenants come online in minutes through standard Kubernetes manifests, the same workflow already used for compute, and run alongside each other, whether that's one large anchor tenant on dedicated infrastructure or dozens of smaller ones sharing it. Observe puts request rates, latency, and per-tenant consumption in one dashboard, so managing hundreds of tenants doesn't require a team that grows with them.

Built for AI at scale

AI cloud providers get an S3 tier that provisions in minutes, holds SLA commitments under concurrent load from hundreds of tenants, and runs on the same commodity hardware as the rest of their stack, so every GPU cycle bills instead of waiting on storage.

Enterprises running private AI get out of the two-system architecture entirely. One S3 deployment handles training, fine-tuning, and inference, with one compliance posture covering every protocol and every site.

Inference operators get model weight reads fast enough to hold time-to-first-token targets under production concurrency, because the object store was built for that load instead of adapted for it after the fact.

Request a benchmark walkthrough at weka.io/contact. Dig into the details by downloading our Technical Brief.

¹ 5,000 concurrent connections per node, versus roughly 1,000 for conventional S3 object stores.