WEKA™ on AWS

Modern analytics platforms are GPU-intensive and require large data sets to deliver the highest levels of accuracy to the training or analytics models. The workloads demand a high-bandwidth, low-latency storage infrastructure to ensure that the GPU cluster is fully saturated with as much data as the application needs. Typical data sets can span from terabytes to tens of petabytes, and the data access pattern for each Epoch is unique and unpredictable. This calls for a data infrastructure that can instantaneously and consistently feed large amounts of random data to multiple GPU nodes in real time, all emanating from a single shared data pool. WekaFS is a modern file system that is uniquely built to meet the performance and scalability needs of data-intensive applications leveraging P3 GPU instances in the AWS cloud. WekaFS on AWS was ranked #1 for supercomputer storage as measured by the IO-500 benchmark. WekaFS has proven scalable performance of over 10 GB/sec bandwidth to a single P3dn.24 GPU instance, delivering 2.4x the performance of FSx for Lustre and 6x the performance of local NVMe SSDs across a shared training set.