Modern analytics platforms need to process large data sets to deliver the highest levels of accuracy to the training and analytics systems. A high bandwidth, low latency storage infrastructure is essential to ensure the compute cluster is fully saturated with as much data as the application needs, otherwise it wastes expensive GPU resources. WekaIO’s parallel and distributed Matrix file system can easily saturate a GPU node and the integrated cloud tiering scales to exabyte of capacity in a single namespace.
Analytics clusters are expensive and need to be kept from idling. A small inefficiency costs tens of thousands of dollars over the lifetime of the machine, and I/O starvation is the main culprit. WekaIO Matrix™ has proven to easily saturate a GPU cluster and deliver over 4GBytes/second per node across an InfiniBand network.
Ever wish you could have access to more GPU resources for a day to speed up your workload? WekaIO Matrix supports cloud bursting, allowing users with on-premises compute clusters to elastically grow their environment in response to peak workload periods. Simply snapshot the file system to AWS and spin up additional resources in the cloud, or keep a backup copy on S3 that can be used to hydrate a cloud copy on demand.
"We are using WekaIO technologies over InfiniBand to address the challenges of data analytics at extreme scale in life sciences, particle physics, geosciences and other fields." Michael Norman, Director of San Diego Supercomputer Center at UCSD
Until now, performance goals were achieved by running file based applications on a local file system, with shared access provided via NFS. Our POSIX compliant file system runs more efficiently than NFS and leverages an innovative, customized protocol to deliver file based semantics. As a result, Matrix exceeds NFS performance in a sharable file system solution through our revolutionary metadata handling.Learn More