What is GPFS?

IBM Spectrum Scale, formerly the General Parallel File System (GPFS) is high-performance clustered file system software developed by IBM in 1998. It was originally designed to support high-streaming applications in the media and entertainment space. It can be deployed in shared-disk or shared-nothing distributed parallel modes, or a combination of these two. The predominant deployment is as a shared-disk solution utilizing SAN block storage for persistent storage.

Advantages of a Modern Parallel File System over GPFS

Although IBM Spectrum Scale (aka IBM GPFS) is a parallel file system, it was developed over 20 years ago to support the high throughput needs of multimedia applications. And for the subsequent 15+ years it was optimized to deliver the highest large file bandwidth performance from hard disk drives. But the world has changed dramatically in the last ten years, flash technology has revolutionized the data center and workloads have changed from large sequential to metadata intensive small File I/O intensive big data workloads. The technology designed and fine-tuned to solve streaming applications is not suitable for the age of big data and analytics. IBM continues to develop its Spectrum Scale product, yet the file system still can not deliver performance for small file, low latency, I/O intensive workloads.

A modern parallel file system leverages new technologies such as NVMe, high speed low latency networking to service small file I/O in a massively parallel manner. In addition to managing small files efficiently, the metadata demands for AI and GPU workloads is orders of magnitude higher. A modern parallel file system utilizes a distributed metadata model where metadata is spread evenly across all the storage nodes and each node contains equal amounts of data and metadata. By contrast, the traditional GPFS model maintains metadata in separate metadata servers. This creates a performance bottleneck when the system contains millions of files.

Advantages of modern file systems

• Best choice for GPU workloads which contain millions of files, large and small
• Full featured, Easy to use, easy to manage, no need for specialized administrators to configure, manage and tune the storage system
• High throughput, high IOPS, with ultra-low latency, fully saturating the network and delivering applications linear performance at exabyte scale, especially for mixed workloads

GPFS vs. Modern Parallel File System (WEKA)

¹ At smallest cluster configuration
² Spectrum Scale on ESS 3000, performance will vary on other hardware

Feature Comparison: GPFS vs. Parallel File System (WEKA)

Learn how WEKA's file system delivers the highest performance for the most data-intensive workloads.

Additional Helpful Resources

Worldwide Scale-out File-Based Storage 2019 Vendor Assessment Report
5 Reasons Why IBM Spectrum Scale is Not Suitable for AI Workloads
NAS vs. SAN vs. DAS
Learn About HPC Storage, HPC Storage Architecture and Use Cases
What is Network File System (NFS)?
Network File System (NFS) and AI Workloads
Block Storage vs. Object Storage
BeeGFS Parallel File System Explained
Introduction to Hybrid Cloud Storage