Weka Wins the IO500 Again, But That’s Only Part of the Story
Matt Miller. November 25, 2019
The bi-annual release of the IO500 results was announced last week at SC19 in Denver, and we were thrilled to have once again taken the top spot in the overall rankings, this time in the overall position, outpacing very large supercomputers and other systems that were designed specifically to produce a good score on the benchmark. You can read the press release here. But equally as impressive is the story behind the top ranking….
What received less attention, but in our minds gets to the essence of why we perform so well compared to other parallel file systems, is that Weka also claimed the award for highest metadata performance. It wasn’t really even close, when compared against other commercially available solutions on the list. Why is this important? Our fully distributed metadata architecture is a critical component of a modern file system that is dealing with data sets that often have millions or billions of files. Legacy parallel file systems chose to separate metadata from the data path, but over time these solutions just have not been able to scale their metadata performance. This quickly becomes a bottleneck, as evidenced by the results.
By contrast, the Weka architecture employs a totally distributed architecture where every node in the storage cluster has exactly the same amount of metadata. This allows metadata performance to scale to massive levels, which is reflected in our score.
What’s also impressive about our top score is the fact it was achieved with Weka running on AWS (i3en.24xlarge instances to be specific) in the public cloud. This shows that customers can take advantage of the public cloud for HPC performance, which can compete with the most advanced on-premises supercomputer clusters. What this also underscores is that customers don’t have to choose between cloud or on-prem, at least as it pertains to performance. In fact, with the Weka file system you can run a hybrid model between the two. Weka has several customers already that utilize the public cloud for high performance computing, such as Tre Altimira for geospatial modeling, Untold Studios for high performance cloud rendering, and Aiden Lab at the Baylor College of Medicine for genomic research.
We should also highlight our versatility across some of the other industry benchmarks. Weka hasn’t just won in the IO500, but we also retain #1 positions in SPEC SFS, and more recently set 8 world records in the STAC-M3 benchmarks along with our partners Penguin Computing and Kx. So our tagline of the world’s fastest file system isn’t just a marketing claim with no independent industry validation (ahem, Panasas).
So it was a great week at SC19, and the Weka momentum continues. We are continuing to expand our customer base and helping customers solve their biggest high performance problems. We hope you’ll come along for the ride!