weka blog detail third level banner

Burning Down the House at Flash Memory Summit

Barbara Murphy
August 12, 2017

Flash Memory Summit grows from strength to strength and the show is fast becoming the go-to-place to learn about new innovations in flash. And while flash is hot, nobody attending Flash Memory Summit could have anticipated that we would have a complete cancellation of the trade show exhibits due to a fire in one of the booths. While it cost us the opportunity to participate with Intel demonstrating NVMe-over-fabrics technology, I can assure you that WekaIO’s smokin’ hot demo was not the culprit.

Despite the fiery setback, WekaIO managed to have a full schedule. The week began with a joint Intel and WekaIO press release announcing that the Matrix file system running natively on Intel’s NVMe-oF reference platform. The new “ruler” form factor packed 1 petabyte in a 1U form factor, delivering 3 million IOPS/U, a record for performance density. The combination of Intel’s hardware innovation and WekaIO’s record setting performance will be a winning solution for next generation data centers.

Liran Zvibel gave a great technical talk on designing file systems to take advantage of NVMe-oF technology. We have seen many announcements about block based solutions that leverage NVMe-oF, but I can safely say that we are the first to demonstrate file services over NVMe-oF. Basically, classic B-tree data structures were designed to optimize I/O to disk, and this structure has simply migrated to flash. Liran showed that Trees are really horrible for NAND Flash and while current file systems are great at providing bandwidth to applications, they cannot deliver the IOPS capability of flash.

Liran’s second presentation provided insight into how modern applications can take advantage of NVMe technology. Hadoop was designed when networking was at 1Gigabit and HDDs were king. But networking is now at 100Gigabit and flash technology is rapidly approaching disk price. Key to his discussion is that fast networks mean that data locality (the bane of Hadoop deployments) is no longer a requirement because at 100Gigabit Ethernet moving a 4K page takes 36 microseconds – and NAND flash is way slower than that. A high performance distributed file system on Flash can provide Hadoop with support for a very large namespace, high speed access to all data from any compute node, multi-protocol support for efficient ingest and processing. The result is no more “long-tail” problems as all CPUs can work simultaneously maximizing compute resources. Other compute workloads can share the same environment as Hadoop, so no more dedicated clusters with trapped data. Compute and capacity are disaggregated so the Hadoop clusters can be managed more cost effectively.

That’s all from the slightly charred show floor at FMS.