S3 Certification now Eliminates Data Silos
Shimon Ben David. October 2, 2020
What is an S3 Gateway
First, let’s step back and explain what S3 is. S3 is a scalable, open source web services interface developed by Amazon that allows you to store and retrieve data over HTTP and/or HTTPs. Multiple private and public clouds now support the S3 protocol. Due to the simplicity of the S3 protocol and the fact that it doesn’t require a connection, it is used by multiple IOT devices, such as cameras, scanners, scopes, recorders, and more to push data to public and/or private clouds.
Since many on-premises or cloud-native applications are still using standard file interfaces such as POSIX or NFS or SMB for analyzing data in high performance environments, typically you would prefer a storage layer that couldingest data using S3 as well as support accessing the data using these standard protocols.
One simple way of achieving this is by using an S3 gateway. The gateway is a software portal that can understand S3 requests (e.g., PUTs or GETs) on one side while communicating with a filesystem over standard interfaces (e.g., POSIX, NFS, or SMB) on the other side. There are multiple software options that can act as S3 gateways, one of which is MinIO.
What are the use cases for using an S3 Gateway?
The typical use case for using an S3 gateway is to allow a storage system with an easily available way of servicing data over S3, thereby eliminating the need to create multiple storage silos that are mandated simply due to the access protocol.
IOT devices push data to an S3 storage environment. The data is then copied over to a filesystem that supports standard interfaces, such as POSIX, NFS or SMB, so that it can be analyzed by compute servers/instances in a high performance environment.
IOT devices push data to an S3 storage environment. Since the same storage environment also support POSIX, NFS, and SMB, the compute servers/instances can now directly run their compute pipeline (such as AI/ML or HPDA, or others ) on it. In the end, Scenario B eliminates the need to double the storage capacity and eliminates the time and effort it takes to copy the data between two separate storage environments.
Weka and Minio S3 Gateway – Benefits
Weka recently tested the MinIO S3 gateway running on WekaIO in the AWS cloud. The results were amazing. MinIO is already considered one of the fastest object solutions in its own right. However, when MinIO acts as a gateway to the Weka file system (WekaFS) it delivered a 48% increase in performance across the board, whether it was a PUT (write), GET (read), DELETE (well, that is obvious), or STAT (statistics).
Part of the reason we could achieve this performance level is that WekaFS is, in fact, built to handle massive throughput, as well as millions of small IO operations. The WekaFS purpose-built technology with its high-throughput and high-IOPS capabilities easily leverages the MinIO ability to translate throughput-heavy S3 communication.
The overall benefit is clear. Using a combination of a MinIO S3 gateway and a WekaFS high-performance, scalable parallel file system enables organizations to eliminate the need for choosing and managing data between multiple storage silos, thereby saving time, effort, and money. Who can resist getting more work completed in less time, all on a single infrastructure that can be located in a physical data center or in a public cloud?
Want to remove data silos
Learn more about Weka multi-protocol supportClick here