# WEKA for AWS ParallelCluster WEKA for AWS ParallelCluster deploys a high-performance data platform for AI model development, lifting GPU utilization above 90% from an average of 30% by removing data bottlenecks. **Type:** Datasheet ## WEKA for AWS ParallelCluster Fast Data Storage, Optimized for HPC Workflows in AWS ### Accelerate Time to Discoveries WEKA for AWS ParallelCluster provides the fastest, most scalable data platform for HPC workloads on AWS, enabling organizations to accelerate HPC workflows from months to days. ### Increase Cluster Utilization High-performance storage from WEKA eliminates data bottlenecks, driving up GPU infrastructure utilization above 90% (from an avg 30%) enabling you to do more with your existing investment. ### Do More Science, Manage Less Infrastructure WEKA software automates data management for cloud and hybrid HPC workflows, eliminating most manual management tasks and enabling more science. ### Seamlessly Integrate Hybrid Cloud Workflows WEKA software accelerates model data load times by 50%, accelerating model training times and improving data scientist and developer productivity. ```json { "_key": "1a3a10ef613a", "_type": "textCallout", "content": [ { "_key": "26e0909b23c7", "_type": "block", "children": [ { "_key": "rp00018", "_type": "span", "marks": [], "text": "Deploy a high-performance data platform for AI model development that scales massively, drives GPU infrastructure utilization, and reduces costs." } ], "markDefs": [], "style": "normal" }, { "_key": "rp00021", "_type": "block", "children": [ { "_key": "rp00020", "_type": "span", "marks": [], "text": "Shrink epoch times from months to days" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "rp00023", "_type": "block", "children": [ { "_key": "rp00022", "_type": "span", "marks": [], "text": "Accelerate model load times by 50%" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "rp00025", "_type": "block", "children": [ { "_key": "rp00024", "_type": "span", "marks": [], "text": "Increase GPU Utilization by 90%" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "rp00027", "_type": "block", "children": [ { "_key": "rp00026", "_type": "span", "marks": [], "text": "Reduce model checkpoint times by 90%" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" } ], "markDefs": null, "title": "WEKA for AWS ParallelCluster." } ``` ### WEKA For AWS ParallelCluster Sample HPC Architecture ![Sample HPC architecture: AWS ParallelCluster head node and GPU or compute nodes with WEKA POSIX clients, a WEKA storage cluster on i3en instances, and an S3 tiering bucket.](https://cdn.sanity.io/images/ult5g8gw/production/5e6a8775b9dc94b47f81fa35318c6720770ec281-1371x621.jpg) WEKA For AWS ParallelCluster Sample HPC Architecture **1. **WEKA cluster deployed on i3en EC2 instances using WEKA Terraform modules for AWS to serve as a single high-performance storage **2. **WEKA POSIX Client software deployed on the ParallelCluster node uses Linux Kernel bypass enabling high throughput and extremely low latency **3. **WEKA seamlessly tiers data between NVMe SSD volumes on i3en EC2 instances and Amazon S3 object storage to optimize storage efficiency ## WEKA For AWS Specs ```json { "_key": "3df23cbfbd29", "_type": "richTableBlock", "headerRows": 0, "markDefs": null, "rows": [ { "_key": "8d0ab38dcb55", "_type": "row", "cells": [ { "_key": "100f4c0cb44e", "_type": "richTableCell", "content": [ { "_key": "rp00051", "_type": "block", "children": [ { "_key": "rp00050", "_type": "span", "marks": [], "text": "Virtual Machines" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "608c9371a27f", "_type": "richTableCell", "content": [ { "_key": "1e650a128ba1", "_type": "block", "children": [ { "_key": "9f35a4a811c4", "_type": "span", "marks": [], "text": "Amazon EC2 i3en" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "a39e92dd6fc9", "_type": "richTableCell", "content": [ { "_key": "133718699b0f", "_type": "block", "children": [ { "_key": "c6143c27ace1", "_type": "span", "marks": [], "text": "No. of files or directories" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "c0184257dd06", "_type": "richTableCell", "content": [ { "_key": "d7b42035f94d", "_type": "block", "children": [ { "_key": "65ac90aca8e1", "_type": "span", "marks": [], "text": "Up to 6.4 trillion files" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "513f88c42dde", "_type": "row", "cells": [ { "_key": "ec9cf7839342", "_type": "richTableCell", "content": [ { "_key": "aef065b36d0a", "_type": "block", "children": [ { "_key": "59798e1c1a77", "_type": "span", "marks": [], "text": "Cluster Size (VMs)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "a79ef85d4006", "_type": "richTableCell", "content": [ { "_key": "f08210757fd1", "_type": "block", "children": [ { "_key": "196f57c40639", "_type": "span", "marks": [], "text": "6 to Hundreds" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "b10b57caa386", "_type": "richTableCell", "content": [ { "_key": "fea902002d95", "_type": "block", "children": [ { "_key": "c969aa51e9c3", "_type": "span", "marks": [], "text": "No. of files in a single directory" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "ab3ad86b9e33", "_type": "richTableCell", "content": [ { "_key": "7ab4593c3ec7", "_type": "block", "children": [ { "_key": "a641c6cd60a2", "_type": "span", "marks": [], "text": "Up to 6.4 billion" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "e0e47813db82", "_type": "row", "cells": [ { "_key": "38e3000a040a", "_type": "richTableCell", "content": [ { "_key": "31d657a8dbd5", "_type": "block", "children": [ { "_key": "3c63e223cf07", "_type": "span", "marks": [], "text": "Cluster Size (VMs)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "c3b06820dfd5", "_type": "richTableCell", "content": [ { "_key": "f87346109f8c", "_type": "block", "children": [ { "_key": "1839b03d8b03", "_type": "span", "marks": [], "text": "Amazon S3" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "65c82a7d929f", "_type": "richTableCell", "content": [ { "_key": "f373fa509ac1", "_type": "block", "children": [ { "_key": "e8504ee9c706", "_type": "span", "marks": [], "text": "Capacity with object store" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "c38f0f422339", "_type": "richTableCell", "content": [ { "_key": "009888f24bf7", "_type": "block", "children": [ { "_key": "343ae630868e", "_type": "span", "marks": [], "text": "Up to 14 Exabytes" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "737c86110e6d", "_type": "row", "cells": [ { "_key": "a145b7c652a6", "_type": "richTableCell", "content": [ { "_key": "7b5414926324", "_type": "block", "children": [ { "_key": "7dc1c1b8e3aa", "_type": "span", "marks": [], "text": "Protocols" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "58aa5c6cc1de", "_type": "richTableCell", "content": [ { "_key": "d03237fd7639", "_type": "block", "children": [ { "_key": "4c09b0e07842", "_type": "span", "marks": [], "text": "POSIX, NFS, SMB, S3, CSI" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "ecae97b4e91d", "_type": "richTableCell", "content": [ { "_key": "c132cf94efc8", "_type": "block", "children": [ { "_key": "f6508591d2b3", "_type": "span", "marks": [], "text": "No. of File Systems" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "a155ab8db359", "_type": "richTableCell", "content": [ { "_key": "4c52ed22f076", "_type": "block", "children": [ { "_key": "ab603bbf6411", "_type": "span", "marks": [], "text": "Up to 1,024 per Cluster" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "9f93d40e6cf6", "_type": "row", "cells": [ { "_key": "19c39a98cfb9", "_type": "richTableCell", "content": [ { "_key": "c30c8a518025", "_type": "block", "children": [ { "_key": "aee45cfa22e9", "_type": "span", "marks": [], "text": "Automation" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "9694cb02896d", "_type": "richTableCell", "content": [ { "_key": "ec705b3bb44c", "_type": "block", "children": [ { "_key": "9697b6851dd7", "_type": "span", "marks": [], "text": "Hashicorp Terraform AWS CloudFormation AWS Marketplace" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "60fbcb73bd27", "_type": "richTableCell", "content": [ { "_key": "edccd915f1da", "_type": "block", "children": [ { "_key": "1a9a56479c17", "_type": "span", "marks": [], "text": "Snapshots" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "ff872911973c", "_type": "richTableCell", "content": [ { "_key": "2b544872c375", "_type": "block", "children": [ { "_key": "a71fa93b9c28", "_type": "span", "marks": [], "text": "Up to 24,576 Readable Snapshots (14,336 if all writable)" } ], "markDefs": [], "style": "normal" } ] } ] } ] } ``` #### Get started with [WEKA in the AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-2mfqnh6p4yurs?sr=0-1&ref_=beagle&applicationId=AWSMPContessa) today! [Download PDF](/api/resource-pdf?slug=weka-for-aws-parallelcluster)