Organizations building modern applications in the cloud increasingly rely on infrastructure as code (IAC) tools to codify best practices and automate deployments, simplify day 2 operations, and codify best practices. IAC tools like AWS CloudFormation or Hashicorp Terraform are transforming how cloud infrastructure gets built to take better advantage of the inherent scalability and flexibility the cloud offers. As organizations embrace multiple clouds as part of their IT estate, they are increasingly standardizing on a single tool for IAC to simplify and streamline operations.

When WEKA launched on AWS in 2017, we set out to completely transform how organizations store, manage, and process their AI and HPC workloads in the cloud – including enabling many of the IAC capabilities available for their compute and network stacks – so they could get started with WEKA on AWS in under 30 minutes with a set of pre-defined AWS CloudFormation templates to automate deployments. In 2022, we added Hashicorp Terraform for automated deployments in Google Cloud. In February 2023, we added Terraform automation for Azure and highlighted customers’ ability to deploy WEKA to Azure in under 30 minutes using IAC practices.

Today, we are pleased to announce that organizations can automate their WEKA deployments on AWS using Terraform. With this release, organizations that use Terraform as their standard way to automate cloud deployments now have ready-to-deploy Terraform modules to set up their WEKA environment on AWS. AWS customers can now use Terraform to automate and manage their WEKA deployments and easily implement advanced capabilities like auto-scaling (out and in), auto-healing, and multiple data access clients. Customers can establish new WEKA on AWS deployments in under 30 minutes. Alternatively, organizations can download and incorporate the automation templates into their existing IAC workflows. Customers can find the latest Terraform modules for AWS on the WEKA Github site to get started.

Major Capabilities in Terraform for WEKA on AWS

Whether organizations use Terraform or CloudFormation, deployment automation for WEKA on AWS includes several helpful capabilities that codify best practices, accelerate deployments, and streamline operations.

WEKA auto-scaling is simple and easy to implement. At initial provisioning, Terraform creates a new AWS Auto Scaling Group (ASG) to support the WEKA cluster. To meet peak demand, WEKA administrators can increase the size of their cluster by simply increasing the size of the ASG either through the AWS Console or AWS CLI. They can then scale the WEKA cluster back down once the peak has passed by changing the size of the ASG (via AWS console or AWS CLI). This unique capability ensures WEKA organizations never have to over-provision storage and never pay for capacity (or performance) they aren’t using.

Terraform modules are also useful for driving auto-healing operations for organizations’ WEKA on AWS deployments. WEKA software monitors the cluster’s health using AWS CloudWarch event rules, triggers actions to heal the cluster back to the declarative state in Terraform, and maintains the high reliability and availability of the WEKA environment.

Terraform modules can automatically deploy the correct client to each application environment, enabling access to the WEKA deployment through multiple storage protocols (such as POSIX, NFS, and SMB) to streamline the set-up process for HPC data pipelines, generative AI workflows, and other use cases where multiple applications access data to accomplish different tasks within the workflow.

WEKA Deployment on AWS with Terraform Walk-Through

This section describes the various deployment tasks that Terraform-driven provisioning of WEKA on AWS automates, enabling fast, easy customer deployment and simplified ongoing operations.

Figure 1: Overview of WEKA Deployment on AWS using Terraform

Step 1: Integrate with Existing IAC workflows

Organizations with standardized Terraform for their cloud deployments can add the Terraform modules for WEKA to their existing Terraform automation.

Step 2: Define the VPC

Organizations can use Terraform-driven automation to either create and configure a new VPC for their WEKA environment and define all Security Groups and network access controls they need or deploy WEKA directly to their existing VPCs and use existing security groups.

Step 3: Deploy EC2 instances into an Auto Scaling Group for the WEKA cluster

This step automatically creates the Auto Scaling Group, deploys the EC2 instances, and starts them up. Once created, the launch template triggers a deployment script to install and configure WEKA software on the cluster.

Step 4: Set up Terraform State

The step sets up an AWS DynamoDB table where the deployment state is maintained and used to drive autoscaling and auto-healing operations.

Step 5: Set up AWS CloudWatch monitoring

WEKA software uses Amazon CloudWatch to monitor the EC2 instances and VPC supporting the WEKA cluster, stores event logs in Amazon CloudWatch Logs files, regularly checks the state of the WEKA cluster, and triggers healing or scaling actions, if needed.

Wrap-up

AWS customers who are standardizing on Terraform for cloud automation can now accelerate their WEKA deployments and streamline ongoing operations. To learn more about this capability, visit our documentation site. Customers can find the latest Terraform modules for AWS on the WEKA Github site to get started.