# Accelerating Large Language Model Training in the Cloud

**Customer:** Stability AI

Stability AI uses WEKA® NeuralMesh Axon™ on AWS for its generative AI model training and tuning environment.

## Results

- **95%** Reduction in Cost per TB
- **93%** GPU utilization efficiency
- **35%** Faster Model Training

Introduction

## Faster, Safer AI Through Open Models in Every Modality

[Stability AI](https://stability.ai/) is a visionary open-source generative AI company on a mission to build an intelligent foundation to activate humanity's potential. The company delivers breakthrough, open-access AI models with minimal resource requirements for imaging, video, 3D, language, code, and audio.

By providing modular, open models in every modality, Stability AI can deliver new AI models to market faster, more sustainably, and with greater safety controls. Stability AI customers can bring pre-trained models into their domain and fine-tune them using their own IP-rich datasets in a secure and controlled manner. This approach improves AI researchers' ability to deeply understand model dynamics, reduce the risk of model hallucination, and build AI applications that are safer and more responsible.

In the Generative AI space, the pace of innovation, already off the charts, is expected to remain high for years as foundation model providers create the AI building blocks that will drive next-generation applications. Stability AI seeks to win the generative AI race based on market speed, model accuracy, responsiveness, and accessibility.

> We can now reach 93% GPU utilization when running our AI model training environment using NeuralMesh Axon on AWS.
>
> — Richard Vencu, MLOps Lead • Stability AI

The Challenge

## Scale and Performance Limitations of a Legacy Lustre File System

Delivering fast time to market for features and new versions of the Stable Diffusion model requires access to the fastest GPU infrastructure. However, Stability found the scale and performance limitations of their legacy Lustre file system deployment in the cloud limited their ability to utilize their GPU infrastructure fully and drove big cost surprises.

### Idle GPU Infrastructure

The fastest networking, compute, and GPUs required to support model training and tuning have become scarce commodities, commanding high prices. Performance and scale limitations in legacy data management caused storage bottlenecks that left GPU infrastructure sitting idle, starved for data.

### High Cloud Storage Costs

Rapid growth in researcher and customer interest drove explosive data growth in the legacy Lustre environment. However, storage over-provisioning requirements and limited data management capabilities led to unexpected cost overruns for storage in the cloud.


> Before NeuralMesh Axon, we’d push a button, and the cloud storage cost would just blow up like $100K per month in a matter of hours.
>
> — Richard Vencu, MLOps Lead • Stability AI

The Solution

## Storage and Model Training on the Same Infrastructure

In NeuralMesh Axon, the data storage environment resides on the same infrastructure resources as the model training environment, in contrast to a traditional architecture where the model training and data storage reside on separate silos of infrastructure. This innovative solution enables Stability to increase resource efficiency and recognize massive savings in cloud storage costs associated with their GPU infrastructure.

> When we switched to NeuralMesh Axon, we got 15 times more cloud storage capacity at about 80% of the previous cost.
>
> — Richard Vencu, MLOps Lead • Stability AI

The Outcome

## Transforming Infrastructure Strategy

Using NeuralMesh Axon, Stability AI transformed its infrastructure strategy, driving increased resource utilization. The new approach helped improve GPU utilization and accelerate model training times. With the new approach, Stability is also moving forward on its goals for sustainable AI.

```json
{
  "_key": "99bd45362503",
  "_type": "customerStoryProofPoints",
  "markDefs": null,
  "proofPoints": [
    {
      "context": "Stability AI reduced their storage costs by 95% by switching to NeuralMesh Axon, which uses existing NVMe resources available in the GPU cluster.",
      "description": "reduction in cost per TB",
      "unit": "%",
      "value": "95"
    },
    {
      "context": "Stability AI achieves 93% GPU utilization by eliminating storage bottlenecks associated with small file handling and metadata look-ups.",
      "description": "GPU utilization",
      "unit": "%",
      "value": "93"
    },
    {
      "context": "Stability AI found AI model epoch times reduced by 3 weeks on average with a faster data platform that eliminated manual data management tasks.",
      "description": "faster model training",
      "unit": "%",
      "value": "35"
    }
  ]
}
```

- **Simplified Data Operations:** Manual data copy, movement, and manage have been eliminated by relying on NeuralMesh Axon zero-copy, zero-tuning data architecture.
- **Increased Sustainability:** Stability AI is able to reduce the carbon footprint associated with model training through Improved GPU utilization, and a reduced storage footprint.
- **Increased Productivity:** Researchers at Stability AI spend less time manually loading and pre-processing data. Model checkpoints are more reliable.

[Download PDF](https://cdn.sanity.io/files/ult5g8gw/production/77340a1fd3540ef7a0522ca2930e7716b04424f2.pdf)