# Accelerating AI Safety Research at Lower Cost in the Cloud

**Customer:** Center for AI Safety

CAIS deployed WEKA® NeuralMesh Axon™ in Oracle Cloud for all operations in the CAIS Compute Cluster to reduce costs through more efficient resource utilization.

## Results

- **90%** Reduced storage costs
- **5x** faster data storage
- **500%** Growth in research

Organization Overview

## Accelerating AI Safety Research

The [Center for AI Safety](https://www.safe.ai/) (CAIS — pronounced ‘case’) is a San Francisco-based nonprofit that supports research and field building that promotes safe and responsible artificial intelligence (AI). CAIS believes that while AI has the potential to benefit the world profoundly, many fundamental problems in AI safety have yet to be solved. CAIS’s mission is to reduce societal-scale risks associated with AI by conducting and building the field of AI safety research and advocating for safety standards.

The Context

## The Rising Importance of AI Safety Research

[CAIS-supported research](https://www.safe.ai/work/research#technical-ml-research) includes topics central to the development of safe AI. Research on the[ robustness of safety guardrails](https://arxiv.org/abs/2311.00117) in large language models (LLMs) highlights the need to limit third-party developers from bypassing safety controls. Research focused on developing technical methods to identify and measure the[ tendency of LLMs to hallucinate](https://arxiv.org/abs/2310.01405) can help make AI systems more truthful and reliable. Work focused on determining the extent to which[ AI systems act based on reward systems versus ethics](https://arxiv.org/abs/2304.03279) will help AI researchers understand how AI systems act according to ethical considerations.

The Landscape

## Artificial Intelligence Research in an Era of GPU Scarcity

AI safety researchers who want to experiment on the latest LLMs face a dilemma. Conducting relevant research requires access to the latest GPU infrastructure to run experiments resembling real-world scenarios. However, the cost, complexity, space, and infrastructure skill sets needed to build an AI research cluster create high barriers for most AI-safety researchers.

The CAIS Compute Cluster is a dedicated GPU-accelerated cluster that provides AI safety researchers with subsidized, on-demand access to state-of-the-art infrastructure for LLM training and other AI safety projects. The CAIS compute cluster is specifically designed for researchers working on the safety of machine learning systems and supports a diverse range of research interests and collaborators.

> We immediately saw our cloud storage costs drop by 90% when we switched to WEKA.
>
> — Steven Basart, R&D Lead • Center for AI Safety

## The Challenge

AI safety research is a hugely diverse and rapidly growing field that requires affordable access to the latest GPU-accelerated infrastructure. As the CAIS Cluster grows to support a rapidly expanding set of researchers, challenges around scale, performance, cost control, and data management have come the foreground:

- **Performance Bottlenecks: **Scaling the CAIS Compute Cluster was blocked by storage IO bottlenecks due to slow metadata handling across millions of small files. Fast metadata processing is a critical challenge for LLM training and tuning.
- **Storage Cost Spiral: **CAIS experienced a data copy sprawl problem born from a lack of data management controls or quotas. Multiple researchers each manage model data in their own way, often storing multiple copies of the same training data set.
- **Low GPU Utilization:** Storage bottlenecks and poor metadata handling in legacy lustre storage led to GPUs sitting idle, starved for data during training and tuning.

> 80% of the storage attached to our GPU cluster was going unused.
>
> — Steven Basart, R&D Lead • Center for AI Safety

The Solution

## NeuralMesh Axon in Oracle Cloud

CAIS deployed NeuralMesh Axon in Oracle Cloud for all operations in the CAIS Compute Cluster to reduce costs through more efficient resource utilization. In Axon, the data storage environment resides on the same infrastructure resources as the model training environment. Contrast this with a traditional architecture where the model training and data storage reside on separate silos of infrastructure. The goal is to drive significant cost savings across the data infrastructure, increase resource utilization in the GPU cluster, and be more efficient in data management.

> We would need a faster network to even stress the WEKA environment.
>
> — Steven Basart, R&D Lead • Center for AI Safety

The Outcome

## Eliminated Data Bottlenecks for Faster AI Model Training

NeuralMesh deployment enabled CAIS to increase utilization of their GPU-intensive cloud infrastructure, reduce cloud storage costs, and increase researcher productivity.

```json
{
  "_key": "iht8g80i",
  "_type": "customerStoryProofPoints",
  "markDefs": null,
  "proofPoints": [
    {
      "context": "CAIS eliminated their dedicated storage silos by using NeuralMesh Axon which takes advantage of unused NVMe storage on GPU/CPU nodes in the compute cluster.",
      "description": "Reduced storage costs",
      "unit": "%",
      "value": "90"
    },
    {
      "context": "CAIS started with a small WEKA cluster to deliver 1.7 Million IOPs. Today it seamlessly handles tens of millions of small files in their AI training data sets.",
      "description": "faster data storage",
      "unit": "x",
      "value": "5"
    },
    {
      "context": "The CAIS compute environment now supports over 200 researchers globally, representing 6x increase in just 18 months and still growing.",
      "description": "Growth in research",
      "unit": "%",
      "value": "500"
    }
  ]
}
```

- **Increased GPU utilization:** CAIS has been able to increase utilization of local NVMe storage in their GPU instances from 20% to 100%.
- **Eliminate data bottlenecks:** Growth in the CAIS Compute Cluster is no longer storage constrained thanks to a data environment optimized to handle metadata intensive operations.
- **Faster AI model training:** The new CAIS data environment accelerates AI model training times, enabling researchers to investigate new aspects of AI safety more quickly.

[Download PDF](https://cdn.sanity.io/files/ult5g8gw/production/e85630adaeb495770fa40e606c5893f1760c9136.pdf)