Accelerating AI Safety Research at Lower Cost in the Cloud

Organization Overview

Accelerating AI Safety Research

The Center for AI Safety (CAIS — pronounced ‘case’) is a San Francisco-based nonprofit that supports research and field building that promotes safe and responsible artificial intelligence (AI). CAIS believes that while AI has the potential to benefit the world profoundly, many fundamental problems in AI safety have yet to be solved. CAIS’s mission is to reduce societal-scale risks associated with AI by conducting and building the field of AI safety research and advocating for safety standards.

The Context

The Rising Importance of AI Safety Research

CAIS-supported research includes topics central to the development of safe AI. Research on the robustness of safety guardrails in large language models (LLMs) highlights the need to limit third-party developers from bypassing safety controls. Research focused on developing technical methods to identify and measure the tendency of LLMs to hallucinate can help make AI systems more truthful and reliable. Work focused on determining the extent to which AI systems act based on reward systems versus ethics will help AI researchers understand how AI systems act according to ethical considerations.

The Landscape

Artificial Intelligence Research in an Era of GPU Scarcity

AI safety researchers who want to experiment on the latest LLMs face a dilemma. Conducting relevant research requires access to the latest GPU infrastructure to run experiments resembling real-world scenarios. However, the cost, complexity, space, and infrastructure skill sets needed to build an AI research cluster create high barriers for most AI-safety researchers.

The CAIS Compute Cluster is a dedicated GPU-accelerated cluster that provides AI safety researchers with subsidized, on-demand access to state-of-the-art infrastructure for LLM training and other AI safety projects. The CAIS compute cluster is specifically designed for researchers working on the safety of machine learning systems and supports a diverse range of research interests and collaborators.

The Challenge

AI safety research is a hugely diverse and rapidly growing field that requires affordable access to the latest GPU-accelerated infrastructure. As the CAIS Cluster grows to support a rapidly expanding set of researchers, challenges around scale, performance, cost control, and data management have come the foreground:

• Performance Bottlenecks: Scaling the CAIS Compute Cluster was blocked by storage IO bottlenecks due to slow metadata handling across millions of small files. Fast metadata processing is a critical challenge for LLM training and tuning.
• Storage Cost Spiral: CAIS experienced a data copy sprawl problem born from a lack of data management controls or quotas. Multiple researchers each manage model data in their own way, often storing multiple copies of the same training data set.
• Low GPU Utilization: Storage bottlenecks and poor metadata handling in legacy lustre storage led to GPUs sitting idle, starved for data during training and tuning.

The Solution

NeuralMesh Axon in Oracle Cloud

CAIS deployed NeuralMesh Axon in Oracle Cloud for all operations in the CAIS Compute Cluster to reduce costs through more efficient resource utilization. In Axon, the data storage environment resides on the same infrastructure resources as the model training environment. Contrast this with a traditional architecture where the model training and data storage reside on separate silos of infrastructure. The goal is to drive significant cost savings across the data infrastructure, increase resource utilization in the GPU cluster, and be more efficient in data management.

The Outcome

Eliminated Data Bottlenecks for Faster AI Model Training

NeuralMesh deployment enabled CAIS to increase utilization of their GPU-intensive cloud infrastructure, reduce cloud storage costs, and increase researcher productivity.

• 0%

  Reduced storage costs

  CAIS eliminated their dedicated storage silos by using NeuralMesh Axon which takes advantage of unused NVMe storage on GPU/CPU nodes in the compute cluster.

• 0x

  faster data storage

  CAIS started with a small WEKA cluster to deliver 1.7 Million IOPs. Today it seamlessly handles tens of millions of small files in their AI training data sets.

• 0%

  Growth in research

  The CAIS compute environment now supports over 200 researchers globally, representing 6x increase in just 18 months and still growing.

• Increased GPU utilization: CAIS has been able to increase utilization of local NVMe storage in their GPU instances from 20% to 100%.
• Eliminate data bottlenecks: Growth in the CAIS Compute Cluster is no longer storage constrained thanks to a data environment optimized to handle metadata intensive operations.
• Faster AI model training: The new CAIS data environment accelerates AI model training times, enabling researchers to investigate new aspects of AI safety more quickly.