# Scaling Deep Learning with WEKA and NVIDIA DGX A100 Basepod

This reference architecture shows how WEKA software and industry-standard servers scale deep learning performance on NVIDIA DGX A100 BasePOD.

**Type:** Reference Architecture

## Executive Summary

Organizations of various sizes, use cases, and technical skills are looking for infrastructure solutions to accelerate their artificial intelligence (AI), machine learning (ML), and deep learning (DL) initiatives.

WekaIO™ (WEKA) and NVIDIA partnered to architect and validate a high-performance, scalable AI solution accessible to everyone. This document contains validation information for the WEKA AI™ reference architecture (RA) solution.

This design was implemented using [NVIDIA DGX BasePOD](https://www.nvidia.com/en-us/data-center/dgx-basepod/) reference architecture with NVIDIA DGX™ A100 systems. The operation and performance of this system were validated by NVIDIA and WEKA using industry-standard benchmark tools.

This architecture delivers excellent linear scaling based on the validation testing results for training workloads. Organizations can start small and quickly and independently scale compute and storage resources to multi-rack configurations with predictable performance to meet any ML workload requirement.

### Tested Solution Details

```json
{
  "_key": "rp00048",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "rp00031",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00016",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00018",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00017",
                  "_type": "span",
                  "marks": [],
                  "text": "Product Name"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00019",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00021",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00020",
                  "_type": "span",
                  "marks": [],
                  "text": "Product Version"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00022",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00024",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00023",
                  "_type": "span",
                  "marks": [],
                  "text": "WEKA Version"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00025",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00027",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00026",
                  "_type": "span",
                  "marks": [],
                  "text": "Source"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00028",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00030",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00029",
                  "_type": "span",
                  "marks": [],
                  "text": "WEKA OS"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00047",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00032",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00034",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00033",
                  "_type": "span",
                  "marks": [],
                  "text": "NVIDIA DGX"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00035",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00037",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00036",
                  "_type": "span",
                  "marks": [],
                  "text": "3.99.9"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00038",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00040",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00039",
                  "_type": "span",
                  "marks": [],
                  "text": "3.x"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00041",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00043",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00042",
                  "_type": "span",
                  "marks": [],
                  "text": "On-Premises"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00044",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00046",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00045",
                  "_type": "span",
                  "marks": [],
                  "text": "CentOS"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

## 1. Introduction

### Audience

We wrote this reference architecture for those who design, manage, and support WEKA. Consumers of this document should already be familiar with NVIDIA DGX and the WEKA® Data Platform.

We have organized this document to address critical items for enabling successful design, implementation, and transition to operation.

### Purpose

This document covers the following subject areas:

- Overview of the WEKA solution.
- Overview of NVIDIA DGX.
- The benefits of NVIDIA DGX with WEKA.
- Recommendations for architecting a complete NVIDIA DGX solution on the WEKA platform.
- Benchmarks for NVIDIA DGX performance with WEKA.

### Document Version History

```json
{
  "_key": "rp00103",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "rp00082",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00073",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00075",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00074",
                  "_type": "span",
                  "marks": [],
                  "text": "Version Number"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00076",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00078",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00077",
                  "_type": "span",
                  "marks": [],
                  "text": "Published"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00079",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00081",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00080",
                  "_type": "span",
                  "marks": [],
                  "text": "Notes"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00092",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00083",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00085",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00084",
                  "_type": "span",
                  "marks": [],
                  "text": "1.0"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00086",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00088",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00087",
                  "_type": "span",
                  "marks": [],
                  "text": "September 2021"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00089",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00091",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00090",
                  "_type": "span",
                  "marks": [],
                  "text": "Original publication."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00102",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00093",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00095",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00094",
                  "_type": "span",
                  "marks": [],
                  "text": "1.1"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00096",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00098",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00097",
                  "_type": "span",
                  "marks": [],
                  "text": "September 2022"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00099",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00101",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00100",
                  "_type": "span",
                  "marks": [],
                  "text": "Updated format and BasePOD"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

## 2. WEKA Data Platform

The WEKA Data Platform is deployed on commercially available NVMe servers. Performance testing was carried out on the HPE ProLiant DL325 server platform with WEKA. An entry starting cluster size requires eight server nodes for full availability with the ability to survive up to a two-node failure. Each server has a CPU, NVMe storage, and high bandwidth networking. The exact configuration for the RA is detailed in the Technology Requirements section. The cluster can be easily scaled to thousands of nodes.

### Performance at Scale

At the core of the WEKA Data Platform is WekaFS is the world’s fastest and most scalable POSIX-compliant parallel file system. It is designed to transcend the limitations of legacy file systems that leverage local storage, NFS, or block storage, making it ideal for data-intensive AI and HPC workloads. WekaFS is a clean sheet design integrating NVMe-based flash storage for the performance tier to the GPU servers, object storage, and ultra-low latency interconnect fabrics such as 100GbE and 400GbE or InfiniBand into an NVMe-over-Fabrics architecture, creating a highly high-performance scale-out storage system. WekaFS performance scales linearly as more servers are added to the storage cluster allowing the infrastructure to scale with the increasing demands of the business.

### Multi-Protocol Ready

In addition to POSIX access, WekaFS supports all the standard file access protocols, including NFS, SMB, and S3, for maximum compatibility and interoperability. Hadoop and Spark environments also benefit from the performance of a shared file system through a fully integrated connector that allows WekaFS to replace HDFS and function as a single, easy-to-manage data lake for all forms of analytics.

### Expandable Global Namespace over S3 Object Store

WekaFS delivers best-of-breed performance from the NVMe flash tier, and the namespace can expand to any S3 object store, on-premises or in the cloud. This optional hybrid storage model with the ability to develop the global namespace to lower-cost hard disk drives in an object store delivers a cost-effective data lake without compromising performance. The integrated tiering to multiple S3 targets enables the cost-effective data lifecycle management for older or less used training data.

### Advanced Durability and Security

Large and agile datasets in AI/ML frequently require a data versioning capability. This is achieved using Weka’s instant and space-efficient snapshots capability for experiment reproducibility and explainability. The snap-to-object feature captures a point-in-time copy of the entire, unified [flash and object store] file namespace that can be presented as another file namespace instance in a private or public cloud. With crucial management integration, Weka’s integrated snapshots and end-to-end encryption features ensure that data is always backed up and secure throughout its lifecycle. WekaFS also provides immutability and data mobility for these datasets with instant recovery. WekaFS can seamlessly back up to multiple cloud targets providing backup, DR, and data governance capability.

### Cloud Bursting and Data Mobility

In addition to providing versioning, WEKA's snap-to-object feature offers additional benefits beyond backup and DR to the public cloud; it enables secure data portability from on-premises to the public cloud for organizations that require access to on-demand GPU resources in the public cloud.

### Container Support

Organizations are increasingly adopting containers deployed on Kubernetes (K8s) for AI workloads. Using the WekaFS K8s CSI plug-in, organizations now have flexibility in how and where they deploy containerized applications. It provides easy data mobility from on-premises to the cloud and back while delivering the best storage performance and latency. Figure X provides an overview of WekaFS in a typical production deployment.


![WekaFS production deployment: GPU and protocol clients over a WekaFS NVMe cluster (10–20% on NVMe) tiering to a private and public S3 data lake (80–90% on disk)](https://cdn.sanity.io/images/ult5g8gw/production/b783b58a69fe386a6a8b065998bee60eab0e94ad-573x239.png)


Figure 1. WekaFS

## 3. NVIDIA BasePOD Architecture

Figure X shows an exploded view of the significant components in the NVIDIA DGX A100 system.


![Exploded NVIDIA DGX A100 system showing eight A100 GPUs, six NVSwitches, dual AMD Rome CPUs, ConnectX-6 NICs, and Gen4 NVMe](https://cdn.sanity.io/images/ult5g8gw/production/21356e94bde48eda97cb825b6a0f7abf4f057d35-582x341.png)


Figure 2. DGX A100 components

### NVIDIA A100 Tensor Core GPU

At the core, the NVIDIA DGX A100 system leverages the NVIDIA A100 GPU (Figure X), designed to accelerate large, complex, efficient AI workloads and several small workloads, including enhancements and new features for increased performance over the NVIDIA V100 GPU. The A100 GPU incorporates up to 80 gigabytes (GB) of high-bandwidth HBM2 memory and more extensive and faster caches and is designed to reduce AI and HPC software and programming complexity.


![NVIDIA A100 Tensor Core GPU module with HBM stacks and gold board connectors](https://cdn.sanity.io/images/ult5g8gw/production/221c16fe84fdeee6dbf75390bef687d38d94b2dc-323x252.png)


Figure 3. NVIDIA A100 GPU

The NVIDIA A100 GPU includes new features to accelerate AI workload and HPC application performance further.

- Third-generation Tensor Cores
- Fine-grained Structured Sparsity
- Multi-Instance GPU

You can find additional [product information](https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/dgx-a100/dgxa100-system-architecture-white-paper.pdf) online.

### Multi-Instance GPU (MIG)

The NVIDIA A100 GPU incorporates a new partitioning capability called Multi-Instance GPU (MIG) for increased GPU utilization. MIG uses spatial partitioning to carve the physical resources of a single A100 GPU into as many as seven independent GPU instances. With MIG, the NVIDIA A100 GPU can deliver the guaranteed quality of service at up to 7 times higher throughput than V100 with simultaneous instances per GPU.

On an NVIDIA A100 GPU with MIG enabled, parallel compute workloads can access isolated GPU memory and physical GPU resources as each GPU instance has its memory, cache, and streaming multiprocessor. This allows multiple users to share the same GPU and run all instances simultaneously, maximizing GPU efficiency.

MIG can be enabled selectively on any number of GPUs in the DGX A100 system - not all GPUs need to be MIG-enabled. However, if all GPUs in a DGX A100 system are MIG enabled, up to 56 users can simultaneously and independently take advantage of GPU acceleration.

Typical uses cases that can benefit from MIG are

- Multiple inference jobs with batch sizes of one that involves small, low-latency models and that doesn’t require all the performance of a full GPU
- Jupyter notebooks for model exploration
- Resource sharing of the GPU among multiple users

### Third Generation NVLink and Second Generation NVSwitch

The DGX A100 system contains six second-generation NVIDIA® NVSwitch™ fabrics interconnecting the A100 GPUs using third-generation NVIDIA NVLink® high-speed interconnects. Each A100 GPU uses twelve NVLink interconnects to communicate with all six NVSwitches, which means there are two links from each GPU to each switch. This provides maximum bandwidth to communicate across GPUs over the links.

The second-generation NVSwitch [Figure 4] is two times faster than the previous version, which was first introduced in the NVIDIA DGX-2 system. The combination of six NVSwitches and third-generation NVLinks enables individual GPU to GPU communication to peak at 600 GB/s, which means that if all GPUs communicate with each other, the total amount of data transferred peaks at 4.8 TB/s for both directions.


![Close-up of NVIDIA GPUs and NVSwitch fabric on a DGX A100 board with heat pipes](https://cdn.sanity.io/images/ult5g8gw/production/b31a5e9430f892feef391c128cd090c51b797a5b-383x222.png)


Figure 4. NVIDIA NVSwitches

### NVIDIA ConnectX-6

Multi-system scaling of AI deep learning and HPC computational workloads requires strong communications between GPUs in multiple systems to match the significant GPU performance of each design. In addition to NVLink for high-speed communication internally between GPUs, the DGX A100 is purpose-built for multi-system AI scaling with eight single-port NVIDIA ConnectX-6 200Gb/s HDR InfiniBand ports (also configurable as 200Gb/s Ethernet ports), providing 3.2 Tb/s of peak bandwidth from a single system that can be used to immediately build a high-speed cluster of DGX A100 systems such as NVIDIA DGX SuperPOD.

The most common methods of moving data to and from the GPU involve leveraging the onboard storage and using the NVIDIA ConnectX-6 network adapters through Remote Direct Memory Access (RDMA). The DGX A100 incorporates a one-to-one relationship between the IO cards and the GPUs, which means each GPU can communicate directly with external sources without blocking other GPUs’ access to the network.

The NVIDIA ConnectX-6 I/O cards offer flexible connectivity as they can be configured as HDR InfiniBand or 200Gb/s Ethernet. This allows the NVIDIA DGX A100 to be clustered with other nodes to run HPC and AI workloads using low latency, high bandwidth InfiniBand, or RDMA over Converged Ethernet (RoCE).


![NVIDIA ConnectX-6 single-port network adapter with NVIDIA-branded heatsink and QSFP cage](https://cdn.sanity.io/images/ult5g8gw/production/51c0e27007dcb1096d41e5a2c49ef91791896a5c-1260x709.png)


Figure 5. Mellanox Single-port ConnectX-6

### NVIDIA Base Command

NVIDIA Base Command provides enterprise-grade orchestration and cluster management, and it now features a complete software stack for maximizing AI developer productivity, IT manageability, and workload performance. The workflow management features of Base Command enable centralized control of AI development projects with simplified collaboration for project teams and integrated monitoring and reporting dashboards.

Base Command works with the NVIDIA AI Enterprise software suite, which is now included with every DGX system. The NVIDIA AI Enterprise enables end-to-end AI development and deployment with supported AI and data science tools, optimized frameworks, and pretrained models.

Base Command also offers enterprise-workflow management and MLOps integrations with DGX-Ready Software providers like Domino Data Lab, [Run.ai](http://Run.ai), and Weights & Biases. It also includes libraries that optimize and accelerate compute, storage, and network infrastructure — while ensuring maximized system uptime, security, and reliability.

## 4. WEKA Solutions built on DGX BasePOD

The WEKA Data Platform powered by DGX BasePOD will enable us to extend these benefits to our customers at a much larger scale. Key highlights of the new solution include:

- The WEKA Data Platform and DGX BasePOD are now directly applicable to mission-critical enterprise AI workflows, including natural language processing and larger-scale workloads for customers in the life sciences, healthcare, and financial services industries, among many others. WEKA can efficiently serve large and small files across various workload types.
- WEKA continued innovation and support of Magnum IO GPUDirect Storage technology provides low-latency, direct access between GPU memory and storage. This frees CPU cycles from servicing the I/O operations and delivers higher performance for other workloads.


![WEKA Data Platform under an Ingest, ETL, Train, Validate, Infer, and Archive pipeline with bidirectional arrows at each stage](https://cdn.sanity.io/images/ult5g8gw/production/abb1deb8f5746d9dcb89aa007141582e8e08c17a-1424x434.png)


Figure 6. WEKA Data Platform

## 5. Solution Design

### Design Decisions

The following tables cover design decisions and rationale for DGX BasePOD with WEKA.

**General Design Decisions: DGX BasePOD**

```json
{
  "_key": "rp00256",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "rp00225",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00216",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00218",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00217",
                  "_type": "span",
                  "marks": [],
                  "text": "Item"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00219",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00221",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00220",
                  "_type": "span",
                  "marks": [],
                  "text": "Details"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00222",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00224",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00223",
                  "_type": "span",
                  "marks": [],
                  "text": "Rationale"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00235",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00226",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00228",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00227",
                  "_type": "span",
                  "marks": [],
                  "text": "Minimum Size"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00229",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00231",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00230",
                  "_type": "span",
                  "marks": [],
                  "text": "6 WEKA Servers 2 NVIDIA DGX systems"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00232",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00234",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00233",
                  "_type": "span",
                  "marks": [],
                  "text": "Minimum size requirements"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00245",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00236",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00238",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00237",
                  "_type": "span",
                  "marks": [],
                  "text": "Scale approach"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00239",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00241",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00240",
                  "_type": "span",
                  "marks": [],
                  "text": "Incremental, modular scale"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00242",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00244",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00243",
                  "_type": "span",
                  "marks": [],
                  "text": "Allow for growth from PoC to a massive scale"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00255",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00246",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00248",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00247",
                  "_type": "span",
                  "marks": [],
                  "text": "Scale Units"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00249",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00251",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00250",
                  "_type": "span",
                  "marks": [],
                  "text": "Servers"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00252",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00254",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00253",
                  "_type": "span",
                  "marks": [],
                  "text": "Granular scale to precisely meets the capacity demands."
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

**General Design Decisions: Networking**

```json
{
  "_key": "rp00289",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "rp00268",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00259",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00261",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00260",
                  "_type": "span",
                  "marks": [],
                  "text": "Item"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00262",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00264",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00263",
                  "_type": "span",
                  "marks": [],
                  "text": "Details"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00265",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00267",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00266",
                  "_type": "span",
                  "marks": [],
                  "text": "Rationale"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00278",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00269",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00271",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00270",
                  "_type": "span",
                  "marks": [],
                  "text": "Compute/DGX"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00272",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00274",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00273",
                  "_type": "span",
                  "marks": [],
                  "text": "InfiniBand is recommended"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00275",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00277",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00276",
                  "_type": "span",
                  "marks": [],
                  "text": "The GPU-to-GPU network needs to be fast"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00288",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00279",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00281",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00280",
                  "_type": "span",
                  "marks": [],
                  "text": "Storage"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00282",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00284",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00283",
                  "_type": "span",
                  "marks": [],
                  "text": "InfiniBand or 200GB"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00285",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00287",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00286",
                  "_type": "span",
                  "marks": [],
                  "text": "WEKA recommends 200GB"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

### NVIDIA DGX BasePOD Sizing

AI is powering mission-critical use cases in every industry—from healthcare to manufacturing to financial services. The DGX BasePOD™ reference architecture provides the critical foundation on which business transformation is realized and AI applications are born.

DGX BasePOD is an all-inclusive system that includes the following components:

- NVIDIA – 2 or more DGX A100
- Networking for the DGX compute layer and the storage layer
- Accelerated Storage Fabric

> **NOTE: **It’s always a good practice to add a buffer for contingency and growth.

### Network

The solution tested for this RA consisted of four DGX A100 systems connected to two NVIDIA Mellanox SN3700 Ethernet switches with two 100 Gb/s network connections per DGX A100 system. Each NIC was connected to a separate NVIDIA Mellanox SN3700 switch for eight network links to the storage system. Each DGX A100 system also had eight single-port Mellanox ConnectX-6 200Gb/s HDR InfiniBand ports (also configurable as 200Gb/s Ethernet ports) for inter-GPU system communication.

The HPE ProLiant DL325 Gen10 Plus servers were each configured with two ConnectX-6 200Gb/s Ethernet NICs. Each NIC was connected to a separate NVIDIA Mellanox SN3700 switch for 16 100Gb/s network links.

The network switch was configured with a message transmission unit (MTU) size of 9000.

WEKA AI RA does not require RoCE to be configured on any portion of the design and does not require priority flow control (PFC) to be configured on the network switch, greatly simplifying the network deployment.


![Network architecture: WekaFS on HPE DL325 servers over SN3700 storage fabric to four DGX A100 systems and a QM8700 compute fabric](https://cdn.sanity.io/images/ult5g8gw/production/3bc8b38e05251403c7c82124ed57e1982a476139-1328x1248.png)


Figure 7. Network Architecture

## 6. Validation and Benchmarking

We conducted this validation using synthetic benchmark utilities and DL benchmark tests to establish the baseline performance and operation of the system. Each test described in this section was performed with the specific equipment and software listed in the Technology Requirements section. The tests outlined in Table 3 were performed with one, two, three, and four DGX A100 systems to validate the essential operation, scalability, and performance of the deployed infrastructure.

### Hardware Requirements

```json
{
  "_key": "rp00357",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "rp00328",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00322",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00324",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00323",
                  "_type": "span",
                  "marks": [],
                  "text": "Hardware"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00325",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00327",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00326",
                  "_type": "span",
                  "marks": [],
                  "text": "Quantity"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00335",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00329",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00331",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00330",
                  "_type": "span",
                  "marks": [],
                  "text": "DGX A100 systems"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00332",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00334",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00333",
                  "_type": "span",
                  "marks": [],
                  "text": "4"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00342",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00336",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00338",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00337",
                  "_type": "span",
                  "marks": [],
                  "text": "HPE ProLiant DL325 Gen10 Plus Server"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00339",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00341",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00340",
                  "_type": "span",
                  "marks": [],
                  "text": "8, includes per server AMD EPYC™ 7402 CPU, 132GB RAM, 7 x 15.3TB Micron 9300 Pro NVMe SSDs per server, 2 x 200Gb/s ConnectX6 NVIVIA NICs"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00349",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00343",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00345",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00344",
                  "_type": "span",
                  "marks": [],
                  "text": "SN3700C Ethernet switch (storage fabric)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00346",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00348",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00347",
                  "_type": "span",
                  "marks": [],
                  "text": "2"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00356",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00350",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00352",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00351",
                  "_type": "span",
                  "marks": [],
                  "text": "QM8700 InfiniBand switch (compute fabric)"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00353",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00355",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00354",
                  "_type": "span",
                  "marks": [],
                  "text": "2"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

### Software Requirements

```json
{
  "_key": "rp00417",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "rp00370",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00364",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00366",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00365",
                  "_type": "span",
                  "marks": [],
                  "text": "Software"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00367",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00369",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00368",
                  "_type": "span",
                  "marks": [],
                  "text": "Version"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00377",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00371",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00373",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00372",
                  "_type": "span",
                  "marks": [],
                  "text": "WekaFS"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00374",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00376",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00375",
                  "_type": "span",
                  "marks": [],
                  "text": "3.9.0"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00384",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00378",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00380",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00379",
                  "_type": "span",
                  "marks": [],
                  "text": "Server OS"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00381",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00383",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00382",
                  "_type": "span",
                  "marks": [],
                  "text": "CentOS 8.2"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00391",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00385",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00387",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00386",
                  "_type": "span",
                  "marks": [],
                  "text": "DGX OS"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00388",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00390",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00389",
                  "_type": "span",
                  "marks": [],
                  "text": "4.99.9"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00398",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00392",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00394",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00393",
                  "_type": "span",
                  "marks": [],
                  "text": "Docker container platform"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00395",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "05ce7d5e9bb9",
              "_type": "block",
              "children": [
                {
                  "_key": "c4e01d2a8d62",
                  "_type": "span",
                  "marks": [],
                  "text": ""
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00402",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00399",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00401",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00400",
                  "_type": "span",
                  "marks": [],
                  "text": "Container version"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00361",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00363",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00362",
                  "_type": "span",
                  "marks": [
                    "rp00360"
                  ],
                  "text": "nvcr.io/nvidia/mxnet:20.06-py3 – MLPerf test tensorflow:20.05-tf2-py3 – other tests"
                }
              ],
              "markDefs": [
                {
                  "_key": "rp00360",
                  "_type": "articleBodyLink",
                  "actionType": "external",
                  "externalUrl": "http://nvcr.io/nvidia/mxnet"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00409",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00403",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00405",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00404",
                  "_type": "span",
                  "marks": [],
                  "text": "OFED version"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00406",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00408",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00407",
                  "_type": "span",
                  "marks": [],
                  "text": "19.03.8"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00416",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00410",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "eb8e47be9bd1",
              "_type": "block",
              "children": [
                {
                  "_key": "4744cb91babc",
                  "_type": "span",
                  "marks": [],
                  "text": ""
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00413",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00415",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00414",
                  "_type": "span",
                  "marks": [],
                  "text": "OFED-internal-5.0-2.1.8"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

### Benchmarks

**Tests Performed**

```json
{
  "_key": "rp00461",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "rp00444",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00438",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00440",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00439",
                  "_type": "span",
                  "marks": [],
                  "text": "Test Name"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00441",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00443",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00442",
                  "_type": "span",
                  "marks": [],
                  "text": "Source Code"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00448",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00445",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00447",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00446",
                  "_type": "span",
                  "marks": [],
                  "text": "NVIDIA NCCL all_reduce_perf test"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00423",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00425",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00424",
                  "_type": "span",
                  "marks": [
                    "rp00422"
                  ],
                  "text": "https://github.com/NVIDIA/nccl-tests"
                }
              ],
              "markDefs": [
                {
                  "_key": "rp00422",
                  "_type": "articleBodyLink",
                  "actionType": "external",
                  "externalUrl": "https://github.com/NVIDIA/nccl-tests"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00452",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00449",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00451",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00450",
                  "_type": "span",
                  "marks": [],
                  "text": "FIO bandwidth and IOPS test"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00427",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00429",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00428",
                  "_type": "span",
                  "marks": [
                    "rp00426"
                  ],
                  "text": "https://fio.readthedocs.io/en/latest/fio_doc.html#source"
                }
              ],
              "markDefs": [
                {
                  "_key": "rp00426",
                  "_type": "articleBodyLink",
                  "actionType": "external",
                  "externalUrl": "https://fio.readthedocs.io/en/latest/fio_doc.html#source"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00456",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00453",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00455",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00454",
                  "_type": "span",
                  "marks": [],
                  "text": "Mdtest for metadata performance"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00431",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00433",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00432",
                  "_type": "span",
                  "marks": [
                    "rp00430"
                  ],
                  "text": "https://github.com/hpc/ior"
                }
              ],
              "markDefs": [
                {
                  "_key": "rp00430",
                  "_type": "articleBodyLink",
                  "actionType": "external",
                  "externalUrl": "https://github.com/hpc/ior"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00460",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00457",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00459",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00458",
                  "_type": "span",
                  "marks": [],
                  "text": "MLPerf Training – ResNet-50"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00435",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00437",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00436",
                  "_type": "span",
                  "marks": [
                    "rp00434"
                  ],
                  "text": "https://github.com/mlperf/training"
                }
              ],
              "markDefs": [
                {
                  "_key": "rp00434",
                  "_type": "articleBodyLink",
                  "actionType": "external",
                  "externalUrl": "https://github.com/mlperf/training"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

Tests completed for this RA. The following sections describe details and results for each of these tests.

### NVIDIA NCCL all_reduce_perf Test

The NVIDIA Collective Communications Library (NCCL) tests the maximum scalability across multiple DGX A100 systems. Within a scenario, the bottleneck should be the NVIDIA NVLink high-speed interconnect bandwidth. Across various methods, the bottleneck should be from whatever InfiniBand or RoCE-enabled Ethernet adapters are assigned for GPU-to-GPU communication across DGX A100 systems.

The results in Figure 4 show that the single system inter-GPU bandwidth reaches NVLink interconnect capabilities. Multi-system inter-GPU bandwidth goes to the aggregate bandwidth of all IB or Ethernet adapters assigned to the test.


![Bar chart of NCCL bandwidth in GB/s across one to four DGX A100 systems](https://cdn.sanity.io/images/ult5g8gw/production/0a7ebd0e3cc1828883f660431624619f1056c0d6-1284x761.png)


Figure 8. NCCL bandwidth test results (GB/s)

### FIO Bandwidth and IOPS Tests

The FIO benchmark provides a baseline I/O testing to measure the combined RA’s maximum raw I/O performance capability. The storage system I/O was tested with FIO (Flexible I/O), a versatile I/O workload generator that can simulate different I/O patterns according to the use case. Two separate I/O configurations were measured the first measured maximum bandwidth from the storage system, and the second measured the maximum I/O operations per second (IOPS). Both designs were run using 100% reads and 100% writes.

Results of the test in Figure 5 demonstrate the linear read performance scalability as more DGX A100 systems were added to the test. Write performance reaches a maximum at a single DGX A100 system as the storage system’s aggregate NVMe SSD write throughput was quickly reached. Similarly, Figure 6 shows the linear scalability of Read throughput IOPS, while the Write throughput IOPS hit a limit at three DGX A100 systems. This is consistent with what is expected from the limited number of drives in the configuration rather than a limitation of WekaFS, the HPE DL325, and DGX A110 systems.


![Grouped bar chart of FIO read and write bandwidth in GB/s across one to four DGX A100 systems](https://cdn.sanity.io/images/ult5g8gw/production/219427488513a968bfa6fb12808f98b08939b6e9-1284x761.png)


Figure 9. FIO bandwidth test results (GB/s)


![Grouped bar chart of FIO read and write IOPS across one to four DGX A100 systems](https://cdn.sanity.io/images/ult5g8gw/production/872753d5132ee1f72a40801be46c59b06cf28ef9-1284x761.png)


Figure 10. FIO IOPS test results (operations/s)

### mdtest

mdtest is a file system metadata performance test designed to run in a message passing (MPI) cluster environment with a parallel file system. In each iteration of the trial, each MPI task creates stats, removes the specified number of directories and files, and measures the performance in ops/second. After all the iterations are complete, the maximum, minimum, mean ops/sec, and the std. Deviations are reported for each operation. The results in Figure 7 again demonstrate the linear performance scaling of WekaFS as the number of DGX A100 systems increases.


![Grouped bar chart of mdtest directory creation and directory stat operations per second across one to four DGX A100 systems](https://cdn.sanity.io/images/ult5g8gw/production/b68fead61e616587a503c298bb769de815b699ed-596x340.png)


Figure 11. mdtest results for directory creation and directory stat

### MLPerf ResNet-50 Test

MLPerf is the industry standard set of benchmark implementations of neural networks. The focus is on the ResNet-50 neural network, a well-known image classification network that can be used with the ImageNet dataset. It is computationally intensive yet capable of driving meaningful storage I/O.

Configurations were tested with one, two, and four DGX A100 systems. Slurm, NVIDIA Pyxis, and Enroot software were used to coordinate the work across all DGX A100 systems involved in the task. The MLPerf v0.7 implementation of ResNet-50 provided by NVIDIA for this RA was built with the MXNet framework. The training data is ImageNet, formatted using RecordIO. The results presented to maintain a uniform batch size per system of 408 images as the workload is scaled (weak scaling). DALI is configured not to use mmap.

The training benchmark was measured at Epoch 0 and compared to the overall run average time, which provides insight into the ability of the storage system to keep up with the read bandwidth demands of a complete training job. Epoch 0 is the most IO-intensive portion of the MLPerf benchmark run and will significantly impact insights on time. The results shown in figure 8 demonstrate that the storage system could provide the exact images/second for Epoch 0 as an overall average, validating that the storage system is not the bottleneck for the workload. The results also demonstrate the linear scalability of WekaFS; as more DGX A100 systems are added, performance scales linearly with the additional compute power.

Many infrastructure architects are concerned that system utilization will deteriorate as more GPU systems are utilized to accelerate the training pipeline. The results in figure 8 also demonstrate that WekaFS delivers linear scaling of time to insights as more DGX A100 systems are added to the workload. The time to insights almost halved from 41 minutes to 22 minutes, going from one to two DGX A100 systems and again to 12 minutes when scaled to 4 systems.


![MLPerf ResNet-50 dual-axis chart of images per second and overall training time for one, two, and four DGX A100 systems](https://cdn.sanity.io/images/ult5g8gw/production/7529bc1680067c2cfde63bd7b8ddc532e98a5ee7-1284x761.png)


Figure 12. MLPerf ResNet-50 Images per Second and Overall Time

## 7. Conclusion

AI, fueled by rapid innovation in DL solutions, is becoming common in many industries. Organizations that invest in AI and turn their data into intelligence and new products will lead their competition. While many organizations want to kickstart their AI initiatives, challenges in building a scalable and AI-optimized infrastructure often hold them back. Traditional compute infrastructures are unsuitable for demanding AI workloads due to slow legacy CPU architectures and varying system requirements. This drives up complexity, increases cost, and limits scale. Engineers at WEKA and NVIDIA partnered to architect a scalable and robust infrastructure that pushes the boundaries of AI innovation and performance.

The results show robust linear performance scalability from one to four DGX A100 systems, allowing organizations to start small and grow seamlessly as AI projects ramp. The results demonstrate that scaling GPU infrastructure to accelerate time to insights will be well supported by WEKA AI. The validated WEKA AI RA makes it easy for teams to focus on developing new products and gain new faster insights with AI/ML.

## 8. Appendix

### References

```json
{
  "_key": "rp00604",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "rp00543",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00537",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00539",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00538",
                  "_type": "span",
                  "marks": [],
                  "text": "Product & Technology"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00540",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00542",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00541",
                  "_type": "span",
                  "marks": [],
                  "text": "URL"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00550",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00544",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00546",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00545",
                  "_type": "span",
                  "marks": [],
                  "text": "WEKA Data Platform"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00547",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "7a14d7313551",
              "_type": "block",
              "children": [
                {
                  "_key": "6cb842412ca9",
                  "_type": "span",
                  "marks": [],
                  "text": ""
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00554",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00551",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00553",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00552",
                  "_type": "span",
                  "marks": [],
                  "text": "WekaFS Datasheet"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00506",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00508",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00507",
                  "_type": "span",
                  "marks": [
                    "rp00505"
                  ],
                  "text": "https://www.weka.io/wp-content/uploads/files/2020/03/WekaFS-DS-W01R14DS201808.pdf"
                }
              ],
              "markDefs": [
                {
                  "_key": "rp00505",
                  "_type": "articleBodyLink",
                  "actionType": "external",
                  "externalUrl": "https://www.weka.io/wp-content/uploads/files/2020/03/WekaFS-DS-W01R14DS201808.pdf"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00558",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00555",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00557",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00556",
                  "_type": "span",
                  "marks": [],
                  "text": "WEKA Architecture White Paper"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00510",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00512",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00511",
                  "_type": "span",
                  "marks": [
                    "rp00509"
                  ],
                  "text": "https://www.weka.io/wp-content/uploads/files/2017/12/Architectural_WhitePaper-W02R6WP201812-1.pdf"
                }
              ],
              "markDefs": [
                {
                  "_key": "rp00509",
                  "_type": "articleBodyLink",
                  "actionType": "external",
                  "externalUrl": "https://www.weka.io/wp-content/uploads/files/2017/12/Architectural_WhitePaper-W02R6WP201812-1.pdf"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00565",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00559",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00561",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00560",
                  "_type": "span",
                  "marks": [],
                  "text": "NVIDIA DGX A100"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00562",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "7f9abb17a93f",
              "_type": "block",
              "children": [
                {
                  "_key": "7a5bc344cf72",
                  "_type": "span",
                  "marks": [],
                  "text": ""
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00569",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00566",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00568",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00567",
                  "_type": "span",
                  "marks": [],
                  "text": "NVIDIA DGX A100 System"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00514",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00516",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00515",
                  "_type": "span",
                  "marks": [
                    "rp00513"
                  ],
                  "text": "https://www.nvidia.com/en-us/data-center/dgx-a100/"
                }
              ],
              "markDefs": [
                {
                  "_key": "rp00513",
                  "_type": "articleBodyLink",
                  "actionType": "external",
                  "externalUrl": "https://www.nvidia.com/en-us/data-center/dgx-a100/"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00573",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00570",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00572",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00571",
                  "_type": "span",
                  "marks": [],
                  "text": "NVIDIA A100 Tensor core GPU"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00518",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00520",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00519",
                  "_type": "span",
                  "marks": [
                    "rp00517"
                  ],
                  "text": "https://www.nvidia.com/en-us/data-center/a100/"
                }
              ],
              "markDefs": [
                {
                  "_key": "rp00517",
                  "_type": "articleBodyLink",
                  "actionType": "external",
                  "externalUrl": "https://www.nvidia.com/en-us/data-center/a100/"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00580",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00574",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00576",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00575",
                  "_type": "span",
                  "marks": [],
                  "text": "NVIDIA Mellanox Networking"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00577",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "c912f9e301c6",
              "_type": "block",
              "children": [
                {
                  "_key": "1c1255ccbf02",
                  "_type": "span",
                  "marks": [],
                  "text": ""
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00584",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00581",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00583",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00582",
                  "_type": "span",
                  "marks": [],
                  "text": "NVIDIA Mellanox Spectrum SN3000"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00522",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00524",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00523",
                  "_type": "span",
                  "marks": [
                    "rp00521"
                  ],
                  "text": "https://www.mellanox.com/products/ethernet-switches/sn3000"
                }
              ],
              "markDefs": [
                {
                  "_key": "rp00521",
                  "_type": "articleBodyLink",
                  "actionType": "external",
                  "externalUrl": "https://www.mellanox.com/products/ethernet-switches/sn3000"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00588",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00585",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00587",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00586",
                  "_type": "span",
                  "marks": [],
                  "text": "NVIDIA Mellanox Quantum QM8700"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00526",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00528",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00527",
                  "_type": "span",
                  "marks": [
                    "rp00525"
                  ],
                  "text": "https://www.mellanox.com/products/infiniband-switches/QM8700"
                }
              ],
              "markDefs": [
                {
                  "_key": "rp00525",
                  "_type": "articleBodyLink",
                  "actionType": "external",
                  "externalUrl": "https://www.mellanox.com/products/infiniband-switches/QM8700"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00595",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00589",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00591",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00590",
                  "_type": "span",
                  "marks": [],
                  "text": "Machine Learning Frameworks"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00592",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "8386713a1512",
              "_type": "block",
              "children": [
                {
                  "_key": "b30a694b506a",
                  "_type": "span",
                  "marks": [],
                  "text": ""
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00599",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00596",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00598",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00597",
                  "_type": "span",
                  "marks": [],
                  "text": "TensorFlow"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00530",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00532",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00531",
                  "_type": "span",
                  "marks": [
                    "rp00529"
                  ],
                  "text": "https://www.tensorflow.org/"
                }
              ],
              "markDefs": [
                {
                  "_key": "rp00529",
                  "_type": "articleBodyLink",
                  "actionType": "external",
                  "externalUrl": "https://www.tensorflow.org/"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "rp00603",
      "_type": "row",
      "cells": [
        {
          "_key": "rp00600",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00602",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00601",
                  "_type": "span",
                  "marks": [],
                  "text": "Horovod"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "rp00534",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "rp00536",
              "_type": "block",
              "children": [
                {
                  "_key": "rp00535",
                  "_type": "span",
                  "marks": [
                    "rp00533"
                  ],
                  "text": "https://eng.uber.com/horovod/"
                }
              ],
              "markDefs": [
                {
                  "_key": "rp00533",
                  "_type": "articleBodyLink",
                  "actionType": "external",
                  "externalUrl": "https://eng.uber.com/horovod/"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

## 9. About WEKA

WEKA offers a modern subscription software-based data platform delivering 10x+ performance and scale demanded by today’s cloud and AI workloads. With the simplicity of NAS, the performance of SAN or DAS, and the scale of object storage, no more compromises between Simplicity, Speed, or Scale. Learn more at [www.weka.io](http://www.weka.io) or follow us on [Twitter @WekaIO](https://twitter.com/wekaio).

[Download PDF](/api/resource-pdf?slug=scaling-deep-learning-with-weka-and-dgx-a100-basepod)
