# NeuralMesh Data Resiliency Technical Guide This NeuralMesh data resiliency guide shows how scale becomes an advantage: failures are expected, and larger clusters recover faster without disrupting workloads. **Type:** Technical Brief ```json { "_key": "de4073735b50", "_type": "textCallout", "content": [ { "_key": "rp00004", "_type": "block", "children": [ { "_key": "rp00003", "_type": "span", "marks": [], "text": "Data resiliency has become a primary design constraint for modern AI, HPC, and large-scale analytics environments. As infrastructure scales, traditional storage architectures experience increased failure rates, longer recovery times, and greater operational risk. In many systems, scale amplifies fragility rather than resilience." } ], "markDefs": [], "style": "normal" }, { "_key": "rp00006", "_type": "block", "children": [ { "_key": "rp00005", "_type": "span", "marks": [], "text": "WEKA NeuralMesh takes a fundamentally different architectural approach. Rather than treating resiliency as a static feature layered onto storage, NeuralMesh is designed such that resiliency emerges and improves as the system scales. By distributing metadata, recovery logic, and fault handling across the entire cluster, NeuralMesh converts scale from a liability into an advantage." } ], "markDefs": [], "style": "normal" } ], "markDefs": null, "title": "WEKA NeuralMesh Native Multi-Tenancy" } ``` ## Executive Overview: A Field Perspective ### Evaluate resiliency by impact, not by fault count Resiliency should be evaluated by impact, not by the presence of failures. In modern distributed environments, component failures, transient network events, and rolling upgrades are expected operating conditions. The relevant question for architects and operators is whether these events affect data availability, user experience, application behavior, or require immediate human intervention. When data remains accessible and workloads continue without disruption, the system is functioning as intended, regardless of background fault activity. WEKA’s NeuralMesh resiliency is driven by an architectural model that differs fundamentally from traditional RAID- or array-based systems. In conventional storage, increasing capacity and host count expands rebuild scope and lengthens recovery times, increasing risk as systems scale. NeuralMesh inverts this model. As cluster size increases, rebuild capacity scales with it because every compute core across the cluster participates in reconstruction. Larger clusters therefore recover faster and spend less time in reduced-protection states, **making scale a resiliency advantage rather than a liability.** ### Protection is automatic and self-adjusting Data protection behavior is controlled through configurable protection levels combined with dynamic rebuild rates. Protection levels determine the number of simultaneous failures that can be tolerated, while rebuild rates automatically adjust based on the current redundancy state. When the system still has available parity, rebuild activity is intentionally conservative to avoid unnecessary performance impact. When parity is exhausted, rebuild rates increase automatically to restore protection as quickly as possible. This behavior is deterministic, consistent, and does not require operator intervention, even under compound failure scenarios. A key operational distinction in NeuralMesh is the separation of data re-protection from physical hardware remediation. **Once data has been fully rebuilt and protection restored, failed hosts no longer represent a data risk.** Clusters frequently continue operating with fewer backends than originally configured, sometimes for extended periods, without compromising data availability or integrity. The only functional impact is reduced aggregate performance capacity. This allows infrastructure teams to defer host replacement, align repairs with maintenance windows, and avoid the urgency traditionally associated with hardware failures. ### Data re-protection and hardware remediation are separate problems Many industry-standard resiliency metrics fail to capture this behavior. Drive rebuild time, commonly requested in RFPs, is typically a non-issue in NeuralMesh environments, often completing in seconds and providing little insight into real system risk. **More meaningful metrics include time spent without full redundancy, probability of additional failure during that window, and observable impact to applications.** NeuralMesh is explicitly designed to minimize these exposure windows while maintaining consistent application performance. ### The metrics that actually matter From a probabilistic standpoint, the likelihood of compounded failures during rebuild is extremely low when modeled using realistic MTBF assumptions ( 1 week). As rebuild times decrease with scale, the statistical risk of additional failures during a reduced-protection window drops accordingly. In large clusters, this probability is often well below one percent. This aligns with large-scale infrastructure reliability models used by hyperscalers and provides a more accurate representation of risk than static availability claims. These characteristics have been validated through testing and WEKA’s internal workbench tools. Rebuild behavior has been measured across different cluster sizes and protection levels by intentionally inducing failures and observing time-to-reprotection. In practice, actual rebuild times meet or exceed calculated estimates, confirming that WEKA’s resiliency model reflects real-world behavior rather than theoretical assumptions. Operational data from customer production environments further reinforces this model. As an illustrative example, one internal sample of 59 recovery events showed a minimum recovery time of 5 seconds, a median of 4.5 minutes, an average of 8 minutes, and a maximum of 1.33 hours results that varied by cluster size and failure type. The consistent pattern: recovery completes quickly and without application disruption. It is common to observe operational clusters that have experienced multiple backend failures yet remain fully protected, with no active rebuilds in progress. In these cases, missing hosts are treated as a capacity consideration rather than a resiliency concern. The system has already completed its primary responsibility: protecting data and maintaining availability. ## The Resiliency Challenge at Scale As infrastructure environments grow, failures are inevitable. Large AI and analytics clusters routinely experience concurrent disk failures, host outages, and network issues. Traditional storage systems struggle in these environments due to several structural limitations: - Centralized or semi-centralized metadata services become bottlenecks during recovery - Serial rebuild processes cannot keep pace with failure rates - Rigid RAID or erasure coding schemes concentrate recovery work - Static failure domains increase impacted areas - Operationally intensive recovery workflows In these systems, adding more hardware often increases the probability of failures and extends rebuild windows and creates a negative scaling curve for resiliency. ## NeuralMesh Design Principles for Resiliency WEKA’s NeuralMesh is built on a set of design principles that directly address these common failure characteristics of large-scale environments: ### Fully Distributed Metadata Metadata is distributed across all participating hosts. There is no central metadata controller or chokepoint. Every host contributes to metadata processing, enabling parallel recovery and eliminating bottlenecks during failure scenarios. It’s like crowdsourcing the recovery effort. ### Shared-Nothing Recovery Logic Recovery and rebuild logic is embedded throughout the data plane. Any compute host can participate in recovery operations, allowing the system to scale the rebuild bandwidth linearly with cluster size. ### Fine-Grained Failure Awareness NeuralMesh tracks data placement and health at ultra-fine granularity. This enables precise isolation of failures and minimizes unnecessary rebuild work. ### Software-Defined Fault Domains Failure domains are defined and enforced in software, allowing data placement and recovery behavior to align with physical realities such as racks, power domains, or availability zones. ## Separation of Data Re-Protection from Hardware Remediation Once data has been fully rebuilt and protection restored, failed hosts no longer represent a data risk. Clusters can continue operating with fewer backends for extended periods without compromising data availability or integrity. Infrastructure teams can defer host replacement and align repairs with maintenance windows. ## Configurable Protection with Dynamic Rebuild Rates Protection levels (D+P) are configurable, supporting stripe widths from 5 to 16 failure domains with D+2 or D+4 parity. D+2 is recommended for most environments; D+4 provides increased redundancy in large-scale and converged deployments. Rebuild rates automatically adjust based on current redundancy state conservative when parity is available, aggressive when parity is exhausted. This behavior is deterministic and requires no operator intervention. ## Control Plane vs Data Plane Resiliency ### Data Plane Behavior The data plane is responsible for all I/O, metadata access, and recovery operations. It is designed to continue operating normally during failures, with recovery occurring transparently in the background. Users will never know that a failure occurred. ### Control Plane Independence The control plane manages configuration and orchestration but is not on the critical path for data access or recovery. This separation ensures that management events or outages do not compromise data availability. ## How NeuralMesh Gets More Resilient as You Scale ### Parallelism as a First-Class Property As WEKA hosts are added to a NeuralMesh cluster, both data capacity and recovery capability increase. Rebuild operations are performed in parallel across all available resources (disks, hosts, and racks), dramatically reducing recovery times compared to systems with centralized rebuild processes. ### Failure Containment Because data and metadata are widely distributed, failures are contained to the smallest possible scope. A disk failure does not cascade into host-wide or cluster-wide disruption, and multiple independent failures can be handled concurrently. ### Adaptive Data Placement NeuralMesh continuously adapts data placement to reflect current cluster health. During and after failures, data is rebalanced intelligently without operator intervention, restoring redundancy while maintaining application performance. A drive or host could be down for weeks or months as the data itself has already been reprotected and resiliency restored. ### Network Path Diversity NeuralMesh leverages multiple network paths and distributed communication patterns. Recovery traffic is spread across the fabric rather than concentrated on a small number of hosts or interfaces. ## Detailed Resiliency Scenario Analysis This section provides a concrete, end-to-end resiliency scenario that Sales Engineers can use to walk technical stakeholders through exactly how NeuralMesh behaves under failure conditions. The goal is to translate architectural principles into observable system behavior. ### Example Cluster Configuration Consider a production AI training environment with the following characteristics: - **Cluster Size: **96 backend servers - **Drives per backend server: **16 NVMe SSDs - **Total Drives: **1,536 NVMe devices - **Usable Capacity: **Multi-petabyte scale - **Network: **100Gb RDMA-enabled fabric - **Workloads: **Concurrent AI training jobs and POSIX-based data preprocessing - **Failure Domains Defined: **Disk → Backend Server → Rack Data is protected using [Distributed Data Protection](/resources/white-paper/distributed-data-protection/), with data and parity stripes intentionally placed across distinct WEKA hosts and racks according to software-defined fault domain policies. ### Rebuild Time Comparison: Using Real-World Scale Data As the table below shows, a larger NeuralMesh cluster rebuilds data significantly faster, and increased scale shortens the time the system spends fully exposed. These results are based on the same protection scheme and failure conditions, differing only in cluster size. In this example, the system is configured with 16+4 erasure coding. After four simultaneous failures, the system has no remaining redundancy (a fifth failure would result in data unavailability). Rebuilding even one level of protection (back to +3) is therefore critical, as it restores fault tolerance and ensures another protected copy of the data. In the 100-server cluster, the return to protected status happens in approximately 1 minute. In the 50-server cluster, the same operation takes roughly 10 minutes, creating a significantly longer risk window. This demonstrates how larger clusters achieve faster rebuild times by leveraging all available cluster compute and network resources. ```json { "_key": "rp00204", "_type": "richTableBlock", "headerRows": 1, "markDefs": null, "rows": [ { "_key": "rp00163", "_type": "row", "cells": [ { "_key": "rp00154", "_type": "richTableCell", "content": [ { "_key": "rp00156", "_type": "block", "children": [ { "_key": "rp00155", "_type": "span", "marks": [], "text": "Protection Level" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00157", "_type": "richTableCell", "content": [ { "_key": "rp00159", "_type": "block", "children": [ { "_key": "rp00158", "_type": "span", "marks": [], "text": "Rebuild Time (100 Servers) (hh:mm:ss)" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00160", "_type": "richTableCell", "content": [ { "_key": "rp00162", "_type": "block", "children": [ { "_key": "rp00161", "_type": "span", "marks": [], "text": "Rebuild Time (50 Servers) (hh:mm:ss)" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00173", "_type": "row", "cells": [ { "_key": "rp00164", "_type": "richTableCell", "content": [ { "_key": "rp00166", "_type": "block", "children": [ { "_key": "rp00165", "_type": "span", "marks": [], "text": "16 + 1" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00167", "_type": "richTableCell", "content": [ { "_key": "rp00169", "_type": "block", "children": [ { "_key": "rp00168", "_type": "span", "marks": [], "text": "00:01:01" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00170", "_type": "richTableCell", "content": [ { "_key": "rp00172", "_type": "block", "children": [ { "_key": "rp00171", "_type": "span", "marks": [], "text": "00:09:44" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00183", "_type": "row", "cells": [ { "_key": "rp00174", "_type": "richTableCell", "content": [ { "_key": "rp00176", "_type": "block", "children": [ { "_key": "rp00175", "_type": "span", "marks": [], "text": "16 + 2" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00177", "_type": "richTableCell", "content": [ { "_key": "rp00179", "_type": "block", "children": [ { "_key": "rp00178", "_type": "span", "marks": [], "text": "00:03:28" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00180", "_type": "richTableCell", "content": [ { "_key": "rp00182", "_type": "block", "children": [ { "_key": "rp00181", "_type": "span", "marks": [], "text": "00:25:30" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00193", "_type": "row", "cells": [ { "_key": "rp00184", "_type": "richTableCell", "content": [ { "_key": "rp00186", "_type": "block", "children": [ { "_key": "rp00185", "_type": "span", "marks": [], "text": "16 + 3" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00187", "_type": "richTableCell", "content": [ { "_key": "rp00189", "_type": "block", "children": [ { "_key": "rp00188", "_type": "span", "marks": [], "text": "00:16:29" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00190", "_type": "richTableCell", "content": [ { "_key": "rp00192", "_type": "block", "children": [ { "_key": "rp00191", "_type": "span", "marks": [], "text": "01:05:47" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00203", "_type": "row", "cells": [ { "_key": "rp00194", "_type": "richTableCell", "content": [ { "_key": "rp00196", "_type": "block", "children": [ { "_key": "rp00195", "_type": "span", "marks": [], "text": "16 + 4" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00197", "_type": "richTableCell", "content": [ { "_key": "rp00199", "_type": "block", "children": [ { "_key": "rp00198", "_type": "span", "marks": [], "text": "01:22:58" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00200", "_type": "richTableCell", "content": [ { "_key": "rp00202", "_type": "block", "children": [ { "_key": "rp00201", "_type": "span", "marks": [], "text": "02:45:27" } ], "markDefs": [], "style": "normal" } ] } ] } ] } ``` ### Key Takeaways - The data volume and protection scheme are identical in both clusters. - Larger clusters dramatically reduce the exposure window after failures. - Recovery accelerates because rebuild work is spread across more CPUs, drives, and network paths. - Scale improves not just performance and capacity, but operational safety. A useful way to explain this to customers: - “In NeuralMesh, scale doesn’t just add storage it shrinks the window where you’re at risk.” - This framing is especially effective for customers operating large AI, analytics, or mission-critical environments where even short periods without redundancy matter. ## Failure Scenarios and System Response ### Disk Failure - Immediate detection and isolation - Parallel rebuild across the cluster - Minimal performance impact ### Host Failure - Data re-accessed from surviving hosts - Recovery bandwidth scales with cluster size - No centralized recovery bottleneck ### Rack or AZ Failure - Software-defined fault domains prevent data loss - Rapid re-establishment of redundancy - Predictable recovery behavior ## Failure Scenario 1: Disk Failure **Event: **One NVMe drive fails on a WEKA host while multiple AI training jobs are actively running. ### System Behavior - The failed device is detected immediately by the local host and marked unhealthy. - Only the specific data extents hosted on that device are affected. - Metadata services already know the full placement map of impacted extents. - No centralized coordinator is contacted to initiate recovery. ### Recovery Mechanics - Rebuild work is distributed across dozens of hosts in parallel. - Each participating host reconstructs small portions of missing data using surviving data and parity fragments. - Rebuilt extents are written to free space across healthy devices in the cluster. ### Observed Outcome - AI jobs continue running without interruption. - Rebuild traffic is spread evenly across the fabric. - Performance impact is minimal and predictable. ## Failure Scenario 2: WEKA Host Failure Under Load **Event: **A full WEKA host unexpectedly goes offline due to a hardware fault. ### System Behavior - All disks on the failed host are immediately treated as unavailable. - Client I/O is transparently redirected to surviving replicas or parity sources. - Metadata lookups continue without delay due to distributed metadata services. ### Recovery Mechanics - Recovery bandwidth scales with cluster size dozens of hosts participate simultaneously. - No single host becomes a rebuild bottleneck. - Recovery proceeds in the background while applications continue operating. ### Observed Outcome - No application-visible outage. - Rebuild time is bounded and largely independent of cluster size. - Larger clusters recover faster due to increased parallelism. ## Failure Scenario 3: Rack-Level Failure **Event: **An entire rack loses power, taking multiple WEKA hosts offline simultaneously. ### System Behavior - Software-defined fault domains ensure that no data stripe loses quorum. - Remaining racks contain sufficient data and parity to sustain operations. - No manual failover or administrator action is required. ### Recovery Mechanics - Recovery proceeds immediately using surviving racks. Redundancy is re-established automatically using surviving infrastructure no waiting for the failed rack to return. Once it comes back online or replacement capacity is added, full cluster capacity is restored. - Recovery work is distributed across all surviving racks. - The system prioritizes restoring protection levels while preserving foreground performance. ### Observed Outcome - Data remains available throughout the event. - Redundancy is restored in a controlled, predictable manner. - Operational risk does not increase with cluster size. ## Operational Resiliency Characteristics Resiliency should be evaluated by impact, not by the presence of failures. In modern distributed environments, component failures, transient network events, and rolling upgrades are expected operating conditions. The relevant question for architects and operators is whether these events affect data availability, user experience, application behavior, or require immediate human intervention. When data remains accessible and workloads continue without disruption, the system is functioning as intended, regardless of background fault activity. NeuralMesh is designed to deliver predictable and stable operational behavior even as failures occur and scale increases. Rebuild times remain consistent and bounded because recovery work is distributed across the entire cluster, allowing rebuild performance to scale with available resources rather than degrade under load. During degraded states, NeuralMesh avoids the performance cliffs commonly seen in traditional storage systems. Applications continue to run while recovery occurs in the background, with foreground I/O prioritized and rebuild activity carefully balanced to prevent disruption. This enables long-running AI, HPC, and analytics workloads to proceed without interruption or forced restarts. Recovery operations are fully automated and require no manual intervention from administrators. Failure detection, rebuild initiation, and restoration of protection levels occur transparently, reducing operational overhead and minimizing the risk of human error during critical events. NeuralMesh also maintains consistent resiliency behavior across mixed workloads. Whether servicing AI training jobs, POSIX-based pipelines, or analytics workloads simultaneously, the system applies the same protection, recovery, and performance principles, ensuring predictable outcomes regardless of workload composition. ## Field Perspective: Framing Resiliency for Technical Stakeholders Drive rebuild time is commonly requested in RFPs and used as a proxy for storage resiliency. In NeuralMesh, individual drive rebuild is typically a non-issue it completes in seconds. The more meaningful metrics are time spent without full redundancy, probability of an additional failure during that window, and observable impact to running applications. In large clusters, the probability of compounded failures during a rebuild window is often well below 1%. Operational data from production environments confirms that clusters with multiple backend failures remain fully protected with no active rebuilds in progress missing hosts are treated as capacity considerations, not resiliency concerns. ## Deployment Model Considerations NeuralMesh delivers consistent resiliency characteristics regardless of how it is deployed. Whether running on bare metal, in cloud-based environments, or within Kubernetes, the same architectural principles govern failure detection, recovery behavior, and rebuild performance. This consistency allows customers to reason about resiliency outcomes independently of infrastructure choices. Importantly, Kubernetes is treated strictly as a deployment and orchestration mechanism rather than a resiliency layer. NeuralMesh does not rely on Kubernetes primitives for data protection or recovery. As a result, failure semantics remain uniform across environments, and data availability is not impacted by control-plane events, scheduler behavior, or container-level disruptions. ## Summary WEKA NeuralMesh fundamentally redefines the relationship between scale and resiliency. By distributing metadata, recovery logic, and fault handling across the entire system, NeuralMesh becomes more resilient as it grows. This architecture enables predictable recovery, minimized blast radius, and operational simplicity in environments where failure is unavoidable. This enables customers to operate large, continuously active clusters with confidence, even in environments with frequent hardware churn. ```json { "_key": "17363d70b21d", "_type": "textCallout", "content": [ { "_key": "rp00373", "_type": "block", "children": [ { "_key": "rp00368", "_type": "span", "marks": [], "text": "Dig further into NeuralMesh by checking out our " }, { "_key": "a2c06fcf0650", "_type": "span", "marks": [ "8add216b206b" ], "text": "Reference Architecture" }, { "_key": "812fb90351b8", "_type": "span", "marks": [], "text": ". If you are interested in discussing how NeuralMesh can help you, " }, { "_key": "58e68b3377e4", "_type": "span", "marks": [ "3a98949700e0" ], "text": "click here to contact us" }, { "_key": "92c12411ce19", "_type": "span", "marks": [], "text": "." } ], "markDefs": [ { "_key": "8add216b206b", "_type": "link", "href": "/resources/white-paper/wekaio-architectural-whitepaper/", "target": "_self" }, { "_key": "3a98949700e0", "_type": "link", "href": "/company/contact-us/", "target": "_self" } ], "style": "normal" } ], "markDefs": null, "title": "Take the Next Step with NeuralMesh" } ``` [Download PDF](/api/resource-pdf?slug=neuralmesh-data-resiliency)