# Why Storage Matters for Spark Jobs: A TPC-DS Performance Analysis Storage matters for Spark jobs because ingest and prep add heavy overhead, especially with small files. TPC-DS benchmarks compare NeuralMesh latency to Amazon S3 as Parquet chunk size shrinks. **Type:** Technical Brief ```json { "_key": "6f9298807430", "_type": "textCallout", "content": [ { "_key": "rp00003", "_type": "block", "children": [ { "_key": "rp00002", "_type": "span", "marks": [], "text": "Data-intensive applications like Apache Spark suffer significant storage overhead during data ingest and preparation. Many datasets need pre-processing before they’re suitable for Spark jobs, even when Spark itself is doing that pre-processing. Replacing the storage with NeuralMesh can significantly boost Spark performance, especially for workloads with small file sizes. In this brief, we explore how small file sizes affect Spark’s performance and where NeuralMesh advantages begin to add value (hint: it’s not that small!)." } ], "markDefs": [], "style": "normal" } ], "markDefs": null, "title": "Why We Ran These Benchmarks" } ``` This brief details the methodology used during performance benchmarks, comparing [WEKA NeuralMesh™*](/product/how-it-works/) to Amazon S3** as storage backends for high-performance Apache Spark analytical workloads. Specifically, we focus on quantifying the latency impact of these two storage choices on TPC-DS queries, with a critical emphasis on how performance characteristics shift as the Parquet data file size—or "chunk size"—is intentionally reduced. Who will benefit from reading this? The objective is to provide data-driven insights to: - **Data Scientists and Data Engineers** interested in solving scaling issues and improving performance profiles when running Spark workloads on S3 object storage. - **Cloud Architects and IT professionals** evaluating storage backends for latency-sensitive, analytical Spark workloads. - **Practitioners and Architects** evaluating modern storage solutions for data lakes, particularly where workloads involve numerous smaller Parquet files which can significantly challenge object storage performance. ## Introduction: Small Files are a Big Bottleneck Apache Spark is the de facto standard for large-scale data processing in the data lake architecture. Spark (and most data processing engine) computation is tied to CPU architecture, which for OLAP type operations is very compute bound and limited. Networking and I/O also play a significant role in accelerating these workloads, but it's a three-legged stool, requiring a balance of each, particularly for I/O-intensive operations like Parquet file ingestion. While cloud object storage, such as Amazon S3, offers unparalleled scalability and durability, its inherent latency and metadata overhead can become performance bottlenecks, especially in scenarios involving a high volume of small files. Crawling buckets to populate Glue databases or other similar techniques aim to mitigate such shortcomings, doing so adds delays, cost and complexity, and does not solve the underlying issues. This study directly compares the performance of the WEKA NeuralMesh parallel filesystem, designed for high-throughput, low-latency access, against Amazon S3. We use the industry-standard TPC-DS benchmark to simulate complex, analytical workloads. The core hypothesis is that while performance may be comparable at optimal Parquet file sizes (monolithic, one or more tables per file), the performance degradation for object storage will accelerate significantly as chunk sizes decrease, a trend that WEKA's architecture is expected to mitigate. ## Benchmark Methodology: The benchmark employs a repeatable, controlled environment to isolate the performance impact of the storage backend. ### 3.1. Test Environment (Constant Parameters) ```json { "_key": "rp00081", "_type": "richTableBlock", "headerRows": 1, "markDefs": null, "rows": [ { "_key": "rp00038", "_type": "row", "cells": [ { "_key": "rp00032", "_type": "richTableCell", "content": [ { "_key": "rp00034", "_type": "block", "children": [ { "_key": "rp00033", "_type": "span", "marks": [], "text": "Component" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00035", "_type": "richTableCell", "content": [ { "_key": "rp00037", "_type": "block", "children": [ { "_key": "rp00036", "_type": "span", "marks": [], "text": "Configuration" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00045", "_type": "row", "cells": [ { "_key": "rp00039", "_type": "richTableCell", "content": [ { "_key": "rp00041", "_type": "block", "children": [ { "_key": "rp00040", "_type": "span", "marks": [], "text": "NeuralMesh Cluster" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00042", "_type": "richTableCell", "content": [ { "_key": "rp00044", "_type": "block", "children": [ { "_key": "rp00043", "_type": "span", "marks": [], "text": "v5.0.3.19; EC2 type i3en.2xlarge, 17 instances (Stripe size of 14+2 With 1 Virtual Hotspare); DPDK mode." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00052", "_type": "row", "cells": [ { "_key": "rp00046", "_type": "richTableCell", "content": [ { "_key": "rp00048", "_type": "block", "children": [ { "_key": "rp00047", "_type": "span", "marks": [], "text": "Spark Clients/Workers" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00049", "_type": "richTableCell", "content": [ { "_key": "rp00051", "_type": "block", "children": [ { "_key": "rp00050", "_type": "span", "marks": [], "text": "EC2 type m5dn.12xlarge, 7 instances." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00059", "_type": "row", "cells": [ { "_key": "rp00053", "_type": "richTableCell", "content": [ { "_key": "rp00055", "_type": "block", "children": [ { "_key": "rp00054", "_type": "span", "marks": [], "text": "Spark Stack" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00056", "_type": "richTableCell", "content": [ { "_key": "rp00058", "_type": "block", "children": [ { "_key": "rp00057", "_type": "span", "marks": [], "text": "Spark v3.5.5 (Hadoop3); Hadoop AWS v3.3.4; Java 11 (Amazon Corretto Devel)." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00066", "_type": "row", "cells": [ { "_key": "rp00060", "_type": "richTableCell", "content": [ { "_key": "rp00062", "_type": "block", "children": [ { "_key": "rp00061", "_type": "span", "marks": [], "text": "Parquet Generator" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00063", "_type": "richTableCell", "content": [ { "_key": "rp00065", "_type": "block", "children": [ { "_key": "rp00064", "_type": "span", "marks": [], "text": "DuckDB v1.4.0" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00073", "_type": "row", "cells": [ { "_key": "rp00067", "_type": "richTableCell", "content": [ { "_key": "rp00069", "_type": "block", "children": [ { "_key": "rp00068", "_type": "span", "marks": [], "text": "Workload Generator" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00070", "_type": "richTableCell", "content": [ { "_key": "rp00072", "_type": "block", "children": [ { "_key": "rp00071", "_type": "span", "marks": [], "text": "spark-sql-perf TPC-DS benchmark suite." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00080", "_type": "row", "cells": [ { "_key": "rp00074", "_type": "richTableCell", "content": [ { "_key": "rp00076", "_type": "block", "children": [ { "_key": "rp00075", "_type": "span", "marks": [], "text": "Benchmark Scale" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00077", "_type": "richTableCell", "content": [ { "_key": "rp00079", "_type": "block", "children": [ { "_key": "rp00078", "_type": "span", "marks": [], "text": "TPC-DS scale factor 1000, 5 iterations, ANSI disabled, Adaptive Query Execution (AQE) enabled." } ], "markDefs": [], "style": "normal" } ] } ] } ] } ``` ### 2.2. Storage Profiles - **NeuralMesh / wekafs:** Queries read external Parquet tables from the NeuralMesh filesystem, mounted locally as a POSIX filesystem on every Spark worker, accessed by Spark via the File connector. - **Amazon S3:** Queries read the identical TPC-DS dataset hosted in an Amazon S3 bucket, accessed by Spark via the S3A connector. ### 2.3. Data Layouts (Variable Parameter) The key variable in this study is the Parquet row group size, which effectively controls the data chunk size. - **Baseline:** Unlimited (1 parquet file per table) - **Smaller Chunk Experiments:** The dataset generation process is repeated at reduced target chunk sizes (e.g., 128MiB, 32MiB, 8MiB, 1MiB) to analyze the performance inflection point as file sizes shrink. ### 2.4. Test Execution Flow - **Deployment:** Provision the NeuralMesh cluster and Spark cluster. - **Data Generation:** Generate the TPC-DS Parquet datasets for each target chunk size on both WEKAFS and Amazon S3 storage backends. External tables are registered in the Spark metastore. - **Benchmark Run:** For each unique combination of storage backend and chunk size, execute all TPC-DS queries for 5 iterations. - **Output Collection and Analysis:** - Collect results produced by `spark-sql-perf`. - Convert results from json to per-run and per-query CSVs. - Aggregate and compute statistical measures (min/max/avg/stddev) per query and storage backend. - **Compute Deltas:** Compare NeuralMesh vs. Amazon S3 performance per query for each chunk size. ## Key Metrics to Report - **Per-Query Duration:** Minimum, maximum, average, and standard deviation (stddev) of execution time across 5 iterations. - **Suite-Level Totals:** Total wall-clock time required for each iteration and the aggregate total across all iterations. - **Stability:** Variation (stddev) across the 5 iterations for each backend and chunk size combination. - **Trends:** Highlight the trend of performance deltas as the Parquet chunk size decreases. ## Reproducibility Notes To ensure the integrity of the results, the following controls will be maintained: - Identical Spark configurations across all runs (e.g., shuffle partitions derived from cluster cores; broadcast disabled via `-1`). - Datasets are fully regenerated for each new chunk size to ensure no mixed row-group sizes. - The dataset used during a benchmark on wekafs for a given chunk size is then copied directly to an AmazonS3 bucket, and stored in Standard Class, so there are no variations in parquet count or content between comparative benchmarks. - Consistent query order and iteration count; No warm-up runs are performed. ## Results & Analysis: When File Size Shrinks, Performance Plummets It is tempting to assume that because data lakes store very large volumes of data, the largest and most readily available storage service is automatically the best choice. In practice, this assumption does not hold once file size and workload characteristics are taken into account. For large files (greater than approximately 128 MiB), differences in Spark performance across common storage backends exist but are relatively modest. However, as workloads introduce small to medium-sized files, processing efficiency begins to degrade for many Spark jobs. When file sizes shrink further, the performance impact becomes severe. ![Bar chart of NeuralMesh POSIX versus Amazon S3 average seconds for TPC-DS queries at SF1000 with 128 MiB Parquet files, taller is slower](https://cdn.sanity.io/images/ult5g8gw/production/40784741d45e0492e134adda7126a86e8af8648d-2048x966.png) Figure 1. Comparing WEKA NeuralMesh (POSIX) with AmazonS3 running Apache Spark TPC-DS Benchmark at SF1000 and 128 MiB Parquet file size (taller is slower) As seen in Figure 1., for large files (greater than approximately 128 MiB), differences in Spark performance across common storage backends exist but are relatively modest. However, as workloads introduce small to medium-sized files (32 MiB), like seen in Figure 2. on the next page, processing efficiency begins to degrade for many TPC-DS queries. ![Bar chart of NeuralMesh POSIX versus Amazon S3 average seconds for TPC-DS queries at SF1000 with 32 MiB Parquet files, taller is slower](https://cdn.sanity.io/images/ult5g8gw/production/0ab4a15da92ecd37991134b2ddf27ea839644941-2048x969.png) Figure 2. Comparing WEKA NeuralMesh (POSIX) with AmazonS3 running Apache Spark TPC-DS Benchmark at SF1000 and 32 MiB Parquet file size (taller is slower) As seen in Figure 3., when Parquet file sizes shrink further to 16MiB, the performance impact becomes much more severe. ![Bar chart of NeuralMesh POSIX versus Amazon S3 average seconds for TPC-DS queries at SF1000 with 16 MiB Parquet files, taller is slower](https://cdn.sanity.io/images/ult5g8gw/production/d3d58eff13f5bf1baf419d8b35712b11d5b7f2fd-2048x969.png) Figure 3. Comparing WEKA NeuralMesh (POSIX) with AmazonS3 running Apache Spark TPC-DS Benchmark at SF1000 and 16 MiB Parquet file size (taller is slower) Figure 4 compiles all test results into a single view, where it’s clear where smaller chunk sizes impact performance for file sizes below 16 MiB. NeuralMesh remains unphased. ![Combined bar chart of NeuralMesh POSIX versus Amazon S3 across Parquet file sizes from 1 MiB to 128 MiB at TPC-DS SF1000](https://cdn.sanity.io/images/ult5g8gw/production/6f64b349901959d934abef34892a2822baf274ed-2048x1073.png) Figure 4. Comparing WEKA NeuralMesh (POSIX) with AmazonS3 running Apache Spark TPC-DS Benchmark at SF1000 comparing Parquet file sizes from 1MiB to 128 MiB ## Results and Analysis: A Single Query, Three Hours vs. 45 Seconds Consider TPC-DS Query 9 at Scale Factor 1000, which performs a conditional aggregation for reporting. In this experiment, the dataset is partitioned into Parquet files of 1 MiB each. Under these conditions, Spark must issue thousands of object metadata and data retrieval requests. This access pattern is fundamentally misaligned with the design goals of Amazon S3, which is optimized for throughput over large objects rather than low-latency access to massive numbers of small objects. Attempting to mitigate this limitation by migrating to S3 Express One Zone is economically impractical: approximately 35,000 monthly 1 MiB PUT and GET operations could result in costs exceeding one million dollars per month. By contrast, the same dataset evaluated on a WEKA NeuralMesh cluster—whether deployed on-premises, on a hyperscaler, or within a neocloud environment—demonstrates dramatically different behavior, and cost. For TPC-DS Query 9 at Scale Factor 1000 in 1MiB Parquet chunks, Spark wall-clock execution time is approximately 45 seconds on NeuralMesh, compared to more than three hours on Amazon S3 Standard for a single query. ## Conclusion: Storage Is the New Performance Layer When aggregated across all TPC-DS queries, NeuralMesh experiences roughly a mere twofold slowdown when moving from 128 MiB to 1 MiB files, whereas Amazon S3 slows by more than sixty times under the same conditions. Despite this, many data scientists are forced to endure such inefficiencies, repeatedly optimizing file layouts, consolidating partitions, and repartitioning data in the hope that analytics workloads will eventually complete within acceptable time bounds. With NeuralMesh’s ability to deliver OneZone level performance metrics at costs closer to S3 Standard, and up to Exabyte scale and beyond, the conclusions are clear. When one considers the problems Apache Spark set out to initially solve (file-based computation using Hadoop MapReduce) with the introduction of in-memory computation, no scalable storage architecture yet existed that could make any impact in comparison to Spark DRAM and local disk caching approach. Fast forward to today, and we see that with the advent of NVMe, high throughput, low latency networking, and the advanced architecture of NeuralMesh, I/O once again becomes the battlefield. ```json { "_key": "3159b6ef98f2", "_type": "textCallout", "content": [ { "_key": "rp00182", "_type": "block", "children": [ { "_key": "rp00178", "_type": "span", "marks": [], "text": "Dig further into NeuralMesh by checking out our Reference Architecture. If you are interested in discussing how NeuralMesh can help you accelerate Spark workloads, click here to contact us." } ], "markDefs": [], "style": "normal" } ], "markDefs": null, "title": "Take the Next Step with NeuralMesh" } ``` [Download PDF](/api/resource-pdf?slug=why-storage-matters-for-spark-jobs-a-tpc-ds-performance-analysis)