Out-Compute Your Competition: Accelerating Algorithmic Trading Model Execution

Barbara Murphy. October 17, 2019
Out-Compute Your Competition: Accelerating Algorithmic Trading Model Execution

Barbara Murphy, Vice President of Marketing at WekaIO, shares her insights in this blog titled “Out-Compute Your Competition: Accelerating Algorithmic Trading Model Execution.”

Success in algorithmic trading is gained by the speed and accuracy of predictive financial models. Quantitative trading organizations seek a competitive advantage through increasing data quantity and quality and improving algorithm development, all of which require data storage infrastructure enhancements to accelerate model execution.

We know in today’s digital world that organizations of all kinds are facing an avalanche of data growth, and the financial services industry is no exception. This creates both management and performance headaches for legacy storage systems. These challenges include limited single namespace scalability, spinning disk limitations with an ever-declining performance ‘throughput-to-TB’ ratio, and a massive performance slowdown when file systems reach 80%+ capacity. When the success of your organization depends on the speed and accuracy of data processing, these exposures are very troubling.

Improving algorithm development can be achieved in many ways, from increasing dev teams to implementing modern CICD (continuous integration, continuous deployment) pipelines and the introduction of machine learning, to name a few. Building cloud computing environments to support modern CICD pipelines requires a high-performance shared file system to serve data sets quickly to the processing instances. In addition, machine learning introduces training and inference steps to the data pipeline. This then requires not only high throughput to a single compute client but also low latency to feed data quickly to GPUs and the ability to handle random, mixed file size workloads – attributes that are not associated with traditional file systems.

The Challenge
To summarize so far, the domain of algorithmic trading incorporates use cases that drive huge amounts of data and require scale-out, high-performance flash storage, and therein lies the rub. While the price of flash has come down dramatically, it is still at a premium compared to that of spinning disk, so most organizations do not want to store huge amounts of data on flash storage…but they NEED the performance density of flash!

How WekaIO Can Help
The Weka File System, WekaFS™, is a highly resilient parallel file system that delivers the highest bandwidth, lowest latency performance to any InfiniBand or Ethernet enabled GPU or CPU based cluster. Built for enterprise-grade analytics, it tightly integrates object storage for best economics at scale.

WekaFS Performance
WekaFS improves the performance of market trading applications as illustrated in recent results with the STAC M3 benchmark suite (read more here). WekaFS set 8 new world records with STAC M3, beating alternative shared file systems such as Lustre in 15 of 17 and NetApp in 12 of 17 benchmarks.

WekaFS performance is further illustrated by the SpecSFS 2014 benchmark for software builds (read more here), achieving one-third of the latency and 249% more builds per client than NetApp, the closest competitor.

And for those machine learning data pipelines, WekaFS can saturate 100Gb connections to a single or multiple GPU-based clients, ensuring the GPUs are fed with data at super low latency. Chart A below shows the linear scalability of WekaFS in the number of images per second that it can process across 9 Nvidia DGX-1 clients, each of which has 8 Nvidia Volta GPUs. Chart B shows the throughput performance of WekaFS to a single GPU client at 11GB/sec versus 1.2GB/sec with NFS and 3GB/sec with internal storage.


Chart A


Chart B

Financial trading is possibly the most competitive market in the world. An organization must out-compute rivals in order to win against them! The speed of data processing and the ability to feed processing instances quickly with data are critical to success. The above benchmarks prove that WekaFS is a unique file system with block-storage like latencies that accelerates time to results for algorithmic trading.

Not Just Performance, But Better Economics
WekaFS™ can be configured either as a flash-only system or as a tiered data system consisting of both flash and object stores with spinning disk. By nature, flash provides high performance density and low latency storage, while object stores are the most cost-effective and scalable solution for storage. Users seeking to balance between performance and cost should consider a tiered data system. With WekaFS, tiering is a completely seamless, automatic and transparent process, with users and applications unaware of the transfer of data from flash to object stores, or from object stores to flash. The data is accessible at all times through a shared namespace, regardless of where it is stored. This flexible tiering approach provides high-performance access for users and applications, combined with the economics, scalability, and resiliency of object storage with spinning disk.

In summary, your success is predicated on your ability to process data faster and more accurately. WekaFS can feed your compute instances with data faster than alternative solutions to give you a competitive edge. Furthermore, WekaFS tiers data automatically to object storage for great economics, scalability, and resiliency. This uniquely provides the ideal blend of performance and economics for ever-growing data volumes.

Learn more about Algorithmic Trading and Financial Services Solutions with WekaIO here.

Related Resources

Solution Brief
Solution Brief



What is the Advanced Technology Center and How You can Access it

Watch Now
Case Studies
Case Studies

Oklahoma Medical Research Foundation (OMRF) Case Study