From the Floor: Big and Little Data Engineering at ICDE 2017
David Hiatt. April 24, 2017
This week I had the privilege of attending the IEEE’s premier event on data management, the 33rd annual International Conference on Data Engineering (ICDE) hosted in San Diego. What a fantastic event. I spoke with members from all over the world who were addressing the most difficult issues related to cloud based data management, new database structures, query optimization and provenance – in other words, deep stuff. Artificial intelligence, machine learning and NVIDIA performance were especially popular topics.
Perhaps the most interesting topic I came across was data stream processing, something I’ve been thinking about for some time. It raises a host of new challenges because the data itself is changing in real-time. Not only are software programming practices affected, but also the underlying system architectures. For these reasons and others, modern scale-out systems rely on compute clusters of commodity servers rather than specialized hardware.
Data stream processing systems are well-suited to applications with pipelined computations and continuous data streams. Example use cases include patient monitoring systems, financial trading systems, online gaming, mobile device location based services, social media analytics, and self-driving cars. The raw data generated may need to be stored for several years for compliance or liability reasons.
The challenge for supporting storage systems is not ingesting hundreds or even thousands of data streams, it’s ingesting, storing, and serving hundreds of thousands of parallel data streams while providing near instantaneous response to other application requests. The idea of one storage system for all workloads seems far-fetched until you consider how far we’ve come with self-driving cars in just a short period. Like data stream processing, all that is required is thinking outside the box.
That’s all from the show floor.
If you’re in Boston next month, stop by and visit our booth at BioIT World. Life Sciences – now, that’s an industry with some serious data.
AI AND HIGH VELOCITY ANALYTICS NEED A NEW FILE SYSTEM
Lectus arcu bibendum at varius vel pharetra vel. In cursus turpis massa tincidunt.