Machine Learning Meets the Internet of Things: Challenges and Opportunities
David Hiatt. November 21, 2018
The Internet of Things era is just beginning — and businesses are only now starting to see the payoff from the IoT efforts, and to see the value that IoT offers in terms of gathering information and making smarter, faster decisions.
Traditional business intelligence (BI) from complex data warehouses informed the past generation of savvy CEOs and COOs. Today, machine learning (ML) using IoT data will boost those businesses with the vision, and computing resources, to take this big leap. Indeed, ML and IoT can be thought of as powering the next generation of BI… and the next generation of CEOs and COOs.
Let’s see how it works with two examples.
Machine learning can produce significant business benefits
Automotive computer scientists build models to understand how self-driving vehicles operate, how people behave — and how the world works, too. Much of this depends on telemetry from the cars, and from external monitoring systems.
According to one estimate, a self-driving car’s sensors generate 150GB per hour of telemetry from radar, sonar, lidar, and cameras – that’s 4TB per day. ML uses this data to build models – and then test the decisions made by those models to see if they are safer and more effective than previous models. The faster those models can be built and tested, the more agile and competitive the carmaker is in evolving its algorithms.
Hospital medical equipment generates vast quantities of data, which is modeled and analyzed to help a doctor perform a diagnosis and determine a course of treatment. Data is generated from electrocardiograms, X-rays, CT scans, MRI, ultrasound, blood sugar monitors. And more.
That data must be captured and fed into machine-learning systems, to be matched against patients’ symptoms, treatments, and outcomes – as always, looking for anomalies and making predictions. IoT and machine learning are a huge opportunity and a massive data-processing challenge, in areas as diverse as oil-and-gas exploration, aircraft design, weather predictions, agriculture, battlefield simulations, pharmaceutical drug design, supply-chain management. and logistics.
The cloud levels the playing field
Historically, organizations performed ML analysis using huge high-performance computing (HPC) server clusters, fed by high-speed, directly attached storage. These clusters, with dozens or hundreds of servers, are expensive, and complex to own and manage. Only big organizations can afford big HPC clusters – and thus enjoy a significant advantage over smaller institutions and startups.
Cloud computing, such as using Amazon Web Services (AWS), changes everything. HPC capabilities can be accessed and used online – as much as needed, for as long as required, and only when needed. What’s more, cloud HPC services can be provisioned and configured in minutes or hours… without any real-estate footprint. This gives the smaller players the same tools as the bigger companies.
The cloud is also an efficient, and affordable, on-ramp to machine learning and IoT processing for smaller companies and startups, who can always choose to move ML and IoT servers into their own data centers later, if their usage grows large enough.
There’s a challenge, though: It’s hard to handle the IoT data properly. Telemetry and feeds – whether it’s numbers from an engine compression sensor, video from a laparoscopic camera, or high-resolution X-rays – are not like the traditional sales and transaction data that enterprises store in their data warehouse.
Without understanding this data, where it comes from, how to store it, and how to feed it quickly into a machine-learning HPC cluster on-premises or in the cloud, systems can choke on the ingest – seeing low throughputs, data errors, data loss, slow ML training, and delays before realizing the benefits of AI algorithms, data models, predictions, anomaly detection, and deep analysis.
The cloud can be an effective ML and IoT platform for smaller businesses, for startups, and for those looking to level the playing field. See how a small autonomous-vehicle startup boosted its machine learning capabilities to match the big companies using Amazon Web Services, or contact us at Weki.io.
AI AND HIGH VELOCITY ANALYTICS NEED A NEW FILE SYSTEM
Lectus arcu bibendum at varius vel pharetra vel. In cursus turpis massa tincidunt.