Inference at Scale – Storage as the New AI Battleground

TL;DR Life sciences breakthroughs are increasingly bottlenecked not by scientific capability, but by data infrastructure. Researchers need instant access to exabytes of “hot data” that’s increasingly scattered across distributed cloud environments and on-premises systems. That’s why WEKA, Guardant Health, and a consortium of other technology vendors are building the Single Namespace Standard (SNS): an open standard that unifies distributed data access, eliminates vendor lock-in, and could be as foundational for bioscience as HTTP was for the internet.

“If the 20th century was the century of physics, the 21st century will be the century of cybernetics, biology and ecology.”

Pioneering genomics researcher Craig Venter and world-renowned geneticist Daniel Cohen made this statement in a joint research paper they led in 2004. They noted that while technological advancement at the time had opened up new frontiers for innovation, it also exposed the next hurdles to overcome when fundamental technology underpinnings can’t keep pace with the innovation itself. Fast-forward to 2025, and it feels remarkably prescient.

At WEKA, we’re developing next-generation AI infrastructure solutions that help our customers stay on the cutting edge of their own industries. That’s why the work WEKA has been doing with Guardant Health is so cool. In the fight against cancer, Guardant and its peers have enlisted a cross-industry consortium of 34 vendors to support their cause, collaborating to develop the Single Namespace Standard (SNS): a new global standard for managing exabyte-scale distributed data for the massive data and performance requirements of 21st-century life sciences research and development.

SNS enables organizations to access all their data—across on-premises and cloud environments, even across vendors—through a single, unified interface, no matter where it’s actually stored. SNS will help researchers and clinicians apply data and discoveries much faster across research, clinical validation, and patient care. It removes a critical roadblock to fast and efficient research when the size of the datasets is incomprehensibly large.

Today’s Reality: Deep Fragmentation, Hidden Cost and Locked-in Silos

In large object-store environments, organizations often try to work around vendor lock-in by building on top of proprietary formats or cloud APIs (for example, treating Amazon S3 as a de-facto standard). But these workarounds bring significant drawbacks: the file/data format may only be readable by that vendor’s stack, egress or migration costs can balloon, and switching to a different storage or compute vendor becomes prohibitively complex.

Moreover, hybrid and multi-cloud strategies often succeed in theory but falter in practice because each vendor’s object-store APIs, metadata semantics, access paradigms and governance models differ, so datasets become “stranded” even when migrated. The result: Organizations end up locked into one vendor’s ecosystem, choice of tools becomes constrained, and the agility required for exabyte-scale life-science workflows is severely impaired.

A New Approach Requires New Standards

One impetus for solving this problem emerged when Guardant found that more than 80% of its data used for life sciences research qualified as “hot data”— data that is frequently accessed —and that archiving it to tape became impractical. This is a departure from traditional data management models, where hot data simply meant data that was recently created or where access patterns affected only a relatively small portion of the data estate. This was a surprising figure, but it makes sense: Researchers are tasked with developing new therapies to target specific proteins or other biological components, and new experiments benefit from larger samples. For institutions with exascale datasets spanning several decades, inoculating infrastructure against complexity while ensuring the best possible performance is essential to supporting research computing environments. And that’s just one use case.

By defining an open, vendor-neutral object namespace standard — one that treats object storage as a universal fabric rather than a closed ecosystem — the consortium aims to restore portability of data, interoperability of solutions, and the freedom for customers to choose the best tools across vendors without rebuilding the data every time.

SNS has the potential to usher in a new era of R&D in the biological sciences by democratizing data movement to accelerate analysis and research. Depending on who you ask, it could be as important as the HTTP, TCP, and HTTPS protocols that enabled the rise of the internet, or as crucial as SMS, which made text messaging a ubiquitous way for people to communicate instantly.

My Three Favorite Things About SNS

The intersection of tech-meets-science, and especially biological sciences, is a particular passion of mine. I’m proud that WEKA is part of this era-defining work. Here are the three aspects of the SNS initiative I’ve found intriguing:

1. True Industry Collaboration – No Strings Attached

First, I’m impressed by the cross-industry effort to develop SNS. Guardant is leading the way, with founding members NetApp, Seagate, IBM, DDN, Genentech, and, of course, WEKA.

One thing about this list that might strike you as odd is that some of these companies compete with one another, which, in my view, is what makes this so promising. It’s so nice to see industry veterans set aside corporate agendas to solve a major industry pain point in the service of scientific advancement. Even though we might compete today, many in the cohort have known each other for decades, and there is beauty in being human together on such an important topic.

2. Customers Can Choose the Right Tools for the Job

Second, vendor lock-in is an innovation killer. Customer choice always wins, and open standards pave the way for customers to choose the right tool for the job at hand. No matter which solutions they choose, they’ll have the confidence of interoperability across different technologies, ensuring they maximize their investments with a best-of-breed approach.

From WEKA’s vantage point, this also means prospective customers can have a streamlined bake-off experience, since our technology can “plug and play” with other solutions they may already have in their environment. On that note, if you are interested in our solutions, let’s connect!

3. Major Breakthroughs In Research Efficiency

Finally, SNS represents a giant leap forward in data management, accelerating life sciences research. Regulatory data retention requirements in this field can be extensive – FDA-approved liquid biopsies alone have retention requirements of 30 years. Meanwhile, a researcher today may run data science experiments every two to three months on progressively larger datasets that require dramatically faster solutions. This ever-present need to access more data quickly and frequently makes a strong case for adopting open standards like SNS to streamline the process.

Life Sciences Today, Other Industries Tomorrow

Biological sciences are in the midst of a renaissance. Concepts developed for manufacturing and electronics design in the last century are now being applied as a force multiplier to accelerate the best industry for advancing positive health outcomes for everyone. Open standards can only make the industry healthier by removing the potential for data services gridlock. 

For as much as SNS seeks to achieve in the life sciences space, this standard could apply to many other industry verticals. Instant access to large volumes of data makes or breaks time-to-market. As AI adoption drives exponentially bigger data environments across all industries, open standards and the right foundational technologies are the key to keeping it all moving.

WEKA is excited to be working alongside Guardant Health and other SNS contributors to effect change in an industry that directly impacts the health of our society. If you’re as fascinated by the work being done in the world of life sciences as I am, I encourage you to check out the work WEKA is doing with other life sciences organizations! 

For more information about this initiative, you can read the full press release here.