# From Inference to Agentic AI

Driving insights, efficiency and improved economic outcomes with purpose-built inference infrastructure.

**Author:** Greg Macatee

**Published:** November 16, 2026

**Type:** Analyst Report

**Publisher:** 451 Research by S&P Global

## Executive summary

We are witnessing profound shifts in the composition of workloads running on AI infrastructure. Just a few years ago, this infrastructure was almost exclusively used to build and train foundational models. It has since evolved, allocated not only to training models but also to supporting other model life cycle workloads, particularly inference. Inference applies the results of model training to previously unseen inputs to generate insights, predictions or other outputs (e.g., text, code, audio and video). It is where the value of AI will ultimately be realized.

Inference workloads are driving substantial demand for AI infrastructure. 451 Research projects that by 2030, inference workloads will drive $746 billion, or 46% of AI infrastructure spending, up from $225 billion, or 34% of spending, in 2026 (see Figure 1).


![Projected AI infrastructure spending by workload type](https://cdn.sanity.io/images/ult5g8gw/production/80b27451bae8cea8411cc5adcc06d4e6aa2a29c5-1665x894.png)


The inference needs of humans and machines, including agents, underpin this demand. In North America alone, we forecast the total number of tokens produced between 2026 and 2028 will increase nearly tenfold, rising from 24 quadrillion to 206 quadrillion. During this period, the share of tokens generated by AI agents is projected to climb from 32% to 72%.

This explosion in token generation places an unprecedented strain on infrastructure and data center resources. Endlessly provisioning compute capacity is becoming less feasible as decision-makers face the compounding friction of hardware scarcity, memory limits and data center constraints such as power, cooling and floor space.

Instead, buyers are deploying purpose-built inference infrastructure equipped with AI data infrastructure platforms to bridge these technical gaps. To overcome system memory bottlenecks, AI data infrastructure platforms incorporate a shared memory and context tier that secures and governs "always on" inference operations while delivering improved token economics. Buyer adoption of these platforms has rapidly expanded as organizations scale operations, grow their user bases and deploy agents: Nearly four in 10 decision-makers now purchase an AI data infrastructure platform as a distinct component for their modern inference stacks.

### Key findings

- The AI workload mix is shifting from predominantly training to a combination of training, inference and other model life cycle tasks, with inference expected to dominate by 2030. This massive growth in inference directly powers executive-level business goals since running models in production is what delivers innovative products, user experiences and measurable financial return on investment (ROI).
- Inference workloads carry unique demands, such as long context windows, repeated turns, high concurrency and strict latency targets, that many existing architectures cannot meet while remaining financially feasible. This is a significant headwind for decision-makers. In response, organizations are seeking credible alternatives that improve inference metrics in a secure and governed manner, leading to stronger outcomes and higher ROI.
- Nearly four in 10 AI infrastructure decision-makers (39%) purchase a dedicated AI data infrastructure platform as a distinct component for their inference stacks. These platforms often incorporate a shared memory and context tier that overcomes system memory bottlenecks while supporting secure, governed and economical inference at scale.
- Organizations expect AI agents to amplify architectural challenges in their inference infrastructure. Agentic workload demands will place unprecedented pressure on data center and AI infrastructure resources across the technology stack, exacerbating ongoing challenges in inference data management and memory.
- During inference, poor data performance and memory limitations bottleneck AI infrastructure, hampering token yields and workload outcomes. Accelerator and memory shortages, alongside data center constraints such as power costs and availability, further compound these challenges.

## Inference drives demand for specialized AI infrastructure

AI workloads throughout the model life cycle impose vastly different requirements on infrastructure across compute, storage and networking. Each of these components must be carefully architected and stitched together to optimize efficiency and increase infrastructure yields. Requirements have remained relatively consistent over the past several years as organizations have built and scaled AI factories. However, as inference needs continue to grow, so have specific infrastructure requirements for inferencing that have evolved in tandem.

Buyer requirements are shifting to meet these workload needs. Our research indicates that decision-makers are increasingly purchasing distinct AI infrastructure components to power dedicated AI workloads. Among organizations that deploy inference workloads in production, 41% buy fundamentally different hardware and software from the rest of their IT infrastructure. For organizations that do not deploy inference workloads, fewer than one in three have separate buying patterns. This gap highlights that as organizations mature and grow their capabilities to serve inference, they increasingly require specific AI infrastructure components to realize value at scale (see Figure 2).


![AI infrastructure components that organizations purchase differently than general IT](https://cdn.sanity.io/images/ult5g8gw/production/47eaefd9d20489599e5c3dce38b102cb3eb8c954-1665x954.png)


## Purpose-built inference infrastructure continues to adopt specialized AI data platforms

Why do decision-makers follow different buying patterns when supporting inference workloads? Because serving inference imposes a starkly different set of requirements on infrastructure than other tasks such as training. Model training is largely a predictable process in which users define and build models with predetermined sizes and configurations. During the training process, checkpointing — a critical process whereby sets of pretrained model weights are saved and persisted — occurs at regular and predictable intervals. This process of calculating, evaluating and checkpointing weights repeats until a model is fully trained and ready for deployment. From an operations perspective, infrastructure teams can generally anticipate these stable training requirements, enabling them to plan and architect their systems accordingly.

Unlike the predictable nature of model training, inference poses an entirely new set of challenges regarding data performance and movement, memory, storage, security and privacy. Rather than constantly feeding raw data to GPUs, inference requires retaining and reusing active session context so accelerators can perform useful work instead of unnecessarily burning cycles recomputing tokens. Prevailing architectures often struggle to meet this demand at scale due to a fundamental gap in the memory and storage hierarchy. High-bandwidth GPU memory (HBM) is fast but has limited capacity and is highly expensive; server memory (DRAM) is also costly, lost on restart and cannot be shared between servers; and traditional enterprise storage, while more economical, is too slow.

Suboptimal key-value (KV) cache management compounds these challenges. When inference is served, model weights and KV cache must share scarce HBM, with models occupying a large up-front footprint. As users build up context (both input prompts and generated outputs) across sessions, the growing KV cache exhausts available HBM and spills into DRAM until capacity limits force cache evictions. This forces GPUs to recompute tokens.

While memory pressure may trigger out-of-memory errors and failed workflows, requests often do not fail; instead, they complete after GPUs spend time rebuilding the context they have already processed. Because GPUs remain busy and appear fully utilized while recomputing, the inefficiency is often hidden, quietly driving up latency and compute costs. Many organizations attempt to compensate by buying more GPUs or DRAM, yet this strategy is both costly and supply-constrained. Adding memory alone is also architecturally limited because context remains isolated to individual nodes and cannot be shared across a cluster.

As a result, organizations are increasingly turning to AI data infrastructure platforms to help mitigate these limitations. These solutions combine specialized storage and data management features to handle extreme scale, high throughput and low-latency workload demand. This enables the allocation of GPUs to perform new, productive work rather than rebuilding context. Many AI data infrastructure platforms also introduce a shared memory and context tier.

By persisting KV cache across GPU servers and between sessions, this architecture helps eliminate token recalculations and provides the massive capacity needed for multi-turn agentic workflows. Additionally, because persisted cache holds sensitive information drawn from user prompts, enterprise AI data infrastructure platforms also protect the shared tier with built-in isolation, encryption, access controls and deletion to safeguard data privacy and governance.

With these capabilities in place, buyer behavior is shifting rapidly. As illustrated in Figure 2 above, 39% of AI infrastructure decision-makers identify an AI data infrastructure platform as a distinct technology component purchased separately from the rest of their IT stack. This demonstrates remarkable progress for an emerging technology category that 451 Research Voice of the Enterprise (VotE) survey respondents did not even identify as a separate resource in our year-ago survey.

Demand for purpose-built inference infrastructure is driven by performance, reliability and economics. As the market gravitates toward highly optimized rack-scale systems and reference designs such as NVIDIA's CMX, organizations are continually working to maximize their infrastructure yields. AI data infrastructure platforms enable users to improve benchmarks such as time to first token, inter-token latency, KV cache hit rates and overall request throughput. These platforms also help maintain outcomes under heavy, sustained loads and at tail latencies (e.g., P95 and P99), rather than solely at the average.

While end users perceive these improvements as a faster, more consistent application experience, the impact for decision-makers aggregates at a higher level. Incorporating AI data infrastructure platforms into purpose-built inference infrastructure helps eliminate redundant compute while supporting higher multitenant concurrency, translating to more tokens per GPU-hour, reduced cost per token and increased revenue per dollar of infrastructure spending within a given service-level agreement (SLA).

## Agentic AI amplifies data and infrastructure challenges across the inference stack

Following the initial surge of generative AI experimentation, organizations have struggled to move projects from pilot phases into production. Scaling these initiatives has proven particularly difficult as foundational infrastructure, tools and operational frameworks buckle under the pressure of inference and other AI workloads. This strain is reflected in survey respondents' top AI infrastructure concerns over the next 12 months, which are led by security, privacy and governance (36%), rising infrastructure costs (22%), keeping pace with innovation (21%), promoting digital sovereignty (18%) and meeting regulatory and compliance requirements (17%).

However, a distinct new challenge is emerging alongside these more familiar hurdles: 19% of organizations highlight the impact of agentic AI on their infrastructure and data centers. These worries are well-founded, as agentic AI threatens to amplify existing pressure points in infrastructure while posing novel challenges of its own (see Figure 3).


![Agentic AI presents new infrastructure challenges while deepening existing ones](https://cdn.sanity.io/images/ult5g8gw/production/3a3161b947049e69cd65882d79418f319575c0fb-1665x1182.png)


In addition to these infrastructure concerns, the rising demand for agentic systems highlights the urgent need for credible solutions. Among organizations that deploy AI, 49% are already running agents in production, 31% are in testing or proof-of-concept stages, and 12% intend to begin working with agents over the next 12 months, according to 451 Research's VotE: AI and Machine Learning, Infrastructure 2026 study. Among the top expectations for these agentic systems, 47% of organizations cite long-term memory, 44% prioritize self-improvement and 43% highlight explainability, according to the VotE: AI & ML, Agentic AI 2025 survey. These capabilities depend on an agent's ability to reference trails of generated context. In many cases, AI data infrastructure platforms can help meet these needs by providing a shared memory and context tier.

By offloading KV cache to a shared tier, organizations can address memory challenges and reduce their physical infrastructure footprints, avoiding the need to overprovision GPUs and memory to support the scale of agentic AI. Simultaneously, enterprise-ready AI data infrastructure platforms pair a unified namespace with strict security and life cycle controls, including tenant isolation, end-to-end encryption, granular permissions and policy-driven residency, retention and deletion, which can help buyers meet regulatory and privacy standards. Ultimately, these capabilities help control infrastructure costs while providing the persistent context that agents need in production.

## Purpose-built inference infrastructure aligns data center realities with executive-level AI ambitions

The success of an AI initiative is increasingly dictated by the infrastructure that supports it. AI infrastructure strategy has become increasingly important in boardroom-level discussions, moving beyond traditional IT planning to a core determinant of overall business execution. Executive leaders are now keenly aware that deploying models is only a fraction of the challenge. They must contend with the physical realities of surrounding data center environments, including power and cooling requirements, floor space limitations and ongoing supply chain shortages of essential hardware.

These physical and logistical constraints are actively reshaping technology roadmaps (see Figure 4). More than half of AI infrastructure decision-makers cite power costs and availability and supply chain shortages of memory and flash storage (solid-state drives) as having a medium, high or very high impact on their plans. The fallout from these bottlenecks is tangible: Organizations are being forced to delay projects, adjust budgets or, in severe cases, cancel AI initiatives outright because the underlying infrastructure cannot be reliably sourced or powered.


![Power and supply chain shortages disrupt AI roadmaps and outcomes](https://cdn.sanity.io/images/ult5g8gw/production/6d86b3081817a8f83eb478fab21fd9fea7bbf2bf-1665x1032.png)


To navigate these shortages and control spiraling power costs, organizations must find ways to do more with less. This is where AI data infrastructure platforms provide a compelling advantage. Deploying these solutions expands shared cache capacity, which can help improve KV cache hit rates and reduce redundant recompute. This makes inference more economical to serve.

For C-suites, AI data infrastructure platforms go beyond saving compute cycles; they directly enhance business agility. By maintaining a persistent thread of information across sessions, they equip human users and autonomous agents with the context needed to perform complex reasoning and make faster, more insightful decisions.

At the same time, this architectural optimization shrinks the overall physical infrastructure footprint required to support inference workloads, mitigating the need to continuously buy scarce GPUs and memory. This leaner footprint creates a positive spillover effect: It reduces adjacent demand for data center power, cooling and floor space. This allows organizations to scale their AI ambitions faster and more economically within constrained supply chains.

## Conclusions

As organizations move beyond generative AI pilots toward production inference and autonomous agents, relying on raw compute power alone is proving unsustainable in the face of memory limits, hardware shortages and power constraints. To scale multi-turn, long-context workflows and agents effectively, organizations will increasingly be pressed to purposefully architect inference stacks to meet their workload demands. When doing so, decision-makers should evaluate whether incorporating a dedicated AI data infrastructure platform can help address workload constraints by asking:

- What share of prefill compute is spent repeating prior context?
- Does inference latency — especially for the slowest requests (P95 and P99) and not just the average — hold steady under heavy, continuous production?
- What is the cost per million tokens at a target SLA? What is the cost per successful task?
- Who owns the architectural and governance decisions across the organization's AI platform, infrastructure and security teams?

As buyers increasingly treat context memory as a distinct tier of inference infrastructure, AI data infrastructure platforms are quickly becoming a foundational component for scaling and maximizing the ROI of AI investments.

Visit weka.io (https://www.weka.io) to learn how WEKA NeuralMesh and Augmented Memory Grid help organizations scale inference.

## About the author

**Greg Macatee - **Senior Analyst

Greg Macatee is a senior analyst leading coverage of AI infrastructure within the Data, AI & Analytics channel of 451 Research by S&P Global. In his research, Greg seeks to identify and highlight the impacts of current and future enterprise AI and machine learning technology trends on the IT infrastructure that underpins them. His areas of expertise include AI infrastructure (compute, storage and networking), software-defined compute (virtualization and containers), storage software, software-defined storage and data protection; and cloud-native and open-source technologies.

Greg joined S&P Global in January 2025 with nearly a decade of experience working in the analyst community. Over that time, he has covered a wide variety of topics ranging from enterprise mobility, smartphones and digital signage to enterprise infrastructure with dedicated interest and focus on cloud, compute, storage and other infrastructure software technologies.

He is frequently quoted and recognized in the press and is a regular speaker at vendor and other technology industry events. Greg currently resides in the Boston area. He holds BA degrees in German Language and Literature and Economics and a master's degree in Economics from Boston University.

### About this paper

A Pathfinder paper navigates decision-makers through the issues surrounding a specific technology or business case, explores the business value of adoption, and recommends the range of considerations and concrete next steps in the decision-making process.

### About 451 Research

451 Research by S&P Global provides essential insight into key trends driving digital transformation across the entire technology landscape. By offering a combination of expert analyst insight and differentiated data, 451 Research provides client organizations with the critical information and perspectives they require to make more effective business decisions.

451 Research offerings are organized around the prevailing themes driving digital transformation today, including data centers and energy, AI, cybersecurity, digital industries, and resilient digital infrastructure.

For more information about 451 Research, please go to:

 [spglobal.com/451research](https://www.spglobal.com/451research).

### Contacts

[www.spglobal.com](https://www.spglobal.com)

[www.spglobal.com/en/enterprise/about/contact-us.html](https://www.spglobal.com/en/enterprise/about/contact-us.html)

### Legal notice

Copyright © 2026 S&P Global Inc. All rights reserved.

These materials, including any software, data, processing technology, index data, ratings, credit-related analysis, research, model, software or other application or output described herein, or any part thereof (collectively the "Property") constitute the proprietary and confidential information of S&P Global Inc its affiliates (each and together "S&P Global") and/or its third party provider licensors. S&P Global on behalf of itself and its third-party licensors reserves all rights in and to the Property. These materials have been prepared solely for information purposes based upon information generally available to the public and from sources believed to be reliable.

Any copying, reproduction, reverse-engineering, modification, distribution, transmission or disclosure of the Property, in any form or by any means, is strictly prohibited without the prior written consent of S&P Global. The Property shall not be used for any unauthorized or unlawful purposes. S&P Global's opinions, statements, estimates, projections, quotes and credit-related and other analyses are statements of opinion as of the date they are expressed and not statements of fact or recommendations to purchase, hold, or sell any securities or to make any investment decisions, and do not address the suitability of any security, and there is no obligation on S&P Global to update the foregoing or any other element of the Property. S&P Global may provide index data. Direct investment in an index is not possible. Exposure to an asset class represented by an index is available through investable instruments based on that index. The Property and its composition and content are subject to change without notice.

THE PROPERTY IS PROVIDED ON AN "AS IS" BASIS. NEITHER S&P GLOBAL NOR ANY THIRD PARTY PROVIDERS (TOGETHER, "S&P GLOBAL PARTIES") MAKE ANY WARRANTY, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, FREEDOM FROM BUGS, SOFTWARE ERRORS OR DEFECTS, THAT THE PROPERTY'S FUNCTIONING WILL BE UNINTERRUPTED OR THAT THE PROPERTY WILL OPERATE IN ANY SOFTWARE OR HARDWARE CONFIGURATION, NOR ANY WARRANTIES, EXPRESS OR IMPLIED, AS TO ITS ACCURACY, AVAILABILITY, COMPLETENESS OR TIMELINESS, OR TO THE RESULTS TO BE OBTAINED FROM THE USE OF THE PROPERTY. S&P GLOBAL PARTIES SHALL NOT IN ANY WAY BE LIABLE TO ANY RECIPIENT FOR ANY INACCURACIES, ERRORS OR OMISSIONS REGARDLESS OF THE CAUSE. Without limiting the foregoing, S&P Global Parties shall have no liability whatsoever to any recipient, whether in contract, in tort (including negligence), under warranty, under statute or otherwise, in respect of any loss or damage suffered by any recipient as a result of or in connection with the Property, or any course of action determined, by it or any third party, whether or not based on or relating to the Property. In no event shall S&P Global be liable to any party for any direct, indirect, incidental, exemplary, compensatory, punitive, special or consequential damages, costs, expenses, legal fees or losses (including without limitation lost income or lost profits and opportunity costs or losses caused by negligence) in connection with any use of the Property even if advised of the possibility of such damages. The Property should not be relied on and is not a substitute for the skill, judgment and experience of the user, its management, employees, advisors and/or clients when making investment and other business decisions.

The S&P Global logo is a registered trademark of S&P Global, and the trademarks of S&P Global used within this document or materials are protected by international laws. Any other names may be trademarks of their respective owners.

The inclusion of a link to an external website by S&P Global should not be understood to be an endorsement of that website or the website's owners (or their products/services). S&P Global is not responsible for either the content or output of external websites. S&P Global keeps certain activities of its divisions separate from each other in order to preserve the independence and objectivity of their respective activities. As a result, certain divisions of S&P Global may have information that is not available to other S&P Global divisions. S&P Global has established policies and procedures to maintain the confidentiality of certain nonpublic information received in connection with each analytical process. S&P Global may receive compensation for its ratings and certain analyses, normally from issuers or underwriters of securities or from obligors. S&P Global reserves the right to disseminate its opinions and analyses. S&P Global Ratings' public ratings and analyses are made available on its sites, www.spglobal.com/ratings (https://www.spglobal.com/ratings) (free of charge) and www.capitaliq.com (https://www.capitaliq.com) (subscription), and may be distributed through other means, including via S&P Global publications and third party redistributors.

Content may be created with the assistance of an AI tool in accordance with our Terms of Use.

[Download PDF](/api/resource-pdf?slug=inference-451-research-pathfinder-report)

## Related Content

- [Benchmarking WEKA's Augmented Memory Grid on Nebius AI Cloud](/article/benchmarking-weka-on-nebius-ai-cloud)
- [The Impact of Storage on the AI Lifecycle](/resources/analyst-report/the-impact-of-storage-on-the-ai-lifecycle)
