AI Storage TCO & Token Economics: How Storage Determines AI Profitability

The token may be the smallest measurable unit of output for AI, but it has nevertheless become the largest determining factor for your AI profitability.

In fact, the cost of this humble unit has upended how organizations think of total cost of ownership (TCO) in the AI era. Whereas legacy data centers look to optimize metrics like the cost of floating-point operations per second (FLOPS) or the total cost of compute, AI economics depends on producing as many tokens as possible for as little as possible. This has set off a race to build the most effective infrastructure for delivering this.

But while many organizations focus on GPUs, networking, and power, the role of storage often gets overlooked. This is a mistake. Here’s how you should be thinking of AI storage TCO and why it’s the highest ROI investment in the stack.

Cost Per Token Is the New North Star

The reason cost per token became so important in the AI economics conversation is simple: you can have incredibly fast compute but still lack the ability to produce tokens efficiently. That’s because tokens depend on so much more besides compute, such as high bandwidth memory (HBM), system DRAM, and NVMe storage. If any of these factors slow down production, GPUs will sit idle and the cost per token will go up. And since tokens are what users pay for (and their  speed of delivery maps directly to the user experience), lowering the TCO of AI means also lowering cost per token.

This is an insight that’s been absorbed across the AI space. To see this, all you have to do is look at what’s happened to token costs over the past few years. Between 2022 and 2025, the cost per token has dropped 4x for frontier models and as much as 100x for mid-tier models. Driven largely by model efficiency improvements and competition among providers, this marks one of the most dramatic price drops in technology – and one that shows no signs of slowing down. By 2030, the cost per token is expected to be 90 percent less than its 2025 cost.

So why, despite this success, are the total costs of AI systems still rising – going from $11.5 billion in 2024 to $37 billion in 2025? The answer has to do with Jevon’s paradox, or the tendency for people to use a service more as costs go down. In the case of AI, although the average costs per token have lowered significantly in the past few years, the rate of consumption has outpaced these gains. With tokens now cheaper than ever, they are being put toward ever more complex models, many of which come with agentic capabilities that require continuous inference. The result? A 320 percent increase in inference spending at the same time there was a decrease in per-token costs.

Far from reducing the significance of cost per token, this situation has only increased its importance. The rising costs of inference demand ever greater gains in efficiency in order for AI systems to be profitable. Accomplishing this will require more than model tweaks or competitive pressures – it will require rethinking how storage is used alongside compute in the AI stack.

The Hidden Math — How Storage Determines GPU ROI

To understand the importance of storage to compute – and why optimizing this relationship is key to AI profitability – consider this fact: on average, only about 5 percent of GPUs are getting utilized at any moment within AI infrastructure.

In other words, organizations are paying for 95 percent of their compute to just sit there and do nothing. This represents one of the largest and most expensive sources of waste in data centers. How much waste? As of this writing, H100 chips cost about $2-$3 per GPU-hour, while B200 chips cost about $5-$8 per GPU-hour. Assuming they remain 95 percent idle over an entire year (8,760 hours), here’s what this could cost:

To help put these numbers into context, even at the lowest end of this table, the amount spent annually on idle GPUs is more than most organizations spend on their IT staff. This is more than significant – it’s transformational. So why is it happening?

While there can be many reasons, nearly all of them have to do with how data moves between GPUs and storage. Here are four of the most common scenarios:

• Checkpoint stalls: During training, a model periodically saves its current state to disk so it can resume if a job fails – a process called checkpointing. However, if the storage cannot take this data in fast enough, the GPU will have to stop and wait until the write is complete. Over a long training job, this can add up to hours of idle time.
• Data pipeline bottlenecks: When processing training batches, GPUs can consume data far faster than most storage systems can supply it. If the GPU finishes a batch before the next one is ready, then it will have to sit idle. This is sometimes also called “GPU starvation.”
• Metadata operations queuing: Metadata includes information such as where a file is located and how big it is. In legacy systems like NAS, this metadata flows through a single controller. A typical training workload can generate millions of these metadata requests per second, idling the GPU as it waits for these to process.
• Memory wall: During inference, the KV cache is kept in GPU HBM so that it can be accessed as quickly as possible. However, the limited size of HBM means that it must often evict data to slower tiers (DRAM, NVMe, and network storage) to make additional room. When that data is needed again, this slow speed may mean the GPU will stall.

This disconnect between GPU and storage may seem like an intractable problem, but it doesn’t have to be. In fact, the issue isn’t in the hardware itself but in how the AI system is architected. Software-defined solutions like WEKA® NeuralMesh™ take advantage of this by redefining how storage and compute interact entirely, enabling them to achieve GPU utilization of 90 percent or more. Approaches like these reach new levels of efficiency by unlocking compute that’s already been paid for, helping to lower the cost of tokens and achieve a profitable TCO.

AI Storage TCO Framework

So how should you calculate the total cost to purchase, build, and manage AI storage? 

When doing this for standard IT storage, the process is fairly straightforward. Just add up the initial acquisition cost of your storage to your estimated costs of operating and maintaining this infrastructure, then divide this by your total useful capacity (usually measured in terabytes). But while this will give you a good idea of the TCO of traditional storage, it will vastly underestimate the TCO of AI storage – whether due to the costs of storage bottlenecks, or because of additional costs like the power density AI infrastructure requires or the high-speed networking it needs.

This is yet another reason why the only effective way to quantify TCO isn’t the cost per terabyte but the cost per token. In other words, every cost category that goes into building, operating, and maintaining the AI storage stack shouldn’t be associated by what you paid for it, but by the value of what it produces. The question is: How efficiently does this element of my AI storage infrastructure help me output tokens? 

With that in mind, here are the different cost categories you’ll find across AI storage.

Capital Costs

Also known as capital expenditures (CapEx), these are the upfront costs associated with building out the infrastructure of your AI stack. These include:

• GPU compute: This is one of the most prominent costs associated with AI infrastructure, as well as one of the fastest to depreciate. For example, NVIDIA’s Blackwell architecture might cost twice as much as older Hopper architecture, but it delivers 35x lower cost per token, underscoring how important more recent GPU hardware can be.
• Networking fabric: Your GPUs are only as fast as the networking that connects them. This makes it necessary to look beyond standard Ethernet and use specialized networks like InfiniBand or RDMA over Converged Ethernet (RoCE), along with the associated switches, transceivers, and optical cabling. 
• Storage hardware: Because standard enterprise storage (like NAS or SAN) can’t feed AI workloads at the required throughput, more expensive NVMe-native architectures are needed to prevent underutilized GPUs. Likewise, RDMA adapters should be used to create direct data paths between flash memory and GPUs, as well as software layers that make parallel file access work across hundreds or thousands of GPU nodes.

Operating Costs

Purchased hardware still requires money to operate and maintain – just like in traditional storage architectures. However, the biggest difference for AI is the scale of these operating expenditures (OpEx). These include:

• Power: The wattage required to keep AI infrastructure up and running can be enormous. At roughly 700W per H100, you’re looking at about 700kW for a 1,000-GPU cluster. Importantly, this cost stays the same whether those GPUs are actively doing useful work or if they’re sitting idle.
• Cooling: The amount of power required for AI infrastructure is also making additional investments in cooling necessary. With traditional air cooling only capable of managing about 15-20kW per rack, more effective liquid cooling systems are needed, although building and managing these systems can add additional capital and operating costs.
• Floor space: Unless you own your own data center, floor space is calculated by power capacity. Typical rates run between $200 and $400 per kW per month, although premium markets can run higher. Just like power expenditures, these costs stay fixed regardless of infrastructure productivity.
• Management overhead: Typically smaller than other operating costs, these should still be factored in. They include workload orchestration, infrastructure monitoring, security tooling, software licensing, and any engineering and staffing costs – all of which can scale with infrastructure complexity.

Hidden Costs

Any honest AI storage TCO framework will also include those expenditures that don’t appear as budget line items but still affect your bottom line. These are the costs that show what you lost. They include:

• GPU idle time: This is the biggest line item that nobody tracks. It’s important to recognize that idle time doesn’t only waste the cost of GPUs, but compounds across other categories as well. This includes costs like floor space, power usage, depreciation, and management. Adding in these costs can raise your cost per token exponentially.
• Data migration: This can include the ongoing engineering and management costs of moving data from legacy to AI storage, between hybrid or multi-cloud environments, when upgrading hardware, and so on. Critically, it also includes any GPU idle time that occurs during the migration, paused training jobs, delayed project timelines, and any other indirect costs.
• Multi-tier management: The most effective AI storage doesn’t just function across one tier, but several. This includes HBM, DRAM, local NVMe, parallel file storage, and object storage. Because each of these tiers have different performance characteristics, rules, and access structures, it can take management to ensure data flows smoothly across these different tiers. As AI needs and models evolve, this management work can become costly.

On-prem vs. Cloud: Which Has a Lower TCO?

One question left unaddressed in the TCO discussion is which type of deployment offers greater economic advantages: on-premises or cloud-based AI storage infrastructure?

Both have clear advantages. Cloud-based infrastructure offers near-instant access to scalable storage capacity, making it possible to deliver high file throughput and low latency. It also doesn’t require any upfront investments or come with any ongoing operational expenses besides monthly usage fees. These funds can instead be reinvested into other AI initiatives, including model development, training, and tooling.

On-premises infrastructure comes with much higher initial costs for the hardware, as well as the maintenance and operations for managing that infrastructure, but it also comes with distinct benefits. These include better privacy and security controls over proprietary data, potential performance improvements through the elimination of cloud data bottlenecks, and much more predictable costs at scale. With complete control over the full stack, you also won’t need to restructure or reconfigure data to fit a third-party platform.

These benefits have long defined these two types of AI infrastructure: Cloud-based infrastructure is great for organizations that want to get started quickly or are interested in experimenting with AI, while on-premises infrastructure is better suited for longer-term initiatives and those requiring more custom configurations. However, recent research highlights a change.

As AI continues to become more firmly established within the operational landscape, and as technical advances in storage and compute continue to compound, the economic advantages of building out on-premises infrastructure are becoming more apparent.

Source: Lenovo Whitepaper “On-Premise vs Cloud: Generative AI Total Cost of Ownership (2026 Edition)”

This can be seen in the “breakeven” point – or the moment at which the money invested in an on-premises solution equals the money invested in a cloud-based solution. Previously, it took 12-18 months to reach this breakeven point. But with improved architecture now available, it’s possible to achieve this milestone in as little as 5-6 months.

These findings are underlined even further when different architectures are analyzed according to their cost efficiency (measured by cost per token). Whereas even the cheapest cloud-based models output tokens at about $1 per million, privately hosted on-prem systems can do the same at just $0.16 per million tokens.

Source: Lenovo Whitepaper “On-Premise vs Cloud: Generative AI Total Cost of Ownership (2026 Edition)”

What does this mean for organizations interested in AI? While cloud-based infrastructure may still be a good bet for experimentation, you’ll get more control over your AI stack, better sustained inference, and increasingly a lower TCO when you go with on-premises.

Token Economics — From Cost Center to Profit Engine

Tokens are helping organizations rethink not only the cost efficiency of AI infrastructure, but the role of AI infrastructure itself.

Up until recently, AI and the architecture it requires have been seen as a little more than an IT expense. But as more organizations understand the value that tokens offer as a unit of output, they’re building out better ways to calculate how these tokens translate into actual profit. For example, if cost per token is a reliable measurement of the efficiency (and, therefore, profitability) of an AI system, then revenue per token is an even more direct measure of the value AI is generating for the business.

While the math is straightforward – revenue per token is total AI revenue divided by the total number of tokens – the implications are not. Because this metric helps connect AI output directly to profit, as opposed to the more infrastructure-minded cost per token, it has become a favorite among the C-Suite. And because it helps draw a direct line from token to profit, it’s creating new ways to imagine what AI infrastructure actually does. 

Consider the AI factory. Popularized by NVIDIA’s Jensen Huang, the AI factory is a metaphor that helps distinguish AI infrastructure’s function from legacy data centers. If the job of the latter was to store and process data, the job of the AI factory is to manufacture intelligence. You start out with raw materials in the form of energy and data, then you end up with a valuable product in the form of tokens. Suddenly, AI infrastructure is not just a line item in the budget, but the core engine of an organization’s profit.

This fundamentally changes the equation. By reframing the AI stack in this way, it becomes essential to ensure that every tier of it is functioning efficiently. That means investing in a storage architecture that can support continuous inference, a network fabric that allows high throughput and low latency, and a data layer with near-instantaneous access. It also means accounting for factors such as power, which has become an increasingly constrained resource in the age of AI. 

With AI data centers around the world already reaching the limits of their power budgets, the electricity needs of these centers are expected to double by 2030. This has given rise to yet another useful metric: tokens per watt. As the ability to produce more tokens becomes limited by fixed resources like power, tokens per watt is becoming a determining factor for whether you can still grow revenue. This is a reality already acknowledged by AI-native storage solutions like NVIDIA CMX, which use dedicated KV cache storage tiers to prevent GPU stalling and deliver up to 5x power efficiency.

It’s also another argument for building out your own on-premises AI infrastructure. Token-per-watt efficiency isn’t something you can optimize on rented infrastructure – but it is something you can control at the hardware and architecture level. In fact, by utilizing on-premises systems for AI, you can realize up to a 17x lower cost per token. This level of control is the difference between AI infrastructure that generates margin and one that subsidizes another company’s bottom line.

The Business Case for Storage Investment

The economics of AI storage aren’t abstract. They show up in the actual performance numbers of organizations that have made the right infrastructure decisions — and in the hidden costs of those that haven’t.

Consider Stability AI. After switching its generative AI training environment to NeuralMesh™ Axon™ on AWS, the company achieved a 95 percent reduction in storage cost per terabyte and 93 percent GPU utilization. Before the change, storage bottlenecks (like slow small-file handling and metadata lookup latency) left GPUs starved for data. Although their infrastructure costs may have been stable, their productivity fell flat. The upgrade they made helped to both reduce their storage bill and make their entire infrastructure budget more efficient.

Similar improvements can be made through smart investments in memory. This is particularly true when it comes to inference. As context windows grow and agentic workloads produce hundreds of thousands of tokens, GPU HBM fills up. When it does, KV cache entries get evicted to slower storage tiers, making it necessary for the model to recompute what it already knows and increasing time-to-first-token (TTFT). While adding GPUs may sound like an obvious fix, it’s also the most expensive one.

Instead, WEKA’s Augmented Memory Grid™ offers a different answer. By offloading the KV cache to persistent NVMe-backed storage, the functional memory extends by 1000x. Using NVIDIA GPUDirect Storage, it’s then possible to retrieve this data at near-HBM speed. In lab testing, the results are as high as a 41x improvement in TTFT – a level of throughput that effectively breaks down the memory wall.

Building the ROI Case

Want to translate these outcomes into a business case you can present? Just calculate cost per token before and after a storage upgrade.

Your inputs are knowable. These may include: 

• Current GPU count and cost per GPU-hour
• Utilization rate
• Checkpoint frequency and average stall time
• Inference request volume and current latency. 

The output is the current cost per token. The next step is to model what changes after adding AI-optimized storage: Does your utilization rate go higher? Does latency go down? Does throughput increase? The resulting cost per token is your ROI.

What to Present to the CFO

You can make your strongest case for a storage investment when you break it down into numbers. Here are four you should focus on:

• Current GPU idle time cost: Calculated as the number of GPUs multiplied by the cost per GPU-hour, multiplied by the percentage of time those GPUs are idle. At enterprise average utilization rates, this is typically the largest single line item in the AI infrastructure budget.
• The storage investment: The CapEx and OpEx of the upgraded architecture, calculated over the expected hardware lifecycle.
• Projected utilization improvement: An estimate of how much idle time the storage upgrade eliminates, based on the specific bottlenecks identified (such as checkpoint stalls, pipeline latency, metadata queuing, and memory wall evictions).
• Payback period: The storage investment divided by the monthly savings from improved utilization. When GPU idle time is the baseline cost and storage is the fix, payback periods in the range of 3 to 6 months are realistic at mid-to-large cluster sizes.

Ultimately, when making this case, try to communicate that every dollar spent on GPUs is only as productive as the storage architecture supporting them. When most organizations crunch the numbers, they’ll likely find that that storage layer is costing them far more than any potential storage upgrade would. WEKA NeuralMesh and Augmented Memory Grid are where those numbers get fixed: not by adding hardware, but by making the hardware you already have do the work it’s meant to do.

Get a full evaluation criteria for AI storage, including an evaluation scorecard you can use during your assessments, in our Buyer’s Guide to AI Storage.