Inference Margins Are a Trap

Inference costs dropped 280x between 2022 and 2024. That stat should feel like good news. For most organizations running inference at production scale, it doesn't. That’s because demand didn't drop. It multiplied.

IDC projects 1,000x growth in inference demand by 2027. Deloitte estimates two-thirds of all enterprise AI compute is already inference. And according to a recent TrendForce analysis, AI infrastructure is now 30% compute-driven and 70% memory-driven. That ratio explains why the procurement playbook – buy more GPUs, provision more capacity, outspend the bottleneck — is failing the organizations that need it most.

The margin trap isn't a GPU problem. It's a memory problem wearing a GPU disguise.

The supply chain that isn't there

Even if buying your way out were the right strategy, the supply chain won't cooperate.

"DRAM is up nearly 90% quarter on quarter," said Steve McDowell, chief analyst at NAND Research, in a recent briefing on the memory shortage. "While flash memory is up close to 60%. And the prices reflect not just cost implication, but also, you know, prices go up when supply is constrained. There's GPUs sitting on shelves waiting for memory to stuff into them."

But this constraint isn’t close to correcting. TrendForce projects NAND flash prices will jump 70–75% quarter-over-quarter in Q2 2026, following steep increases in Q1. Samsung has hiked memory prices up to 60% since September 2025. Phison's CEO told investors that "every NAND manufacturer told us 2026 is sold out". SK Hynix's entire 2026 HBM capacity is already allocated, with data centers consuming more than 70% of all high-end memory chips produced this year.

Steve's assessment of the timeline is blunt: "The supply shortages and the pricing premiums are not going away in calendar 2026. I think if you took consensus across the analyst community and what the semiconductor and hard drive guys are all saying on their earnings calls, this is gonna stretch in at least till the first half of 2027."

For anyone building inference margin models on the assumption that hardware costs stabilize soon: recalculate.

The math that changes the conversation

When you run inference at scale with long context windows and concurrent users, the KV cache — the working memory that stores every active session's context — consumes GPU high-bandwidth memory at a ferocious rate. A single 100,000-token context window can eat 40 GB of HBM. Multiply that across hundreds of concurrent sessions, a model zoo cycling through specialized models, and multi-turn agent workflows, and you exhaust memory long before you exhaust compute.

Val Bercovici, Chief AI Officer at WEKA, frames it as a unit economics problem: "You're adding roughly a thousand times more capacity, three orders of magnitude more memory capacity at memory performance for inference, and that instantly transforms negative unit economics on generating tokens to positive."

💡 The proof point: On OCI, WEKA's Augmented Memory Grid drove 10x more concurrent users and 10x higher token throughput on a single 72-GPU bare-metal H100 cluster — the effective equivalent of turning 100 GPUs into 1,000 without purchasing a single additional card. At production scale, that delta compounds into billions of additional tokens served per day on the same hardware.

Same GPUs, far more work out of them — which is why this reads on the P&L, not just the benchmarks.

The OT mindset shift

The organizations getting this right are thinking about inference infrastructure the way manufacturers think about factory floors — not as IT provisioning, but as operational technology where utilization equals revenue.

"The reason we call these things AI factories is it's actually an OT operation," Bercovici explained. "The mentality in OT is fundamentally different from IT. If you're not cranking out and maximizing a factory's utilization at near peak capacity all the time, you're probably not making money. You're certainly not making any kind of gross margin."

This mindset shift has practical implications. Cohere and DeepSeek both published architectures that eliminate the traditional separation between training and inference infrastructure — running inference during peak hours and reclaiming those same GPUs for training overnight. No stranded assets. No idle capacity. Maximum return on constrained hardware.

What's different is that inference demand is continuous, concurrent, and memory-bound. The infrastructure that serves it profitably has to be, too.

Software-defined memory: The lever most teams haven't pulled

The structural answer to the margin trap isn't better hardware procurement. It's treating memory as a software-defined layer rather than a fixed hardware constraint.

"You have the ability now through technologies like WEKA to implement software-defined memory, particularly in AI context," said Bercovici. "And that is a bold claim to be making because the memory in GPUs is implicitly high-performance memory. It's not virtual memory in a legacy sense."

What this means in practice: NVMe flash can be configured to present memory-class performance to the inference stack.

The result is structural. You stop buying GPUs to compensate for a memory problem. You stop treating the shortage as a procurement crisis. You start treating it as what it actually is: an architecture opportunity.

What this means for your margin model

The inference market is heading toward $255 billion by 2030. The organizations that capture defensible margins in that market won't be the ones that spent the most on hardware. They'll be the ones that solved the memory economics first.

Three questions worth pressure-testing with your infrastructure and finance teams:

1. What's your real cost-per-token under concurrent production load — not synthetic benchmarks, not single-prompt demos? If the number changes significantly between demo and production, your memory architecture is the variable.
2. How much of your GPU spend is compensating for a memory constraint you could solve architecturally? The 100-to-420 GPU math isn't theoretical. It's the delta between a procurement-first approach and a memory-first approach.
3. What happens to your margin model if NAND prices increase another 70% next quarter? Because that's what TrendForce is projecting. If the answer is "we absorb it" or "we pass it through," the model isn't durable.

The memory shortage isn't going away. The organizations navigating it best aren't waiting for supply to normalize. They're rearchitecting now and turning a constrained market into a structural advantage.

Get related resources:

→ Get Our Memory Shortage Survival Guide: a practical framework for protecting inference economics when supply won't cooperate.

→ Watch the Full Analyst Briefing: Practical Strategies for Navigating the Memory Shortage — NAND Research's Steve McDowell and WEKA's Val Bercovici walk through the shortage timeline, the memory hierarchy, and what the best-positioned teams are doing differently.