# AI Teams Are Getting the Memory Layer Wrong. Here’s What to Do.

**Author:** WEKA

**Published:** October 2, 2026

![Monochrome image of a futuristic, multi-level library with a glowing central core, surrounded by bookshelves and conveyor belts carrying discs.](https://cdn.sanity.io/images/ult5g8gw/production/862dc327460c15ae13beb59513e9cdc3e5357d2b-1584x672.jpg)

## TL;DR

Rethink AI infrastructure by treating memory and context as foundations for scalable, high-performance enterprise systems.

- Overcome the memory wall with storage tiers repackaged as memory.
- Design agent architectures for full-context concurrency.
- Prepare for token-based software models and token warehouses.

The AI industry has a memory problem, and most enterprises are still underestimating it. On a recent episode of the [MindMakers podcast](https://www.youtube.com/playlist?list=PLcEvyGCx7NKgIjjowuMsFe7msbKbi6ngx), WEKA’s [Val Bercovici](https://www.linkedin.com/in/valentinbercovici/) sat down with host [John Kim](https://www.linkedin.com/in/doshkim/), co-founder and CEO of [Sendbird](https://www.linkedin.com/company/sendbird/) and [delight.ai](https://www.linkedin.com/company/get-delight/), to discuss the infrastructure constraints quietly limiting AI performance and what the next era of AI-native business actually looks like.

The conversation surfaced something counterintuitive: The competitive edge in AI right now isn’t just about which model you’re running; it’s about memory.

## What is the AI memory wall and why does it limit enterprise performance?

Most AI teams are familiar with context windows and token limits, but fewer have grappled with what’s happening at the hardware layer. When a model processes a large prompt, this data needs to find a home fast, and this home is high-bandwidth memory (HBM) packaged with GPUs. The problem is that the latest models consume all available HBM before inference even starts. This is the [memory wall](/video/bypassing-the-inference-memory-wall-to-scale-agentic-ai).

“The models themselves are so big,” Val said. “The DeepSeek-class and commercial OpenAI/Anthropic-class models are trillion-parameter models now. So they hog essentially most of the memory before working memory even gets going.”

The result has a cascading effect. Memory spills from GPU to CPU DRAM to storage tiers, with performance dropping at each level. WEKA’s approach repackages these storage tiers as memory, creating what Val calls an “abundance mindset” of available memory versus the scarcity most teams are engineering around today. The practical outcome is up to 6.5x more concurrent tokens at the same GPU spend and energy cost, without sacrificing latency.

For enterprises running agent swarms or high-throughput inference workloads, this is not a marginal improvement. It’s a rearchitecting of what’s possible.

👉 Check out this [WEKA Learn Article](/learn/ai-ml/ai-memory-wall/) for a deep dive on the AI memory wall

## What is context engineering and why are AI teams moving away from prompt engineering?

One of the most actionable shifts Val and John discussed is the industry’s move from prompt engineering to [context engineering](/article/the-context-era-has-begun) and why this matters for anyone building production AI systems today.

Current models don’t natively understand memory hierarchies. Instead, they optimize by compressing, summarizing, and ultimately discarding context. AI engineering teams have responded with “compaction”: periodically summarizing agent turns to preserve context window space. But Val argues this is a workaround, not a solution.

The real opportunity, he explained, is in parallelization: “Instead of summarizing and compacting, you give full context to a new sub-agent, let it start to evolve, and fill up its own context window.” WEKA calls this [context platform engineering](https://www.youtube.com/watch?v=NTBX-wxUhHs&pp=ygUhY29udGV4dCBwbGF0Zm9ybSBlbmdpbmVlcmluZyBXRUtB), and it points toward a future where agent swarms become the standard deployment model instead of single agents.

The practical tip for builders right now is to treat context as part of your infrastructure from the start, not as an afterthought bolted on later. Design your agent architecture with concurrency in mind.

## Is SaaS dying? How AI is changing the enterprise software business model

Val and John also tackled one of the most debated topics in enterprise tech today: Is AI killing SaaS? Val’s answer is nuanced. Systems of record aren’t going away — COBOL and mainframes are still with us, for example — but the growth layer has definitively moved.

John put it plainly: “Now I get the marketing team sending me links not to Google Docs or slides, but to apps. It’s crazy what’s happening.”

Val’s prediction: The most durable AI-era software business model is [selling tokens](/useCasePage/reduce-ai-token-costs), and mid-tier SaaS companies and neoclouds will increasingly merge out of necessity. The active system of record is shifting from structured databases to [token warehouses](/article/wekas-augmented-memory-grid-pioneering-a-token-warehouse-for-the-future): a world where a prompt is prefilled once and decoded forever, enabling real-time intelligence rather than quarterly snapshots.

## What should guide enterprise AI strategy beyond performance and profit?

Among the most striking moments of the conversation may have been the simplest. When asked what he’d want from a perfect, all-knowing AI, Val didn’t ask for productivity or profit. He invoked Issac Asimov and Gene Roddenberry to suggest a vision of AI that is safe, purposeful, and genuinely inclusive.

“We have to guide AI toward that kind of world, create incentives for AI to help us get there, and have some basic safety rules as well,” he said.

For companies navigating the noise of AI right now, this is a key reminder. The infrastructure decisions being made today — how memory is managed, how context is engineered, how agents are architected — aren’t simply technical choices; they’re the foundation of what AI becomes.