# Security Has Three Jobs Now. Most Teams Are Only Doing Two. **Author:** Betsy Chernoff **Published:** August 24, 2026 ![Two people working at multi-monitor computers displaying data and network graphs in a dark, modern room.](https://cdn.sanity.io/images/ult5g8gw/production/1d70cdd2a6b289785f151cbc6fb4a1d72a6537c3-1408x768.jpg) ## TL;DR Reframe security to cover AI data in use, protecting model memory as a new boundary. - Define governance, inference, and memory as a single security chain. - Isolate KV cache so tenants never share working context. - Preserve long-lived AI memory while enforcing strict ownership boundaries. We know how to protect data when it's sitting still. We know how to protect it when it's moving. But what about when it's thinking? For thirty years, security has treated data as a noun: a thing to be locked in storage or escorted across a network. Stateful AI just turned data into a verb. The moment a model starts working on your behalf, your prompts, retrieved documents, tool outputs, and reasoning history all become part of its live working memory while the answer takes shape. It's not at rest. It's not in transit. It's in use. Most security architectures have no name for it, no owner for it, and no line item for it. ## The model remembers more than the application reveals Early generative AI was ask, answer, forget. Today's AI doesn't forget. That's the whole point. Agents carry context across tools and sessions. Copilots build on hours of conversation. Guardrails compare this moment's event against last week's baseline. And AI platforms serve dozens of teams, customers, or agencies from the same pool of GPUs. All of that state has to live somewhere. Much of it lives in the KV cache, [the model's working memory](/learn/ai-ml/what-is-kv-cache/), built as it reads your context and consulted as it generates each token. Caching that memory instead of recomputing it is what [makes long-context AI affordable](/learn/ai-ml/inference-optimization/) at all. But every cache is a record of what someone was thinking about. Which raises a question your last security review probably didn't ask: who's allowed to know that a piece of context exists, and who's allowed to reuse it? ## The attack doesn't break in. It listens. Researchers name their exploits like heist movies, and the past year has produced a full marquee: PROMPTPEEK. InputSnatch. Early Bird. Shadow in the Cache. Most of them work off a tell. When a model has seen your content before, it answers faster. That's the cache doing its job. On shared infrastructure, that speed becomes a signal anyone can read. An attacker doesn't need to see another tenant's prompt. They ask their own questions and time the answers, reconstructing what the system already knows one fast response at a time. It's less a break-in than a poker game, and the infrastructure has a tell. The uncomfortable part is that encryption never fails in these attacks. Not once. The leak is the behavior of the [inference system](/learn/ai-ml/how-ai-inference-works/) itself, which means it lives precisely in the blind spot between your at-rest controls and your in-transit controls. That's what promotes the KV cache from performance feature to security boundary. ## Three layers, one promise There is no switch labeled "secure data in use" and no single vendor is responsible for it. Just like most security stacks, it’s a shared responsibility model. What works is a chain of custody for the model's memory. - **Governance decides who you are:** it authenticates each request and pins it to a project, tenant, or workload. - **Inference decides what you can reuse:** cached memory is found and retrieved only under the identity that created it. - **Memory keeps the promise:** the boundary holds as state persists and moves across sessions, machines, and GPU hosts. Identity, in other words, gets baked into the memory itself. Two teams can send the same model the exact same prompt and, as far as the cache is concerned, exist in different universes. Break any link in that chain and the other two can't save you. ## Embedded LLM tried to break the boundary Principles are cheap. So [Embedded LLM](https://embeddedllm.com/), the team building a federated AI utility powered by vLLM, built the architecture and then tried to catch it failing. Their TokenVisor platform is the governance layer that makes project-scoped isolation possible: it authenticates every request, maps it to the right organization and project, enforces access controls and metering, and injects a project-specific value into the cache's identity. Everything downstream depends on that layer answering one question correctly: whose memory is this? The test was deliberately simple. Two projects, one shared serving environment, one identical 42,000-token prompt. The only variable was who was asking. The result was exactly the asymmetry a secure system should produce. The original project, returning to its own context, got the warm path: time-to-first-token improved 9.2 to 20.1 times, depending on the model. The second project got nothing — 0.94 to 1.00 times the cold baseline, as if the cache didn't exist. Same prompt. Same model. Same infrastructure. Different memory. ## AI can't afford to forget The obvious reaction: if memory creates risk, erase it constantly. That's secure the way shredding every document after reading it is secure. It also makes the most valuable AI nearly impossible. A supply chain guardrail can't judge a package update in isolation; the danger emerges only against weeks of dependency history, CI/CD edits, and maintainer behavior. An incident-response agent builds its case the same way, carrying surviving conclusions into each next step. Force it to rebuild the investigation every time a session ends and it gets slower, costlier, and less capable just as the incident grows. A model can remember too broadly. It can also forget too much. Both are architectural failures. The goal is not less memory. It's memory with a property line. That's where WEKA solutions enter the design: [NeuralMesh™](/product/neuralmesh) with Augmented Memory Grid keeps reusable KV state available across sessions and GPU hosts. It doesn't decide who owns the context; governance does. It doesn't decide whether the cache may be reused; inference does. Its job is to make sure the governed boundary survives when the memory moves. ## The new job in town: data in use Security reviews have spent decades on two questions: where is the data stored, and how is it protected while it moves? Stateful AI adds a third: when the model remembers, whose memory is it? Who can detect it exists? Who can retrieve it? Who benefits from work already done? Does the answer hold when the request lands on a different GPU? The security boundary has moved into inference memory. The organizations that see that now will build AI that remembers what matters — without forgetting who the memory belongs to. **We cover this in depth in our latest white paper** [_Securing Stateful AI Inference_](/resources/white-paper/your-kv-cache-is-now-part-of-the-attack-surface/). Read it to learn about the full KV cache attack surface, the shared-responsibility model, and the project-scoped architecture designed and validated by Embedded LLM.