# Your KV Cache Is Now Part of the Attack Surface Your KV cache is now part of the attack surface. This paper covers governance, cache isolation, and persistent memory as prompts and reasoning history become working context. **Type:** White Paper ## Introduction AI is changing how organizations think about security and governance across the infrastructure stack. As systems become more stateful and agentic, sensitive context no longer exists only in stored files, model artifacts, or API traffic. Prompts, documents, tool outputs, and reasoning history also move through the inference environment as working context, often through the key-value, or KV, cache. This paper examines what that shift means for secure AI architecture. It explores why the KV cache is becoming part of the security boundary, how shared inference environments require cache reuse to be scoped to the right tenant, project, or workload, and why security AI needs persistent context to reason across long-running workflows such as threat detection, supply chain monitoring, and incident response. Addressing these requirements is a shared responsibility across the AI stack. Governance services authenticate requests, establish identity, and define policy boundaries. Model-serving and KV-cache layers determine where context may be reused. Memory and storage infrastructure must then keep that state fast, persistent, and isolated as work moves across sessions, replicas, and GPU hosts. Secure stateful inference depends on these layers working together rather than relying on any one control in isolation. Augmented Memory Grid fits into this architecture as the persistent memory layer for stateful inference. Built on WEKA® NeuralMesh™ and the Token Warehouse™, Augmented Memory Grid externalizes KV cache from individual GPU hosts and makes it available across the inference environment. When paired with governance that binds each request to an authenticated project or workload, Augmented Memory Grid helps preserve context within the appropriate boundary—supporting authorized reuse, maintaining separation across projects, and making KV cache state available for long-running security workflows. This gives organizations a practical foundation for sharing infrastructure without sharing inference state, while enabling AI systems to retain the context they need to operate effectively at scale. ## Stateful Inference Changes the Security Boundary Early versions of generative AI served one request at a time: ask, answer, forget. Today's AI (from agents to copilots to always-on guardrails) runs continuously over hours or days and often reuses and extends context across many steps instead of starting fresh each time. This shift is visible in two places. **The first is shared inference.** AI clouds, sovereign AI platforms, and enterprise GPU clusters increasingly serve multiple tenants, teams, agencies, or projects on common infrastructure. This improves utilization and reduces cost, but it also changes the trust model. A shared model instance may be acceptable. Shared inference state is not. **The second is security AI.** Guardrails, threat detection agents, supply chain monitors, and incident response copilots need to reason across time. They must compare the current event against prior package versions, dependency graphs, CI/CD changes, tool outputs, behavioral signals, and investigation history. This requires application state across prompts, tools, and can benefit from reusable KV cache state across sessions and GPU hosts. Both trends converge at the same infrastructure layer: inference memory. The KV cache is no longer only a performance optimization. It is part of the security boundary. Traditional architectures treat inference state as ephemeral. KV cache is held in GPU HBM or host DRAM until it is evicted, lost, or rebuilt. This creates operational trade-offs. Systems can recompute prefill, pin sessions to GPU hosts, duplicate cache across replicas, or allow broad cache reuse. Each choice creates cost, complexity, or security exposure. A different architecture is required. The inference memory layer must be persistent enough to support long-running context, fast enough to remain in the inference path, and scoped tightly enough to preserve tenant and project boundaries. ## The Challenge: AI Security Now Includes Data in Use Once inference begins, sensitive context becomes part of the model’s working state. Prompts, retrieved documents, tool outputs, and prior reasoning are materialized in active tensors and the KV cache, where they remain accessible throughout generation. The security question therefore shifts from how data is stored or transported to how live inference state is isolated, reused, and retained. ```json { "_key": "rp00069", "_type": "richTableBlock", "headerRows": 1, "markDefs": null, "rows": [ { "_key": "rp00038", "_type": "row", "cells": [ { "_key": "rp00029", "_type": "richTableCell", "content": [ { "_key": "rp00031", "_type": "block", "children": [ { "_key": "rp00030", "_type": "span", "marks": [], "text": "Data State" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00032", "_type": "richTableCell", "content": [ { "_key": "rp00034", "_type": "block", "children": [ { "_key": "rp00033", "_type": "span", "marks": [], "text": "Typical Controls" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00035", "_type": "richTableCell", "content": [ { "_key": "rp00037", "_type": "block", "children": [ { "_key": "rp00036", "_type": "span", "marks": [], "text": "What It Covers" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00048", "_type": "row", "cells": [ { "_key": "rp00039", "_type": "richTableCell", "content": [ { "_key": "rp00041", "_type": "block", "children": [ { "_key": "rp00040", "_type": "span", "marks": [], "text": "Data at rest" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00042", "_type": "richTableCell", "content": [ { "_key": "rp00044", "_type": "block", "children": [ { "_key": "rp00043", "_type": "span", "marks": [], "text": "Storage encryption, key management, backup and retention controls" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00045", "_type": "richTableCell", "content": [ { "_key": "rp00047", "_type": "block", "children": [ { "_key": "rp00046", "_type": "span", "marks": [], "text": "Files, model weights, checkpoints, datasets, and persistent storage" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00058", "_type": "row", "cells": [ { "_key": "rp00049", "_type": "richTableCell", "content": [ { "_key": "rp00051", "_type": "block", "children": [ { "_key": "rp00050", "_type": "span", "marks": [], "text": "Data in transit" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00052", "_type": "richTableCell", "content": [ { "_key": "rp00054", "_type": "block", "children": [ { "_key": "rp00053", "_type": "span", "marks": [], "text": "TLS, API gateways, service-mesh policy, and network segmentation" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00055", "_type": "richTableCell", "content": [ { "_key": "rp00057", "_type": "block", "children": [ { "_key": "rp00056", "_type": "span", "marks": [], "text": "API calls and data moving between systems" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00068", "_type": "row", "cells": [ { "_key": "rp00059", "_type": "richTableCell", "content": [ { "_key": "rp00061", "_type": "block", "children": [ { "_key": "rp00060", "_type": "span", "marks": [], "text": "Data in use" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00062", "_type": "richTableCell", "content": [ { "_key": "rp00064", "_type": "block", "children": [ { "_key": "rp00063", "_type": "span", "marks": [], "text": "Runtime isolation, project-scoped cache keys, governed reuse, and memory-tier controls" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00065", "_type": "richTableCell", "content": [ { "_key": "rp00067", "_type": "block", "children": [ { "_key": "rp00066", "_type": "span", "marks": [], "text": "Active tensors, KV cache, tool outputs, and intermediate inference state" } ], "markDefs": [], "style": "normal" } ] } ] } ] } ``` ![Three data states in AI systems: data at rest and in transit are covered by traditional security, while data in use — KV cache, active tensors, and intermediate inference state — is an unprotected gap.](https://cdn.sanity.io/images/ult5g8gw/production/9a312dc98cb15d7988fb35796239497daeb9c9bd-924x250.png) Figure 1: The three states of data in AI systems. Traditional controls protect data at rest and in transit. The KV cache — data in use — sits outside their scope. During inference, the prefill phase processes the input context and creates the KV cache. The decode phase then generates tokens while repeatedly reading from that cache. This improves efficiency, but it introduces two distinct security requirements. **Isolation across trust boundaries. **Shared inference frameworks use techniques such as prefix caching to reuse KV state when requests contain matching token sequences. In a multi-tenant environment, cache hits and misses can have measurably different latency. If KV state is not scoped by tenant or project, those differences can become timing side channels that reveal whether another user’s context is cached—even when stored data and network traffic remain encrypted. **Persistence for security AI. **Long-running guardrails cannot evaluate every event in isolation. A software supply chain defense system may need to compare current activity with earlier dependency graphs, CI/CD pipeline changes, maintainer behavior, signing metadata, runtime signals, and known payload patterns. That requires context to persist across prompts, tools, sessions, and time. > The core issue is the same in both cases: Inference state is now part of the security boundary. It must be isolated when workloads share infrastructure and preserved when security systems need to reason across long-running workflows. ## The KV Cache Attack Surface: What the Research Shows Over the past year, security researchers have documented a growing class of attacks and vulnerabilities that target how the KV cache is handled, rather than the network or data at rest. Read together, they describe a common attack surface that emerges when KV state is shared, observable, or recoverable across a trust boundary. The five examples below are representative. None requires breaking encryption or compromising the network; each instead exploits how inference systems store, identify, or reuse cached state to improve performance. ```json { "_key": "rp00145", "_type": "richTableBlock", "headerRows": 1, "markDefs": null, "rows": [ { "_key": "rp00094", "_type": "row", "cells": [ { "_key": "rp00085", "_type": "richTableCell", "content": [ { "_key": "rp00087", "_type": "block", "children": [ { "_key": "rp00086", "_type": "span", "marks": [], "text": "Attack / Vulnerability" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00088", "_type": "richTableCell", "content": [ { "_key": "rp00090", "_type": "block", "children": [ { "_key": "rp00089", "_type": "span", "marks": [], "text": "Mechanism" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00091", "_type": "richTableCell", "content": [ { "_key": "rp00093", "_type": "block", "children": [ { "_key": "rp00092", "_type": "span", "marks": [], "text": "Primary Risk" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00104", "_type": "row", "cells": [ { "_key": "rp00095", "_type": "richTableCell", "content": [ { "_key": "rp00097", "_type": "block", "children": [ { "_key": "rp00096", "_type": "span", "marks": [ "7737a0cebc0a" ], "text": "PROMPTPEEK" } ], "markDefs": [ { "_key": "7737a0cebc0a", "_type": "articleBodyLink", "actionType": "external", "externalUrl": "https://www.ndss-symposium.org/ndss-paper/i-know-what-you-asked-prompt-leakage-via-kv-cache-sharing-in-multi-tenant-llm-serving/" } ], "style": "normal" } ] }, { "_key": "rp00098", "_type": "richTableCell", "content": [ { "_key": "rp00100", "_type": "block", "children": [ { "_key": "rp00099", "_type": "span", "marks": [], "text": "Uses TTFT differences from shared prefix caching to reconstruct victim prompts across tenant boundaries." } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00101", "_type": "richTableCell", "content": [ { "_key": "rp00103", "_type": "block", "children": [ { "_key": "rp00102", "_type": "span", "marks": [], "text": "Prompt leakage through timing behavior" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00114", "_type": "row", "cells": [ { "_key": "rp00105", "_type": "richTableCell", "content": [ { "_key": "rp00107", "_type": "block", "children": [ { "_key": "rp00106", "_type": "span", "marks": [ "57c1804fef0f" ], "text": "InputSnatch" } ], "markDefs": [ { "_key": "57c1804fef0f", "_type": "articleBodyLink", "actionType": "external", "externalUrl": "https://arxiv.org/abs/2411.18191" } ], "style": "normal" } ] }, { "_key": "rp00108", "_type": "richTableCell", "content": [ { "_key": "rp00110", "_type": "block", "children": [ { "_key": "rp00109", "_type": "span", "marks": [], "text": "Uses response-time variation from prefix and semantic cache sharing to expose private input content." } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00111", "_type": "richTableCell", "content": [ { "_key": "rp00113", "_type": "block", "children": [ { "_key": "rp00112", "_type": "span", "marks": [], "text": "Private input leakage" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00124", "_type": "row", "cells": [ { "_key": "rp00115", "_type": "richTableCell", "content": [ { "_key": "rp00117", "_type": "block", "children": [ { "_key": "rp00116", "_type": "span", "marks": [ "f4c36a3dd706" ], "text": "Early Bird" } ], "markDefs": [ { "_key": "f4c36a3dd706", "_type": "articleBodyLink", "actionType": "external", "externalUrl": "https://arxiv.org/abs/2409.20002" } ], "style": "normal" } ] }, { "_key": "rp00118", "_type": "richTableCell", "content": [ { "_key": "rp00120", "_type": "block", "children": [ { "_key": "rp00119", "_type": "span", "marks": [], "text": "Uses streaming output timing to distinguish cache hits from misses." } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00121", "_type": "richTableCell", "content": [ { "_key": "rp00123", "_type": "block", "children": [ { "_key": "rp00122", "_type": "span", "marks": [], "text": "Token-level recovery through timing" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00134", "_type": "row", "cells": [ { "_key": "rp00125", "_type": "richTableCell", "content": [ { "_key": "rp00127", "_type": "block", "children": [ { "_key": "rp00126", "_type": "span", "marks": [ "6ee18577ec58" ], "text": "Shadow in the Cache" } ], "markDefs": [ { "_key": "6ee18577ec58", "_type": "articleBodyLink", "actionType": "external", "externalUrl": "https://arxiv.org/abs/2508.09442" } ], "style": "normal" } ] }, { "_key": "rp00128", "_type": "richTableCell", "content": [ { "_key": "rp00130", "_type": "block", "children": [ { "_key": "rp00129", "_type": "span", "marks": [], "text": "Uses KV tensor inversion, collision, and injection attacks against leaked or exfiltrated cache state." } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00131", "_type": "richTableCell", "content": [ { "_key": "rp00133", "_type": "block", "children": [ { "_key": "rp00132", "_type": "span", "marks": [], "text": "Prompt reconstruction and cache manipulation" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00144", "_type": "row", "cells": [ { "_key": "rp00135", "_type": "richTableCell", "content": [ { "_key": "rp00137", "_type": "block", "children": [ { "_key": "rp00136", "_type": "span", "marks": [ "7aacd1fa1754" ], "text": "CVE-2025-25183" } ], "markDefs": [ { "_key": "7aacd1fa1754", "_type": "articleBodyLink", "actionType": "external", "externalUrl": "https://github.com/vllm-project/vllm/security/advisories/GHSA-rm76-4mrf-v9r8" } ], "style": "normal" } ] }, { "_key": "rp00138", "_type": "richTableCell", "content": [ { "_key": "rp00140", "_type": "block", "children": [ { "_key": "rp00139", "_type": "span", "marks": [], "text": "Uses predictable hash collisions to enable prefix-cache poisoning and incorrect reuse." } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00141", "_type": "richTableCell", "content": [ { "_key": "rp00143", "_type": "block", "children": [ { "_key": "rp00142", "_type": "span", "marks": [], "text": "Wrong reuse across prompts or tenants" } ], "markDefs": [], "style": "normal" } ] } ] } ] } ``` ![Hub-and-spoke diagram of the KV cache attack surface in shared inference: PromptPeek, InputSnatch, Early Bird, Shadow in the Cache, and CVE-2025-25183.](https://cdn.sanity.io/images/ult5g8gw/production/758e39bea3c21c4aa035f08647cf7bedbea4c290-2048x1189.png) Figure 2. KV Cache Attack Surface. Five documented attacks exploit KV cache sharing, timing behaviour, and tensor exposure in shared inference infrastructure. All five operate without breaking encryption or compromising the network. The common issue is not simply that a KV cache exists. It is that cache state, cache behavior, or cache identity becomes visible or reusable outside the trust boundary that created it. The corresponding defense is architectural: bind cache keys to authenticated project identity, prevent cross-project lookup and reuse, and protect persisted KV blocks below the inference layer. ```json { "_key": "cbd63ce3a6e3", "_type": "textCallout", "content": [ { "_key": "rp00152", "_type": "block", "children": [ { "_key": "rp00151", "_type": "span", "marks": [], "text": "No single layer of the AI stack can defend against KV-cache attacks on its own. Governance, inference, and persistent memory infrastructure each own a distinct part of the defense." } ], "markDefs": [], "style": "normal" }, { "_key": "rp00154", "_type": "block", "children": [ { "_key": "rp00153", "_type": "span", "marks": [], "text": "Governance authenticates requests and establishes identity before inference begins" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "rp00156", "_type": "block", "children": [ { "_key": "rp00155", "_type": "span", "marks": [], "text": "The inference and KV-cache layer uses project identity to control cache lookup and reuse" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "rp00158", "_type": "block", "children": [ { "_key": "rp00157", "_type": "span", "marks": [], "text": "Tensor obfuscation and selective sharing protect KV representations against inversion and leakage" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "rp00160", "_type": "block", "children": [ { "_key": "rp00159", "_type": "span", "marks": [], "text": "The persistent memory tier retains project-scoped KV state with storage-layer access controls" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" } ], "markDefs": null, "title": "Securing data in use is a shared responsibility" } ``` ## Common Defenses: A Shared Responsibility Across the Stack No single layer of the AI stack can defend against these attacks on its own, and no single vendor — WEKA included — solves them alone. Securing data in use is a shared responsibility. Governance authenticates each request and scopes it to the correct tenant or project. The inference runtime and KV-cache manager determine what may be reused, when, and under which cache key. The persistent memory layer preserves those boundaries as KV state is stored and retrieved across the infrastructure. Several approaches are emerging to reduce KV-cache risk across the AI stack. They operate at different points in the request and inference path, including cache-key derivation, tenant isolation, selective sharing, tensor protection, and model-runtime safety. Production systems can combine these controls based on their trust boundaries and the threats they need to address. ### Defenses for cross-tenant isolation and cache privacy - **Project-scoped cache salting **binds authenticated tenant or project identity into cache-key derivation. An entry is reused only when both the token context and project identity match. When implemented consistently across the stack, this prevents one project from resolving another project’s KV blocks. - **Selective sharing **separates cache entries that are safe to share from those that must remain private. SafeKV, for example, keeps sensitive entries in private caches while allowing non-sensitive content to retain some of the performance benefits of shared caching. - **KV-Cloak **protects the KV representation itself. It applies a reversible, matrix-based transformation that obfuscates KV tensors against inversion, collision, and injection attacks while allowing the inference runtime to recover them for authorized computation. ### Defenses for prompt injection and jailbreaks - **CachePrune **targets indirect prompt injection by identifying and pruning task-triggering features from the KV cache, reducing the model’s tendency to treat untrusted retrieved content as instructions. - **RobustKV **addresses jailbreak attacks by selectively evicting KV entries associated with low-ranked tokens, weakening the concealed harmful request the jailbreak prompt is attempting to preserve. No single control covers the full attack surface. Project-scoped keys prevent unauthorized cache reuse. Selective sharing reduces opportunities for timing leakage. Tensor obfuscation protects KV state if it is exposed. Cache-pruning methods address malicious instructions and jailbreak behavior at the model-runtime layer. A production architecture layers these protections across governance, inference, and persistent memory based on the risks it needs to address. ## Augmented Memory Grid: KV Cache That Securely Carries Context Across Sessions Augmented Memory Grid moves reusable KV state out of per-host GPU memory and into a shared Token Warehouse built on NeuralMesh. Instead of requiring each GPU host to retain session state locally in HBM or DRAM, Augmented Memory Grid persists KV blocks on NVMe and makes them available to the inference system over a high-throughput RDMA path. The Token Warehouse is shared across GPU hosts, but cache lookup and reuse are not global. A governance layer binds authenticated project identity into cache-key generation so KV state can be reused within the correct project boundary without being reused across projects. ### Control Responsibilities Across the Stack The architecture divides responsibility across three logical layers: ```json { "_key": "rp00226", "_type": "richTableBlock", "headerRows": 1, "markDefs": null, "rows": [ { "_key": "rp00204", "_type": "row", "cells": [ { "_key": "rp00198", "_type": "richTableCell", "content": [ { "_key": "rp00200", "_type": "block", "children": [ { "_key": "rp00199", "_type": "span", "marks": [], "text": "Layer" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00201", "_type": "richTableCell", "content": [ { "_key": "rp00203", "_type": "block", "children": [ { "_key": "rp00202", "_type": "span", "marks": [], "text": "Role" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00211", "_type": "row", "cells": [ { "_key": "rp00205", "_type": "richTableCell", "content": [ { "_key": "rp00207", "_type": "block", "children": [ { "_key": "rp00206", "_type": "span", "marks": [], "text": "Governance layer" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00208", "_type": "richTableCell", "content": [ { "_key": "rp00210", "_type": "block", "children": [ { "_key": "rp00209", "_type": "span", "marks": [], "text": "Authenticates the request, maps it to an organization and project, applies RBAC and policy, and injects a project-specific cache_salt derived from authenticated identity — not supplied by the client." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00218", "_type": "row", "cells": [ { "_key": "rp00212", "_type": "richTableCell", "content": [ { "_key": "rp00214", "_type": "block", "children": [ { "_key": "rp00213", "_type": "span", "marks": [], "text": "Inference and KV cache layer" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00215", "_type": "richTableCell", "content": [ { "_key": "rp00217", "_type": "block", "children": [ { "_key": "rp00216", "_type": "span", "marks": [], "text": "vLLM, TensorRT-LLM, SGLang, LMCache, or another KV manager computes and stores KV blocks using cache keys derived from request tokens and the project-specific salt." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00225", "_type": "row", "cells": [ { "_key": "rp00219", "_type": "richTableCell", "content": [ { "_key": "rp00221", "_type": "block", "children": [ { "_key": "rp00220", "_type": "span", "marks": [], "text": "Persistent memory layer" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00222", "_type": "richTableCell", "content": [ { "_key": "rp00224", "_type": "block", "children": [ { "_key": "rp00223", "_type": "span", "marks": [], "text": "Augmented Memory Grid on NeuralMesh stores KV blocks in a shared namespace with project-level scoping, access controls, data protection, and high-speed GPU access over RDMA and GPUDirect Storage." } ], "markDefs": [], "style": "normal" } ] } ] } ] } ``` Each layer owns a different part of the security model. Governance establishes trusted project identity. The inference and KV-cache layer uses that identity to control lookup and reuse. The persistent memory layer preserves those boundaries as KV blocks are stored and retrieved across the infrastructure. ## Core Isolation Mechanism: Project-Scoped Cache Keys Cache-key isolation is the central control in this design. A project-specific salt is bound to authenticated identity and included in cache-key derivation. Conceptually this looks like: > cache_key = SHA-256(tokens + project_cache_salt) The client does not supply the salt. The governance layer injects it after authentication, and the KV-cache manager uses it when computing the cache key. Two projects can therefore submit identical prompts without resolving to the same KV object, because their project-specific salts produce different keys. This separates model sharing from cache sharing. Multiple projects can use the same model-serving infrastructure, while KV-cache reuse remains scoped to the project that created the state. ## Reference Architecture The following diagram shows how five deployable layers—governance, inference runtime, KV-cache management, Augmented Memory Grid, and GPU HBM—compose into a secure multi-tenant inference stack. ![Reference architecture: a user or security agent issues requests through a governance layer that injects cache_salt, an inference runtime and KV-cache manager that derives project-scoped keys, and Augmented Memory Grid on NeuralMesh at 308 GB/s over GDS / RDMA.](https://cdn.sanity.io/images/ult5g8gw/production/4304068259a433506f09369aa26e9e34f61f34ac-924x528.png) Figure 4. The three logical layers of a secure inference deployment: a governance layer enforcing identity and policy, an inference and KV cache layer computing and storing cache blocks, and Augmented Memory Grid providing persistent project-scoped storage. The architecture is not tied to a specific governance product. Any trusted governance layer can participate, provided it authenticates the request, derives project identity, and passes a project-scoped value that the KV-cache manager uses during cache-key generation. ## Architecture stack: Secure multi-tenant inference with Augmented Memory Grid ```json { "_key": "888bdb46f54c", "_type": "textCallout", "content": [ { "_key": "rp00249", "_type": "block", "children": [ { "_key": "rp00248", "_type": "span", "marks": [ "strong" ], "text": "User / Security Agent" } ], "markDefs": [], "style": "normal" }, { "_key": "25a8db96b5ca", "_type": "block", "children": [ { "_key": "rp00250", "_type": "span", "marks": [ "strong" ], "text": "Governance Layer:" } ], "markDefs": [], "style": "normal" }, { "_key": "rp00251", "_type": "block", "children": [ { "_key": "rp00250", "_type": "span", "marks": [], "text": "Authenticates the request and maps to org/project identity" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "c5dc35f51c6a", "_type": "block", "children": [ { "_key": "rp00250", "_type": "span", "marks": [], "text": "Enforces RBAC, metering, and policy" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "94f074739ca9", "_type": "block", "children": [ { "_key": "rp00250", "_type": "span", "marks": [], "text": "Injects project-specific cache_salt" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "rp00253", "_type": "block", "children": [ { "_key": "rp00252", "_type": "span", "marks": [ "strong" ], "text": "Inference Runtime:" } ], "markDefs": [], "style": "normal" }, { "_key": "aa6ecef1b76a", "_type": "block", "children": [ { "_key": "rp00252", "_type": "span", "marks": [], "text": "vLLM, TensorRT-LLM, SGLang, or equivalent" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "rp00255", "_type": "block", "children": [ { "_key": "rp00254", "_type": "span", "marks": [ "strong" ], "text": "KV Cache Manager:" } ], "markDefs": [], "style": "normal" }, { "_key": "d5d85666994a", "_type": "block", "children": [ { "_key": "rp00254", "_type": "span", "marks": [], "text": "LM Cache, or equivalent" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "55cc97f2b03c", "_type": "block", "children": [ { "_key": "rp00254", "_type": "span", "marks": [], "text": "Computes project-scoped cache keys for lookup and reuse" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "bf916a4e80d9", "_type": "block", "children": [ { "_key": "rp00254", "_type": "span", "marks": [], "text": "Stores and retrieves KV blocks within the authorized project scope" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "25f2d70a4488", "_type": "block", "children": [ { "_key": "rp00256", "_type": "span", "marks": [ "strong" ], "text": "Augmented Memory Grid on NeuralMesh:" } ], "markDefs": [], "style": "normal" }, { "_key": "rp00257", "_type": "block", "children": [ { "_key": "rp00256", "_type": "span", "marks": [], "text": "Token Warehouse" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "7f8cceac72e2", "_type": "block", "children": [ { "_key": "rp00256", "_type": "span", "marks": [], "text": "Shared namespace across all GPU hosts" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "4f6f999c88eb", "_type": "block", "children": [ { "_key": "rp00256", "_type": "span", "marks": [], "text": "RDMA data path; project-scoped KV blocks" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" }, { "_key": "rp00259", "_type": "block", "children": [ { "_key": "rp00258", "_type": "span", "marks": [ "strong" ], "text": "GPU HBM:" } ], "markDefs": [], "style": "normal" }, { "_key": "88310747967f", "_type": "block", "children": [ { "_key": "rp00258", "_type": "span", "marks": [], "text": "Active working set; decode reads hot KV state" } ], "level": 1, "listItem": "bullet", "markDefs": [], "style": "normal" } ], "markDefs": null } ``` The essential requirement is that governance authenticate the request, derive project identity, and bind that identity into cache-key generation before KV lookup occurs. ## Runtime Flow At runtime, each request follows the same sequence: establish project identity, derive a project-scoped cache key, check for reusable KV state, create or retrieve the required KV blocks, and place the active working set in GPU HBM for generation. ### 1. The request is authenticated and scoped to a project The governance layer authenticates the request and maps it to an organization, project, tenant, or workload boundary. It then injects a project-specific cache_salt derived from authenticated identity. The client cannot choose or alter this value. ### 2. The KV-cache manager derives a project-scoped cache key The KV-cache manager combines the request tokens with the governance-injected cache_salt to derive the cache key. This ensures that cache identity reflects both the content being processed and the project authorized to reuse it. ### 3. The runtime checks for reusable KV state The KV-cache manager looks for KV blocks that match the project-scoped key. A cache hit is valid only when both the token context and project identity match. Identical context stored under another project produces a different key and therefore behaves as a cache miss. ### 4. A cache hit is retrieved; a cache miss runs prefill When matching KV blocks exist, the runtime retrieves them and avoids or reduces repeated prefill work. When no matching blocks exist, the model processes the long context during prefill and creates a new KV cache. That context may include a long prompt, source repository, dependency graph, policy document, or investigation history. ### 5. Augmented Memory Grid stores and transfers the KV blocks Augmented Memory Grid persists reusable KV blocks in the NeuralMesh Token Warehouse. GPU HBM retains the active working set, while less-active state is stored on NVMe and retrieved when needed. This expands the available KV-cache tier without requiring the application to manage cache locality or bind the session to one GPU host. Augmented Memory Grid uses RDMA to move KV data between the Token Warehouse and GPU memory without routing every transfer through the CPU and system DRAM. WEKA’s published benchmark shows over 308 GB/s of direct Token Warehouse-to-GPU throughput per host. ### 6. Decode uses the active KV state in GPU HBM Once the required KV blocks are available in HBM, the model enters decode and generates output one token at a time. The runtime reads the active KV state during generation, while Augmented Memory Grid provides the persistent memory tier behind that working set. ## Architectural Properties This flow produces infrastructure-level properties that matter for secure multi-tenant inference: cache reuse remains tied to authenticated project identity, persisted KV state is available across authorized replicas, and GPU HBM is reserved for active inference rather than duplicated cache state. ```json { "_key": "rp00338", "_type": "richTableBlock", "headerRows": 1, "markDefs": null, "rows": [ { "_key": "rp00302", "_type": "row", "cells": [ { "_key": "rp00296", "_type": "richTableCell", "content": [ { "_key": "rp00298", "_type": "block", "children": [ { "_key": "rp00297", "_type": "span", "marks": [], "text": "Architectural property" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00299", "_type": "richTableCell", "content": [ { "_key": "rp00301", "_type": "block", "children": [ { "_key": "rp00300", "_type": "span", "marks": [], "text": "Result" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00309", "_type": "row", "cells": [ { "_key": "rp00303", "_type": "richTableCell", "content": [ { "_key": "rp00305", "_type": "block", "children": [ { "_key": "rp00304", "_type": "span", "marks": [], "text": "Project-scoped reuse" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00306", "_type": "richTableCell", "content": [ { "_key": "rp00308", "_type": "block", "children": [ { "_key": "rp00307", "_type": "span", "marks": [], "text": "Warm KV reuse is available only when the request context and authenticated project identity match." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00316", "_type": "row", "cells": [ { "_key": "rp00310", "_type": "richTableCell", "content": [ { "_key": "rp00312", "_type": "block", "children": [ { "_key": "rp00311", "_type": "span", "marks": [], "text": "Cross-project blocking" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00313", "_type": "richTableCell", "content": [ { "_key": "rp00315", "_type": "block", "children": [ { "_key": "rp00314", "_type": "span", "marks": [], "text": "Identical prompts submitted by different projects produce different cache keys and do not resolve to the same KV blocks." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00323", "_type": "row", "cells": [ { "_key": "rp00317", "_type": "richTableCell", "content": [ { "_key": "rp00319", "_type": "block", "children": [ { "_key": "rp00318", "_type": "span", "marks": [], "text": "Lower HBM pressure" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00320", "_type": "richTableCell", "content": [ { "_key": "rp00322", "_type": "block", "children": [ { "_key": "rp00321", "_type": "span", "marks": [], "text": "HBM holds the active working set, while reusable KV blocks persist outside the GPU." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00330", "_type": "row", "cells": [ { "_key": "rp00324", "_type": "richTableCell", "content": [ { "_key": "rp00326", "_type": "block", "children": [ { "_key": "rp00325", "_type": "span", "marks": [], "text": "Replica-independent scaling" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00327", "_type": "richTableCell", "content": [ { "_key": "rp00329", "_type": "block", "children": [ { "_key": "rp00328", "_type": "span", "marks": [], "text": "Persisted KV state is retained once within its project scope rather than duplicated across every serving replica." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00337", "_type": "row", "cells": [ { "_key": "rp00331", "_type": "richTableCell", "content": [ { "_key": "rp00333", "_type": "block", "children": [ { "_key": "rp00332", "_type": "span", "marks": [], "text": "Persistent workflow context" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00334", "_type": "richTableCell", "content": [ { "_key": "rp00336", "_type": "block", "children": [ { "_key": "rp00335", "_type": "span", "marks": [], "text": "Guardrails and incident-response agents can retain reusable inference state across turns, tools, sessions, and GPU hosts." } ], "markDefs": [], "style": "normal" } ] } ] } ] } ``` The result is an inference architecture where isolation and persistence are properties of the infrastructure, not requirements each application has to implement on its own. Shared GPU infrastructure can preserve warm-cache performance inside the correct project boundary while preventing KV state from crossing into another project, tenant, or workload. These properties only matter if they hold at production scale. Isolation that forces full recompute, sticky session routing, or per-replica cache duplication may be sound on paper but impractical to run. [Joint testing with Oracle Cloud Infrastructure](https://blogs.oracle.com/ai-and-datascience/scaling-long-context-inference-on-oci-with-wekas-augmented-memory-grid) speaks to that directly: on OCI bare-metal H100 infrastructure at 72-GPU cluster scale, Augmented Memory Grid supported 10x more concurrent users before saturation, delivered 10x higher token throughput, and served 7x more total tokens in the same test window than a DRAM-only baseline. The point for this architecture is not the economics on their own; it is that project-scoped isolation and persistent KV memory stay practical on shared cloud GPU infrastructure, without giving up the performance that makes cache reuse worth doing. ## Partner Architecture Spotlight: Validating Project-Scoped Isolation [Embedded LLM](https://embeddedllm.com/) is building a federated AI utility powered by vLLM, designed to deliver fast, interoperable inference across hardware platforms while aligning operators, contributors, and partners around a shared ecosystem. For this project, Embedded LLM designed and tested the project-scoped KV-cache isolation architecture validated in this paper, with TokenVisor providing the governance layer for request authentication, project routing, RBAC, metering, and cache-salt injection. The project-scoped KV-cache isolation architecture was built around a specific question: can an inference platform preserve the latency benefit of cache reuse within a project while preventing that cached state from being detected or reused by another project? The architecture uses a shared vLLM instance with a single copy of the model weights in GPU HBM. Embedded LLM’s TokenVisor authenticates each request, maps it to the correct organization and project, enforces RBAC and metering, and injects a per-project cache_salt. LMCache derives project-scoped cache keys from the prompt tokens and that salt. Augmented Memory Grid persists the resulting KV blocks on NeuralMesh’s Token Warehouse, making them available across authorized replicas while keeping reuse scoped to the project that created them. The project validates two properties of the architecture: KV state remains isolated across project boundaries, and authorized cache reuse within a project retains its latency benefit. ### How to Interpret the Test The validation uses time-to-first-token, or TTFT, as the signal for whether KV cache was reused. Uncached TTFT, also called the cold path, means the runtime did not find a matching KV cache and had to compute it during prefill. Cached TTFT, also called the warm path, means the runtime found a matching project-scoped KV cache and could avoid or reduce repeated prefill work. For isolation to hold, the behavior should be asymmetric. A project reusing its own previously created cache should receive the warm path. A different project submitting the same prompt should receive the cold path, because it is not authorized to reuse the first project’s KV state. ![TokenVisor routes multi-tenant inference through Augmented Memory Grid while keeping each project’s KV cache isolated via per-project cache_salt.](https://cdn.sanity.io/images/ult5g8gw/production/5bfa77f3aa300e9c38fef538ffe40c61a766e3c0-924x534.png) Figure 5: TokenVisor routes multi-tenant inference through WEKA's Augmented Memory Grid while keeping each project's KV cache isolated via per-project cache_salt. Same-project reuse delivers 9.2–20.1x faster time-to-first-token; cross-project requests force full recomputation — 0.94–1.00x, confirming zero cache leakage between tenants. ### Validation Results Two projects were provisioned on the same shared vLLM instance with a single copy of the model weights in GPU HBM. Project 1 submitted a 42,000-token prompt, creating and warming its project-scoped KV cache. Project 2 then submitted the same prompt under a different bearer token. The prompt, model, and serving infrastructure were the same. Only the authenticated project identity changed. If isolation failed, Project 2 would have reused Project 1’s KV cache and shown cached TTFT. Instead, Project 2 matched the uncached TTFT baseline, showing that KV state was not reused across the project boundary. Project 1 still received the warm-path benefit when reusing its own cache, demonstrating that isolation did not disable authorized reuse. ![Embedded LLM April 2026 validation chart: cross-project TTFT matched the cold baseline, while within-project warm performance reached 9–20× speedup.](https://cdn.sanity.io/images/ult5g8gw/production/c2f472cb806e18752ce27014cf0f3dc5827e519a-924x432.png) Figure 6. Embedded LLM April 2026 validation: cross-project TTFT matched cold baseline with zero exceptions (isolation confirmed), while within-project warm performance reached 9–20× speedup depending on model — with no full-stack duplication required. ## Secure Stateful Inference in Practice The following use cases show both sides of secure stateful inference: protecting KV cache from crossing trust boundaries, and preserving project-scoped inference memory so security AI can reason across long-running workflows. ### Use Case #1: Shared Infrastructure, Isolated Inference State Shared GPU infrastructure can improve utilization, but shared compute should not imply shared inference state. AI clouds, sovereign AI platforms, and large enterprises need to serve multiple teams, agencies, or customers from the same model-serving infrastructure while keeping each project’s KV state inside its own trust boundary. The architecture separates two concerns that are often conflated: - **Model sharing: **Multiple projects use the same model instance and GPU-serving pool. - **Cache reuse: **KV state is reused only within the authenticated project that created it. A governance layer binds each request to project identity. The KV-cache manager incorporates that identity into cache-key derivation, and Augmented Memory Grid persists the resulting KV blocks on NeuralMesh. Any authorized replica can retrieve the project’s cache, but an identical prompt submitted under another project resolves to a different cache key. Embedded LLM validated this pattern using two projects on one shared vLLM instance with a single copy of the model weights in GPU HBM. Project 1 submitted a 42,000-token prompt to create its project-scoped KV cache. Project 2 then submitted the same prompt under a different bearer token. If cross-project reuse were possible, Project 2 would have shown cached TTFT. Instead, its TTFT matched the uncached baseline, showing that it did not reuse Project 1’s KV state. Project 1 still received the cached-TTFT benefit when reusing its own cache. This result shows that model sharing and cache isolation can coexist without duplicating the serving stack for every tenant. Separate WEKA performance testing measured 308 GB/s of direct Token Warehouse-to-GPU throughput per host, demonstrating that persisted KV blocks can remain available over a high-throughput data path. The same persistent-memory pattern also holds at cloud scale: in the joint OCI testing described above, it sustained large, highly concurrent production workloads on a 72-GPU H100 cluster against a DRAM-only baseline. Isolation, in other words, does not have to be traded against scale. Project-scoped cache keys directly address attack patterns that depend on cross-tenant cache matches or observable hit-and-miss behavior, including timing-based prompt and input inference. Other KV-cache risks, such as exposed-tensor inversion or injection, require additional storage, runtime, and tensor-protection controls. Together, these layers provide the defense-in-depth model needed for sovereign and regulated deployments. The following controls summarize the highest-priority requirements for a production multi-tenant implementation: ```json { "_key": "rp00435", "_type": "richTableBlock", "headerRows": 1, "markDefs": null, "rows": [ { "_key": "rp00399", "_type": "row", "cells": [ { "_key": "rp00393", "_type": "richTableCell", "content": [ { "_key": "rp00395", "_type": "block", "children": [ { "_key": "rp00394", "_type": "span", "marks": [], "text": "Requirement" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00396", "_type": "richTableCell", "content": [ { "_key": "rp00398", "_type": "block", "children": [ { "_key": "rp00397", "_type": "span", "marks": [], "text": "Mechanism" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00406", "_type": "row", "cells": [ { "_key": "rp00400", "_type": "richTableCell", "content": [ { "_key": "rp00402", "_type": "block", "children": [ { "_key": "rp00401", "_type": "span", "marks": [], "text": "Prevent cross-project KV reuse" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00403", "_type": "richTableCell", "content": [ { "_key": "rp00405", "_type": "block", "children": [ { "_key": "rp00404", "_type": "span", "marks": [], "text": "Derive cache keys from request tokens plus a project-specific cache_salt bound to authenticated identity." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00413", "_type": "row", "cells": [ { "_key": "rp00407", "_type": "richTableCell", "content": [ { "_key": "rp00409", "_type": "block", "children": [ { "_key": "rp00408", "_type": "span", "marks": [], "text": "Enforce storage-layer boundaries" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00410", "_type": "richTableCell", "content": [ { "_key": "rp00412", "_type": "block", "children": [ { "_key": "rp00411", "_type": "span", "marks": [], "text": "NeuralMesh applies organization-level isolation, filesystem ACLs, CIDR-based network access policies, and encryption to persisted KV data." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00420", "_type": "row", "cells": [ { "_key": "rp00414", "_type": "richTableCell", "content": [ { "_key": "rp00416", "_type": "block", "children": [ { "_key": "rp00415", "_type": "span", "marks": [], "text": "Preserve authorized reuse within a project" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00417", "_type": "richTableCell", "content": [ { "_key": "rp00419", "_type": "block", "children": [ { "_key": "rp00418", "_type": "span", "marks": [], "text": "The same token context and project identity produce the same cache key across authorized requests and replicas." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00427", "_type": "row", "cells": [ { "_key": "rp00421", "_type": "richTableCell", "content": [ { "_key": "rp00423", "_type": "block", "children": [ { "_key": "rp00422", "_type": "span", "marks": [], "text": "Make project-scoped KV available across replicas" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00424", "_type": "richTableCell", "content": [ { "_key": "rp00426", "_type": "block", "children": [ { "_key": "rp00425", "_type": "span", "marks": [], "text": "Augmented Memory Grid persists reusable KV blocks in the Token Warehouse, allowing any authorized replica to retrieve them without full per-replica duplication or sticky routing." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00434", "_type": "row", "cells": [ { "_key": "rp00428", "_type": "richTableCell", "content": [ { "_key": "rp00430", "_type": "block", "children": [ { "_key": "rp00429", "_type": "span", "marks": [], "text": "Keep active inference fast" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00431", "_type": "richTableCell", "content": [ { "_key": "rp00433", "_type": "block", "children": [ { "_key": "rp00432", "_type": "span", "marks": [], "text": "GPU HBM holds the active working set while Augmented Memory Grid retrieves persisted KV over RDMA." } ], "markDefs": [], "style": "normal" } ] } ] } ] } ``` ### Use Case #2: Persistent Context for Supply Chain Guardrails Software supply chain attacks exploit the time between a trusted artifact entering a development workflow and defenders recognizing it as malicious. A poisoned package can execute lifecycle scripts, access credentials, modify build artifacts, or establish persistence through an ordinary install or release process before a scanner or analyst has correlated the full sequence of events. March 2026 provided two examples. [Malicious LiteLLM releases appeared on PyPI](https://docs.litellm.ai/blog/security-update-march-2026) in an incident researchers linked to a broader campaign involving compromised security tooling. The payload harvested credentials, encrypted exfiltrated data, attempted lateral movement across Kubernetes clusters, and installed persistent malware. Days later, [attackers used a compromised Axios maintainer](https://github.com/axios/axios/issues/10636) account to publish poisoned npm versions containing a malicious dependency that installed a cross-platform remote access trojan through a standard package installation. Both incidents used trusted software distribution paths to reach developer and build environments. An AI guardrail can help narrow this detection window only if it can retain and correlate evidence over time. A package update may not look malicious in isolation. The risk becomes clearer when the system compares it with a known-good release, a recent CI/CD change, unusual publisher activity, install-time execution, or subsequent runtime behavior. ![Comparison of a stateless inference call that evaluates only the current event with a stateful guardrail backed by Augmented Memory Grid that compares the event against accumulated context.](https://cdn.sanity.io/images/ult5g8gw/production/28a063d17472771ebda1738e905badecbe043835-924x513.png) Figure 7. A stateless inference call evaluates only the current event. A stateful guardrail backed by Augmented Memory Grid compares the current event against accumulated context — package history, dependency changes, behavioural baselines — enabling pattern detection that a single prompt cannot provide. ### Context the guardrail needs to retain ```json { "_key": "rp00495", "_type": "richTableBlock", "headerRows": 1, "markDefs": null, "rows": [ { "_key": "rp00459", "_type": "row", "cells": [ { "_key": "rp00453", "_type": "richTableCell", "content": [ { "_key": "rp00455", "_type": "block", "children": [ { "_key": "rp00454", "_type": "span", "marks": [], "text": "Signal" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00456", "_type": "richTableCell", "content": [ { "_key": "rp00458", "_type": "block", "children": [ { "_key": "rp00457", "_type": "span", "marks": [], "text": "Why It Matters" } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00466", "_type": "row", "cells": [ { "_key": "rp00460", "_type": "richTableCell", "content": [ { "_key": "rp00462", "_type": "block", "children": [ { "_key": "rp00461", "_type": "span", "marks": [], "text": "Package and dependency baseline" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00463", "_type": "richTableCell", "content": [ { "_key": "rp00465", "_type": "block", "children": [ { "_key": "rp00464", "_type": "span", "marks": [], "text": "Compares current files, manifests, hashes, and dependency graph with the last trusted release." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00473", "_type": "row", "cells": [ { "_key": "rp00467", "_type": "richTableCell", "content": [ { "_key": "rp00469", "_type": "block", "children": [ { "_key": "rp00468", "_type": "span", "marks": [], "text": "CI/CD workflow edits" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00470", "_type": "richTableCell", "content": [ { "_key": "rp00472", "_type": "block", "children": [ { "_key": "rp00471", "_type": "span", "marks": [], "text": "Identifies modified workflows, new actions, altered build paths, or changes in the release pipeline." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00480", "_type": "row", "cells": [ { "_key": "rp00474", "_type": "richTableCell", "content": [ { "_key": "rp00476", "_type": "block", "children": [ { "_key": "rp00475", "_type": "span", "marks": [], "text": "Maintainer and token activity" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00477", "_type": "richTableCell", "content": [ { "_key": "rp00479", "_type": "block", "children": [ { "_key": "rp00478", "_type": "span", "marks": [], "text": "Detects unusual publisher behavior, credential misuse, unexpected token activity, or possible account takeover." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00487", "_type": "row", "cells": [ { "_key": "rp00481", "_type": "richTableCell", "content": [ { "_key": "rp00483", "_type": "block", "children": [ { "_key": "rp00482", "_type": "span", "marks": [], "text": "Install-time and runtime behavior" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00484", "_type": "richTableCell", "content": [ { "_key": "rp00486", "_type": "block", "children": [ { "_key": "rp00485", "_type": "span", "marks": [], "text": "Correlates lifecycle scripts, downloads, child processes, network activity, credential access, and persistence attempts." } ], "markDefs": [], "style": "normal" } ] } ] }, { "_key": "rp00494", "_type": "row", "cells": [ { "_key": "rp00488", "_type": "richTableCell", "content": [ { "_key": "rp00490", "_type": "block", "children": [ { "_key": "rp00489", "_type": "span", "marks": [], "text": "Prior findings and threat intelligence" } ], "markDefs": [], "style": "normal" } ] }, { "_key": "rp00491", "_type": "richTableCell", "content": [ { "_key": "rp00493", "_type": "block", "children": [ { "_key": "rp00492", "_type": "span", "marks": [], "text": "Connects current behavior with earlier investigations, indicators of compromise, and known attacker techniques." } ], "markDefs": [], "style": "normal" } ] } ] } ] } ``` Package scanners, CI/CD platforms, endpoint tools, SIEMs, and threat-intelligence systems provide the evidence the guardrail evaluates. The guardrail uses that evidence to establish baselines, identify changes, and connect activity across the investigation. Augmented Memory Grid supports the workflow at the inference-memory layer by persisting the project-scoped KV state created during that analysis. As new evidence arrives, the guardrail can continue from its prior context across sessions and GPU hosts rather than rebuilding the full inference state at every step. ### Use Case: Incident Response Agents That Carry Context Forward Security operations and incident response are inherently iterative. An AI agent may begin with an alert, inspect logs, query threat intelligence, review code, compare prior incidents, estimate blast radius, and recommend response actions. Each step adds evidence and intermediate conclusions that later steps may depend on. Without persistent inference memory, the agent may need to rebuild its working context whenever cache is evicted, a session resumes, or another GPU host serves the next request. Reprocessing the investigation history through prefill increases latency, consumes additional GPU cycles, and becomes less practical as the investigation expands. With Augmented Memory Grid, the project-scoped KV state created as the agent reasons over the investigation can persist across turns, sessions, and GPU hosts. Augmented Memory Grid keeps the model’s reusable inference state available so the agent can continue from prior analysis rather than rebuilding it at every step. The agent can carry forward working context such as: - Alert history and triage notes - Log summaries and correlated events - Related prior incidents and resolution patterns - Tool outputs from sandboxing, code analysis, or threat intel queries - Response actions taken already and their outcomes As new evidence arrives, the inference system can retrieve the existing KV state and extend the investigation from that point. This reduces repeated prefill, allows an authorized replica to continue the workflow, and preserves continuity across a long-running response process. The application still determines which evidence belongs in the investigation and how context is organized; Augmented Memory Grid provides the persistent memory tier that keeps the resulting inference state available. ## Closing The security boundary for AI systems now extends into inference memory. Sensitive prompts, documents, tool outputs, and reasoning context are materialized as runtime state while a model generates output. Published attacks and vulnerabilities show why that state must remain inside the trust boundary that created it. The architecture described in this paper provides five important capabilities: - **Project-scoped cache reuse. **Authenticated project identity is bound into cache-key derivation. The same project can reuse its own KV state, while requests from another project resolve to different cache keys. - **Persistent inference memory. **Reusable KV state remains available across sessions and authorized GPU replicas, allowing long-running guardrails and agents to continue from prior inference context instead of repeatedly rebuilding unchanged context through prefill. - **Lower HBM pressure. **GPU HBM holds the active working set, while reusable KV blocks move to a larger, NVMe-backed Token Warehouse that can extend effective KV-cache capacity by up to 1,000×. - **Replica flexibility. **Any authorized replica can retrieve the correct project-scoped KV state from the shared Token Warehouse, reducing the need for full per-replica cache copies or sticky session routing. - **Storage-layer reinforcement. **NeuralMesh can reinforce the project boundary through organization-level logical isolation, filesystem ACLs, CIDR-based network access policies, and encryption and key-management controls beneath the inference stack. Augmented Memory Grid provides the persistent memory layer for this architecture. Paired with project-aware governance and scoped cache-key derivation, it preserves the performance benefits of KV-cache reuse without making cached state globally available. For shared inference, that means projects can use common GPU and model-serving infrastructure while keeping KV state inside the correct trust boundary. For security AI, it means guardrails can reuse accumulated inference context across sessions and GPU hosts rather than starting again from a single prompt. Secure stateful inference is not the result of one control or one product. Governance establishes trusted identity, the inference layer controls cache lookup and reuse, and the persistent memory layer preserves those boundaries as state moves across the infrastructure. For sovereign and regulated AI, those protections have to be designed into the architecture from the start. Ready to learn more about WEKA Augmented Memory Grid? Visit [weka.io/contact](https://www.weka.io/contact) to connect with a WEKA solutions architect. [Download PDF](/api/resource-pdf?slug=your-kv-cache-is-now-part-of-the-attack-surface)