# The Inference Economics Scorecard

A Buyer's Framework for Evaluating Production AI Infrastructure

**Author:** Steve McDowell

**Published:** November 16, 2026

**Updated:** November 16, 2026

**Type:** Analyst Report

**Publisher:** NAND Research

## The Inference Divide

AI spending is shifting from one-time model training to continuous production inference, which can account for [80–90% of the lifetime compute cost](https://www.mckinsey.com/industries/semiconductors/our-insights/frontiers-of-compute-the-technologies-to-reduce-ai-inference-costs) of a production AI system as usage scales.

Agentic workloads are accelerating this shift. Because they involve loops with multiple reasoning steps, tool calls, and retries, agentic workflows can consume 5 to 30 times more tokens per task than a typical chatbot exchange. Per-token prices may be falling, but overall usage is rising faster. More agents doing more token-intensive work means total inference costs are not only increasing but also harder for organizations to predict. The economic advantage goes to architectures that waste the least capacity per useful result.

This changes how IT organizations should evaluate AI infrastructure. While training clusters are optimized to complete large batch jobs quickly, inference fleets are different. Inference-focused infrastructure must continuously adapt to new model architectures and agentic workflows while delivering consistent quality and latency under unpredictable, bursty, concurrent, always-on demand, all within power, space, and cost constraints.

The difference is architectural. Purpose-built inference systems prioritize memory efficiency, dynamic data movement, and multi-tenant isolation. By contrast, training-oriented or traditional storage platforms often make design assumptions that do not translate into value in production inference.

Repurposing them is also not economically viable at the enterprise scale: the mismatch between their design assumptions and inference’s demands compounds into a persistent cost penalty as deployments scale. Serving production inference economically requires infrastructure purpose-built for the workload, not infrastructure inherited from training.

This distinction also changes how infrastructure performance should be measured. During inference, GPUs can appear fully utilized even when performing wasteful work, such as recomputing dropped context, retrying failed calls, or generating tokens that are ultimately discarded.

GPU utilization is a reasonable measure of efficiency during training, when the job is a known, fixed computation run to completion, but it is a poor measure of inference productivity, where busy hardware and useful output are no longer the same thing.


![ Figure 1:  A utilization dashboard reports a busy fleet, but decomposing the same GPU-hours by inference stage tells a different story. Each eviction forces context to be recomputed before decoding can resume, and this prefill tax grows with session count and context length. Profile is illustrative.](https://cdn.sanity.io/images/ult5g8gw/production/5eb7d6095b58df97f1f0bb9dae9e4092aec7ea3c-1080x608.png)


Every hatched block is billed GPU time that serves no new tokens. Profile is illustrative; block widths are not measured data.

The same logic applies to raw token throughput: tokens lost to recomputation, retries, stale context, failed requests, or low-quality responses constitute overhead rather than useful output. What matters is the cost per successful request under real-world concurrency, not how busy the GPUs appear.

Because physical architecture, system integration, supply-chain control, and rack density are difficult to change after deployment, these elements shape inference economics for years. Software can be replaced quickly; infrastructure cannot. Getting the architecture right from the start is critical.

## New Constraints on AI Infrastructure

Power, cooling, rack density, component availability, and lead times are converging into near-term deployment constraints. For most current AI buildouts, capital was the limiting factor, and whoever could raise the most money could buy the most GPUs. That is no longer the binding constraint.

In the inference economy, every rack unit, watt, and GPU must deliver more useful work because buyers cannot assume that additional capacity will be available when demand spikes. A design that requires twice the rack footprint or power draw to meet a given throughput target is harder to site, harder to permit, and slower to bring online.

While GPU supply remains constrained, the more acute near-term bottleneck has shifted to memory, operating at two levels. At the market level, high-bandwidth memory (HBM) now competes directly with GPU compute for advanced packaging and wafer capacity, as memory-intensive frontier models and AI workloads are consuming an increasing share of total DRAM and NAND production.


![Figure 2:  The binding constraint on AI buildout has shifted from capital to GPU supply to power, memory, and footprint. Demand does not stop shifting there. Agentic AI is only the first wave; physical AI, world models, and sovereign inference compound demand against the same fixed budget of watts and rack units.](https://cdn.sanity.io/images/ult5g8gw/production/6c89f9a3557f6785f5b544f9c65b1a3636c1b876-1074x604.png)


The system that extracts the most useful work from a fixed power and footprint budget wins the buying decision. Each new wave of AI raises that bar.

This dynamic is [expected to persist](https://www.trendforce.com/presscenter/news/20260602-13074.html) for the foreseeable future. At the system level, memory is also a primary constraint on token economics. Agentic inference workloads rapidly exhaust the GPU’s limited HBM, hitting the ‘[AI memory wa](/article/how-ai-s-memory-wall-is-reshaping-infrastructure-strategy-beyond-gpus)ll’ and forcing systems to repeatedly evict the KV cache and recompute tokens.

That means paying to rebuild the context an agent generated moments earlier. This wasteful recomputation tax compounds across workloads, eroding inference margins. The global supply shortage determines memory costs; the AI memory wall determines whether inference is cost-efficient.

Infrastructure plans must account for both operating economics and procurement risk because a component shortage can delay an expansion as effectively as a lack of power or capital.

## The Inference Economics Scorecard

The goal is now the **`/goal`** — a very popular command that spawns many agent swarms in production today, for more than just coding workflows. An increasing share of knowledge work and cybersecurity now also depends on agent swarms.

The industry operates in an environment where _infrastructure efficiency_, not peak performance alone, sets the ceiling on AI's return on inference workloads. When power, space, and components are scarce, the system that extracts the most useful work from a fixed power and footprint budget wins the buying decision.

This reframing is why a scorecard built around useful work per watt, per rack, and per dollar matters more now than it did during the training-cluster buildout of the past several years.

### Stark differences between training and inference when placed side by side:

```json
{
  "_key": "ntbl0",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "n0r0",
      "_type": "row",
      "cells": [
        {
          "_key": "n0r0c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r0c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r0c0s",
                  "_type": "span",
                  "text": "Dimension"
                }
              ],
              "markDefs": [],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r0c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r0c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r0c1s",
                  "_type": "span",
                  "text": "Training"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r0c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r0c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r0c2s",
                  "_type": "span",
                  "text": "Inference"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n0r1",
      "_type": "row",
      "cells": [
        {
          "_key": "n0r1c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r1c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r1c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Primary workload"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r1c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r1c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r1c1s",
                  "_type": "span",
                  "text": "Large batch jobs run to completion"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r1c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r1c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r1c2s",
                  "_type": "span",
                  "text": "Continuous, always-on serving"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n0r2",
      "_type": "row",
      "cells": [
        {
          "_key": "n0r2c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r2c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r2c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "I/O profile"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r2c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r2c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r2c1s",
                  "_type": "span",
                  "text": "Write-heavy: checkpoints, gradients, and optimizer state written in large sequential batches"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r2c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r2c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r2c2s",
                  "_type": "span",
                  "text": "Read-heavy: high-IOPS, low-latency, often random reads of weights, KV caches, and embeddings"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n0r3",
      "_type": "row",
      "cells": [
        {
          "_key": "n0r3c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r3c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r3c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Governing constraint"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r3c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r3c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r3c1s",
                  "_type": "span",
                  "text": "Throughput against a hard synchronization deadline — every worker must converge before the job advances"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r3c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r3c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r3c2s",
                  "_type": "span",
                  "text": "Latency and concurrency against a service-level response time"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n0r4",
      "_type": "row",
      "cells": [
        {
          "_key": "n0r4c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r4c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r4c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Data movement"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r4c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r4c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r4c1s",
                  "_type": "span",
                  "text": "Centralized and predictable, staged around scheduled checkpoint intervals"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r4c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r4c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r4c2s",
                  "_type": "span",
                  "text": "Distributed and replicated across the fleet, bursty and hard to predict"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n0r5",
      "_type": "row",
      "cells": [
        {
          "_key": "n0r5c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r5c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r5c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Footprint"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r5c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r5c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r5c1s",
                  "_type": "span",
                  "text": "Concentrated in large, purpose-built training clusters"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r5c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r5c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r5c2s",
                  "_type": "span",
                  "text": "Spread across many inference nodes, often geographically distributed"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n0r6",
      "_type": "row",
      "cells": [
        {
          "_key": "n0r6c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r6c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r6c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Cost of a delay"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r6c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r6c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r6c1s",
                  "_type": "span",
                  "text": "A slower job costs time"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r6c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r6c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r6c2s",
                  "_type": "span",
                  "text": "A slow or failed response costs revenue and user trust, in real time"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n0r7",
      "_type": "row",
      "cells": [
        {
          "_key": "n0r7c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r7c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r7c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Optimization target"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r7c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r7c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r7c1s",
                  "_type": "span",
                  "text": "Peak throughput / time-to-train"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n0r7c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n0r7c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n0r7c2s",
                  "_type": "span",
                  "text": "Useful work per watt, per rack, and per dollar"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

This divide runs through the data pipeline as much as through compute. Training is fundamentally a write-heavy throughput problem with a hard synchronization deadline: gradients, optimizer state, and checkpoints must be written and synchronized across the cluster before the job can advance to the next step, and nothing is complete until every worker converges.

Inference inverts this. It is a latency- and IOPS-intensive problem, driven by rapid, high-concurrency reads of weights, KV caches, and embeddings. A data pipeline architecture tuned for training’s write-heavy, synchronized batches is solving the wrong problem for inference’s read-heavy, distributed, latency-bound reality.


![Figure 3:  Training and inference diverge from the same infrastructure decision. Training runs large synchronized batches to completion; inference serves continuous, bursty, concurrent demand with no endpoint. An architecture tuned for one solves the wrong problem for the other.](https://cdn.sanity.io/images/ult5g8gw/production/5ed6be6a34c1467ac7c040dd9784fd94ff9c5c32-1080x608.png)


A data pipeline tuned for training solves the wrong problem for inference.

Put simply, understanding the infrastructure for AI training is relatively straightforward, with raw IOPS and peak throughput paramount. Inference, on the other hand, requires production metrics tied to useful work, concurrency, failures, power, space, and fully loaded cost.

## The Scorecard

The Inference Economics Scorecard is a buyer's framework for evaluating production AI infrastructure on useful work per watt, per rack, and per dollar rather than on peak throughput or GPU utilization, which can inflate the appearance of productivity whether or not the work is useful. The scorecard’s  nine metrics give buyers a basis for comparing vendors under real load.


![Figure 4:  The Inference Economics Scorecard. Nine measures of useful work per watt, per rack, and per dollar, grouped by what they protect: efficiency, experience, and scale. Table 1 details the calculation and minimum disclosure for each.](https://cdn.sanity.io/images/ult5g8gw/production/1c1967d550b2c83c0d135a66a3cb1bea1dc73890-3840x2160.png)


```json
{
  "_key": "ntbl1",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "n1r0",
      "_type": "row",
      "cells": [
        {
          "_key": "n1r0c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r0c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r0c0s",
                  "_type": "span",
                  "text": "Metric"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r0c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r0c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r0c1s",
                  "_type": "span",
                  "text": "Calculation"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r0c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r0c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r0c2s",
                  "_type": "span",
                  "text": "Minimum disclosure"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n1r1",
      "_type": "row",
      "cells": [
        {
          "_key": "n1r1c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r1c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r1c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Tokens per watt"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r1c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r1c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r1c1s",
                  "_type": "span",
                  "text": "Useful output tokens meeting the quality and service-level target / total energy consumed during the test (Wh)."
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r1c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r1c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r1c2s",
                  "_type": "span",
                  "text": "Model and precision; input/output mix; concurrency; quality rule; p50/p95/p99 latency; test duration; GPU, storage, network, and cooling boundary; PUE if used."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n1r2",
      "_type": "row",
      "cells": [
        {
          "_key": "n1r2c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r2c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r2c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Tokens per rack"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r2c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r2c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r2c1s",
                  "_type": "span",
                  "text": "Useful output tokens meeting the target / occupied rack-equivalents during the test."
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r2c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r2c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r2c2s",
                  "_type": "span",
                  "text": "Rack units occupied by the full serving stack; rack power and thermal limits; equipment included; workload controls; sustained rather than peak result."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n1r3",
      "_type": "row",
      "cells": [
        {
          "_key": "n1r3c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r3c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r3c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Cost per inference"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r3c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r3c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r3c1s",
                  "_type": "span",
                  "text": "Annualized infrastructure, energy, facilities, software and operations cost / successful requests served."
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r3c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r3c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r3c2s",
                  "_type": "span",
                  "text": "Utilization; amortization period; energy price; staffing and license assumptions; failed requests and retries; model, request size, and service-level target."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n1r4",
      "_type": "row",
      "cells": [
        {
          "_key": "n1r4c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r4c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r4c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "GPU Goodput"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r4c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r4c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r4c1s",
                  "_type": "span",
                  "text": "Successful requests or useful output tokens meeting quality and latency targets / GPU-time consumed."
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r4c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r4c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r4c2s",
                  "_type": "span",
                  "text": "GPU model and count; model and precision; batching; concurrency; cache state; failure/retry rate; quality and latency thresholds."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n1r5",
      "_type": "row",
      "cells": [
        {
          "_key": "n1r5c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r5c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r5c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Time to first token and end-to-end latency"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r5c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r5c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r5c1s",
                  "_type": "span",
                  "text": "Report p50, p95 and p99 time to first token and request completion latency at the target concurrency."
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r5c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r5c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r5c2s",
                  "_type": "span",
                  "text": "Input and output token counts; queueing; model; concurrency; cache-hit rate; streaming method; warm/cold state; timeout and failure treatment."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n1r6",
      "_type": "row",
      "cells": [
        {
          "_key": "n1r6c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r6c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r6c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Memory efficiency"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r6c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r6c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r6c1s",
                  "_type": "span",
                  "text": "Active context and Key-Value (KV) cache capacity delivered per GPU, relative to onboard HBM, without breaching the quality or latency target."
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r6c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r6c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r6c2s",
                  "_type": "span",
                  "text": "Model; context length; precision; concurrent sessions; cache-hit rate; persistence tier; recomputation; latency impact; recovery behavior."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n1r7",
      "_type": "row",
      "cells": [
        {
          "_key": "n1r7c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r7c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r7c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Operational density"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r7c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r7c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r7c1s",
                  "_type": "span",
                  "text": "Usable capacity and sustained performance per rack under declared power, thermal, protection and failure-state conditions."
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r7c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r7c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r7c2s",
                  "_type": "span",
                  "text": "Raw and effective capacity; data-reduction assumption; rack units; power; temperature; protection level; rebuild/failure state; sustained throughput."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n1r8",
      "_type": "row",
      "cells": [
        {
          "_key": "n1r8c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r8c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r8c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Agent concurrency"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r8c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r8c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r8c1s",
                  "_type": "span",
                  "text": "Maximum simultaneous agents, sessions, or subtasks sustained before a declared latency, quality, or failure-rate threshold is crossed."
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r8c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r8c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r8c2s",
                  "_type": "span",
                  "text": "Agent topology; turns; tool calls; context length; model routing; shared-cache behavior; arrival pattern; latency, quality, and error thresholds."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n1r9",
      "_type": "row",
      "cells": [
        {
          "_key": "n1r9c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r9c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r9c0s",
                  "_type": "span",
                  "marks": [
                    "strong"
                  ],
                  "text": "Context retention"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r9c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r9c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r9c1s",
                  "_type": "span",
                  "text": "Long-horizon agents require low latency access to attention in KV cache for hours and days until completion of the popular /goal command"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n1r9c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n1r9c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n1r9c2s",
                  "_type": "span",
                  "text": "End-end time scale; token throughput and latency differences across as many relevant intervals as possible; KV cache hit rates."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

To most effectively use the scorecard, adhere to the following guidelines:

- **Workload controls**: Use the same model, precision, request mix, concurrency target, cache state, quality threshold, and service-level objective across all systems.
- **Measurement boundary**: Measure the entire serving stack and state whether cooling and facility overhead are included.
- **Test duration**: Run to steady state; report p50, p95, and p99 results, along with failures, retries, and discarded output. This should be at least 8 hours, extending to multiple days where possible.
- **Useful output**: Count only successful output that meets both the quality and service-level targets; label modeled estimates separately from measured results.

## How to Use It: Questions Buyers Should Be Asking

Applying the scorecard to a vendor evaluation means asking questions that the vendor's marketing materials rarely answer directly, such as:

- How can I maximize my effective KV cache hit rate to match the logical cacheability of my users’ agents and chats?
- How can I maximize my tokens per watt, and per GPU without sacrificing user SLOs such as token latency and throughput?
- How much effective capacity can I get per rack unit, and what power, footprint, protection, and thermal conditions does that figure assume?
- Who designed the hardware my software runs on, and what does that imply for thermal behavior, density ceiling, and supply chain control?
- If NAND or memory prices change significantly, how would that affect the cost of my next expansion?
- How many copies of my data does this platform require my inference pipeline to maintain across training, fine-tuning, and serving?
- What enables my agent swarms to complete more subtasks in parallel instead of serializing work that could run concurrently?
- Can this platform isolate workloads at both the hardware and network levels on the same system, or do I have to choose one and live with the trade-off?

Where Legacy Infrastructure Breaks Down

Most infrastructure marketed as AI-ready today traces back to one of three lineages:

- Enterprise storage built for transactional and backup workloads
- HPC architectures built for large, sequential training jobs
- General-purpose training-era designs assembled before inference became the dominant workload.

Organizations can no longer brute-force their way through an inference workload by simply adding more GPUs while still meeting the power, space, cost, and performance requirements that production AI now demands at scale.

Inference today requires a different approach to AI infrastructure, involving detailed systems co-design across compute, memory, networking, and storage.

## Common Architectural Limitations

Traditional AI architectures share many recurring limitations, regardless of lineage. Some examples include:

- Incorrect assumptions that HPC networks are still slower than PCI and DRAM channels on motherboards.
- GPU, CPU, and storage-controller locality assumptions are designed for static HPC network topologies rather than for the dynamic token and tensor movement patterns seen in inference workloads.
- Hardware co-design stops at compute and networking, where GPUs, NICs, and switches are engineered as a unit, while storage remains a separate, loosely coupled layer that systems must stitch back in with software. This leaves the storage path as the integration point where latency and consistency are hardest to control.
- Separate file and object infrastructures that require a translation gateway between them introduce latency and operational complexity.
- Multiple full copies of the same data are transferred across the training, fine-tuning, and serving stages, consuming capacity without generating output.
- Inefficient rack utilization due to general-purpose chassis not designed for sustained, high-density NVMe workloads.
- Dependence on OEM hardware channels limits how tightly software can be tuned to the underlying system’s thermal and density characteristics.


![Figure 5:  When context outgrows HBM on legacy architectures, each eviction cycle spends more compute per token served while p95 latency climbs past the SLO. Persistent KV cache breaks the cycle: retrieval replaces recomputation and latency holds steady as sessions grow. Latency curves are illustrative.](https://cdn.sanity.io/images/ult5g8gw/production/f249c6569fada89265b7abf8f39127e486036e7a-1078x607.png)




## WEKA's Approach: Purpose-Built, Supply-Chain-Controlled, and Dense




![WEKA AI Infrastructure vs. Legacy Storage Infrastructure technology stacks](https://cdn.sanity.io/images/ult5g8gw/production/2d67d4abaef2c420712635cd7efb81147955d2ef-1080x608.png)


### Purpose-Built for Agentic Inference

WEKA designed its WEKApod hardware and WEKA® NeuralMesh™ software as a single integrated system. This deliberate co-design is evident in four areas of the architecture:

- **GPU cluster-native design: **tensor locality, which keeps tensor movement close to the fastest available fabric rather than binding it to a single GPU server or controller, remains supported on NVIDIA ConnectX SuperNIC networking, Spectrum-X Ethernet, and a PCIe Gen 6 internal fabric.
- **Memory-centric design: **NeuralMesh Augmented Memory Grid™ persists the KV cache on WEKA-managed NVMe, extending effective GPU memory to up to an exabyte and delivering >10 TB/sec of effective HBM bandwidth.
- **Unified data architecture: **NeuralMesh’s native file-and-object protocol stack on NVMe makes the same physical data blocks addressable via S3 and POSIX simultaneously, eliminating duplicate data copies that conventional, legacy storage systems create during training, fine-tuning, and serving.
- **Operational simplicity: **Kubernetes Operator and NeuralMesh Observe, both included at no additional cost, complete the operational layer.

These architectural choices translate into concrete capabilities for production inference:

- **Multi-tenant isolation and QoS: **a single NeuralMesh cluster hosts many customers or models side by side, each with its own network isolation, security policies, and guaranteed performance tier, enabling AI clouds and enterprises to serve more tenants per GPU cluster while providing each with a real SLA rather than a best-effort promise.
- **Deterministic performance under concurrency: **NeuralMesh keeps response times steady as concurrent request volume rises, rather than optimizing throughput only under ideal conditions. For inference, consistency matters more than raw speed, as users notice slow or unpredictable responses far more than they notice average performance.
- **Instant snapshots and clones: **NeuralMesh creates an instant snapshot of a model or dataset at any size and turns it into a fully usable copy without duplicating the underlying data, making it fast and inexpensive to roll back a model version or run two versions side by side for testing.
- **Resilient by design: **storage is architected so that a hardware failure affects only a small slice of data, and recovery runs in parallel across many machines, keeping inference services running through routine hardware failures without the downtime or maintenance windows required by legacy storage systems.
- **Direct data path to the GPU: **NeuralMesh uses RDMA and NVIDIA GPUDirect Storage to move data directly from storage to GPU memory, bypassing the server’s operating system and CPU, reducing latency and overhead on every read, and resulting in faster token generation and better utilization of each GPU.

WEKA’s approach has delivered impressive real-world results. NeuralMesh with Augmented Memory Grid has delivered measurable gains, as shown by production benchmarks: WEKA and Oracle Cloud Infrastructure recently reported 10x higher token throughput, 10x more concurrent users, and 7x more tokens served per GPU on OCI H100 systems in production.

A second deployment with Firmus showed (https://www.weka.io/customers/firmus/) that the system delivered up to 6.5x more tokens per GPU and achieved a 4x speedup in time to first token.

Augmented Memory Grid also delivers a 4x to 20x improvement in time-to-first-token across validated deployments.

### Supply Chain Control

WEKApod is the first AI storage and memory appliance WEKA has designed and engineered from the chassis up. By delivering a turnkey, purpose-built solution, WEKA gains direct control over component sourcing and inventory for WEKApod.

This distinguishes the approach from competitors that rely on third-party hardware from whatever ODM or OEM solution is available at deployment, a critical differentiator amid supply-chain constraints:

- In a NAND market where prices and availability have been volatile, that control gives customers a degree of pricing and lead-time predictability that OEM-dependent vendors cannot match independently.
- WEKA's collaboration with Solidigm, including joint work through the Storage.AI initiative and engineering efforts to bring QLC flash economics closer to TLC performance, exemplifies the kind of component-level partnership that direct sourcing control enables.

### Consolidation and Density

WEKApod ships in three configurations based on three distinct storage technologies:

1. TLC flash prioritizes performance and endurance.
2. QLC flash prioritizes capacity and cost.
3. AlloyFlash combines them at different points on the performance-to-capacity curve.

These are detailed in the following table:

```json
{
  "_key": "ntbl2",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "n2r0",
      "_type": "row",
      "cells": [
        {
          "_key": "n2r0c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r0c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r0c0s",
                  "_type": "span",
                  "text": "SKU"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r0c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r0c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r0c1s",
                  "_type": "span",
                  "text": "Chassis"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r0c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r0c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r0c2s",
                  "_type": "span",
                  "text": "Drives"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r0c3",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r0c3b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r0c3s",
                  "_type": "span",
                  "text": "Performance"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r0c4",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r0c4b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r0c4s",
                  "_type": "span",
                  "text": "Optimized for"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n2r1",
      "_type": "row",
      "cells": [
        {
          "_key": "n2r1c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r1c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r1c0s",
                  "_type": "span",
                  "text": "WEKApod Nitro"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r1c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r1c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r1c1s",
                  "_type": "span",
                  "text": "2U, 4-node"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r1c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r1c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r1c2s",
                  "_type": "span",
                  "text": "56 TLC drives"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r1c3",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r1c3b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r1c3s",
                  "_type": "span",
                  "text": "800 Gb/s via dual-port NVIDIA ConnectX"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r1c4",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r1c4b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r1c4s",
                  "_type": "span",
                  "text": "Throughput-bound, performance-critical workloads"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n2r2",
      "_type": "row",
      "cells": [
        {
          "_key": "n2r2c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r2c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r2c0s",
                  "_type": "span",
                  "text": "WEKApod Prime"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r2c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r2c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r2c1s",
                  "_type": "span",
                  "text": "2U, 4-node"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r2c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r2c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r2c2s",
                  "_type": "span",
                  "text": "56 drives, AlloyFlash TLC/QLC mix"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r2c3",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r2c3b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r2c3s",
                  "_type": "span",
                  "text": "Balanced throughput and capacity"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r2c4",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r2c4b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r2c4s",
                  "_type": "span",
                  "text": "Balanced capacity and performance"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n2r3",
      "_type": "row",
      "cells": [
        {
          "_key": "n2r3c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r3c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r3c0s",
                  "_type": "span",
                  "text": "WEKApod Prime Max"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r3c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r3c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r3c1s",
                  "_type": "span",
                  "text": "2U, 2-node"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r3c2",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r3c2b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r3c2s",
                  "_type": "span",
                  "text": "70 NVMe drives, incl. Micron 245.76 TB 6600 ION SSDs"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r3c3",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r3c3b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r3c3s",
                  "_type": "span",
                  "text": "1.1 EB effective, 10.2 TB/s, 210M IOPS per 56U rack"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n2r3c4",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n2r3c4b",
              "_type": "block",
              "children": [
                {
                  "_key": "n2r3c4s",
                  "_type": "span",
                  "text": "Maximum capacity and density"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

Three software-driven capabilities extend the core capabilities:

1. AlloyFlash's automated placement of latency-sensitive writes on TLC and bulk data on QLC makes the highest-density configurations production-viable rather than merely capacity-on-paper.
2. NeuralMesh's native multi-tenancy includes Composable Clusters, which provide hardware-level isolation, and Virtual Multi-Tenancy, which provides network-level isolation through per-tenant VLANs and independent encryption keys. WEKA reports more than 1,000 logical tenants per cluster, with sub-30-minute provisioning, and up to 50,000 logically isolated tenants across the two tiers within a single physical cluster.
3. Configurable SKUs now scale WEKApod from under 1PB to more than 100PB in a single deployment, extending the platform’s reach to buyers who previously could not justify the highest-density tier.

### Scoring WEKApod and NeuralMesh

The value of WEKApod and NeuralMesh ultimately depends on how effectively these architectural choices translate into improved inference economics.

The following scorecard evaluates the platform against the metrics that increasingly matter for production-grade AI infrastructure:

- **Utilization**: how much useful work can be extracted from each GPU, per watt, and per rack.
- **Efficiency**: how efficiently memory and storage resources are used.
- **Responsiveness**: how quickly the system responds under load.
- **Agentic Readiness**: how well it supports the concurrency and isolation requirements of agentic AI.

Viewed through this lens, WEKA’s differentiation extends beyond storage performance and capacity density to the efficiency of the entire AI infrastructure stack.

```json
{
  "_key": "ntbl3",
  "_type": "richTableBlock",
  "headerRows": 1,
  "markDefs": null,
  "rows": [
    {
      "_key": "n3r0",
      "_type": "row",
      "cells": [
        {
          "_key": "n3r0c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r0c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r0c0s",
                  "_type": "span",
                  "text": "Metric"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n3r0c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r0c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r0c1s",
                  "_type": "span",
                  "text": "WEKA"
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n3r1",
      "_type": "row",
      "cells": [
        {
          "_key": "n3r1c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r1c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r1c0s",
                  "_type": "span",
                  "text": "Tokens per watt"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n3r1c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r1c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r1c1s",
                  "_type": "span",
                  "text": "WEKApod's 35°C-rated thermal design and AlloyFlash placement reduce power draw per unit of density, and Augmented Memory Grid reduces the GPU count needed per token."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n3r2",
      "_type": "row",
      "cells": [
        {
          "_key": "n3r2c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r2c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r2c0s",
                  "_type": "span",
                  "text": "Tokens per rack"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n3r2c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r2c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r2c1s",
                  "_type": "span",
                  "text": "WEKApod Prime Max: 1.1 exabytes effective capacity, 10.2 TB/s throughput, 210 million IOPS in a single 56U rack."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n3r3",
      "_type": "row",
      "cells": [
        {
          "_key": "n3r3c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r3c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r3c0s",
                  "_type": "span",
                  "text": "Cost per inference"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n3r3c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r3c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r3c1s",
                  "_type": "span",
                  "text": "Direct component sourcing plus always-on data reduction with a contractual performance guarantee; Augmented Memory Grid reduces the GPU footprint needed to serve a given token volume."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n3r4",
      "_type": "row",
      "cells": [
        {
          "_key": "n3r4c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r4c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r4c0s",
                  "_type": "span",
                  "text": "GPU goodput / concurrent users per GPU"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n3r4c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r4c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r4c1s",
                  "_type": "span",
                  "text": "10x more concurrent users per GPU. [Firmus](https://www.weka.io/customers/firmus/): up to 6.5x more tokens per GPU."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n3r5",
      "_type": "row",
      "cells": [
        {
          "_key": "n3r5c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r5c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r5c0s",
                  "_type": "span",
                  "text": "Time to first token / latency"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n3r5c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r5c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r5c1s",
                  "_type": "span",
                  "text": "4x faster time to first token. Augmented Memory Grid broadly: 4 to 20x time-to-first-token improvement across validated deployments."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n3r6",
      "_type": "row",
      "cells": [
        {
          "_key": "n3r6c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r6c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r6c0s",
                  "_type": "span",
                  "text": "Memory efficiency"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n3r6c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r6c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r6c1s",
                  "_type": "span",
                  "text": "Augmented Memory Grid extends effective GPU memory capacity by up to 1000x beyond onboard HBM by persisting KV cache to NeuralMesh-managed NVMe."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n3r7",
      "_type": "row",
      "cells": [
        {
          "_key": "n3r7c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r7c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r7c0s",
                  "_type": "span",
                  "text": "Operational density — Direct for storage"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n3r7c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r7c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r7c1s",
                  "_type": "span",
                  "text": "WEKApod Prime Max's 1.1-exabyte, 441.5-PB raw-capacity single rack, combined with AlloyFlash tiering that keeps the density production-viable rather than theoretical."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    },
    {
      "_key": "n3r8",
      "_type": "row",
      "cells": [
        {
          "_key": "n3r8c0",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r8c0b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r8c0s",
                  "_type": "span",
                  "text": "Agent swarm parallelism / subtask concurrency"
                }
              ],
              "style": "normal"
            }
          ]
        },
        {
          "_key": "n3r8c1",
          "_type": "richTableCell",
          "content": [
            {
              "_key": "n3r8c1b",
              "_type": "block",
              "children": [
                {
                  "_key": "n3r8c1s",
                  "_type": "span",
                  "text": "NeuralMesh's Composable Clusters and Virtual Multi-Tenancy, plus shared KV cache infrastructure that keeps context live across concurrent agents, users, and sessions without forcing redundant computation."
                }
              ],
              "style": "normal"
            }
          ]
        }
      ]
    }
  ]
}
```

## Implications to IT Buyers & AI Cloud Providers

For AI cloud providers and neoclouds, higher multi-tenancy density translates directly into more tenants, better GPU utilization, and more tokens served per GPU within the same physical footprint.

The inference proof points are solid. Oracle Cloud Infrastructure, Firmus, and Nebius (which selected NeuralMesh to power its next-generation AI cloud) are production references at the scale this buyer type operates at. CoreWeave also lists WEKA as one of the storage partners through which it offers customers dedicated clusters.

Enterprise buyers should evaluate any AI infrastructure architecture against a short checklist derived from the scorecard: does it measurably improve GPU utilization and productivity, reduce datacenter footprint, lower cost per inference, simplify operations, and scale to agentic workloads without requiring re-architecture? Is it keeping pace with radically different new models that may introduce new KV cache attention algorithms?

That last criterion is rapidly becoming the binding one. Agentic AI swarms create demands that most static infrastructure was not designed to handle:

1. High concurrency: tens to hundreds of subtasks running at once.
2. Thousands of turns per session.
3. Long-context windows in the range of 250,000 to 750,000 tokens per turn.
4. Persistent memory requirements that surge inference demand past what static GPU allocations were sized for.
5. Increased cadence of new frontier model attention algorithms per layer.

Agentic AI is likely only the first workload to expose this requirement, not the last. Physical AI and other emerging inference-heavy domains will place comparable, and in some cases greater, demands on infrastructure that must scale economically, making the architectural decisions enterprises make now a compounding advantage or liability rather than a one-time choice.

## The Bottom Line

Purpose-built architecture, hardware designed specifically for the software running on it, direct control over the supply chain that produces that hardware, and rack-level density sufficient to compete with GPUs given limited power and floor space are moving from an advantage to a baseline requirement for anyone operating AI infrastructure at production inference scale.

WEKApod and WEKA NeuralMesh are offerings from one vendor's attempt to meet that baseline, backed by production references at Oracle Cloud Infrastructure, Firmus, and Nebius, and by a density claim of 1.1 exabytes per rack, which no competitor has yet matched at the rack level.

The scorecard against which those figures are measured is no longer optional. Tokens per watt, tokens per rack, cost per inference, GPU goodput, latency, memory efficiency, operational density, and agent concurrency will be the metrics by which the industry evaluates every infrastructure vendor from here on.

Organizations still evaluating storage based solely on IOPS and peak throughput are already measuring the wrong metrics.



**Steve McDowell, Chief Analyst and Founder, [NAND Research Company](https://nand-research.com/)**  
Steve McDowell is a technologist with over 25 years of deep industry experience in a variety of strategy, engineering, and strategic marketing roles, all with the unifying theme of delivering data, storage and systems technologies into the enterprise market.

[Download PDF](/api/resource-pdf?slug=the-inference-economics-scorecard)
