# LinkedIn’s AI Infrastructure Secrets for 1.2B Users **Author:** WEKA **Published:** August 17, 2026 ![Black and white image of four silhouetted people in a data center with rows of server racks.](https://cdn.sanity.io/images/ult5g8gw/production/8e510ab198098b826660d356ec963b516333fad0-1408x768.jpg) ## TL;DR Explore how LinkedIn builds efficient, resilient GPU infrastructure to run diverse AI workloads at massive scale. - Treat GPUs as specialized assets and design around their architecture. - Break silos between ML, infra, storage, and networking teams. - Optimize KV cache and memory hierarchy before adding storage tiers. What does it actually take to run AI at the scale of 1.2 billion registered users? Not in theory, but in production, every day, across recommendation models, large language models, and everything in between? LinkedIn’s Animesh Singh, who leads the GPU fleet powering LinkedIn’s AI platform, sat down with WEKA’s Val Bercovici for a candid conversation about what’s working, what’s hard, and what every enterprise AI team should prioritize right now. The challenges LinkedIn is solving today are those most organizations will navigate within the next two years. ## Why AI infrastructure efficiency is the defining challenge of this decade The cost curve for AI isn’t flattening. It’s accelerating. As Animesh put it: “AI is expensive. It takes a lot to put up a scalable AI infrastructure, both for continuous pre-training, fine-tuning, and the post-training plus inference landscape. And it increasingly is getting more expensive as the models are getting smarter and have more architecture.” LinkedIn runs a sprawling model portfolio, from traditional recommendation and ranking models that power the feed, job search, and people recommendations to generative AI applications like profile summarization and LIHA, LinkedIn's fully agentic hiring assistant. The challenge isn't running any one workload. It's running all of them cost-effectively, simultaneously, and at a massive scale. The answer isn't a panacea. It's a disciplined, layered approach to efficiency, from GPU silicon up through the software stack. 👉 Read more about AI efficiency metrics in [this eGuide](/resources/eguide/ai-infrastructure-efficiency-metrics-explained/). ## Stop treating GPUs like cattle. They’re pets now. The most important mental model shift for enterprise AI teams? GPUs are fundamentally different from the CPU infrastructure that preceded them. “It's whether GPUs are pets or cattle,” Animesh explained. “And CPUs, you used to treat them like a cattle farm … you have to totally shift the pendulum here that these are pets.” That means deeply understanding GPU architecture — HBM memory, Streaming Multiprocessor behavior, node topology — and designing infrastructure around these characteristics rather than abstracting them away. It also means building for failure from day one. LinkedIn observes roughly 10% thermal stress failure rates on H100 and H200 nodes, far higher than CPU equivalents. “Resiliency is not an afterthought,” Animesh said. “When you are running distributed training at scale across a fleet of 100 GPUs, 1,000 GPUs, 2,000 GPUs, failure is bound to happen." Fast checkpointing, rapid job recovery, and intelligent rescheduling aren’t nice-to-haves. Every minute of recovery time equates to expensive GPU capacity sitting idle. ## Breaking organizational silos is the most underrated AI cost lever If there’s one structural change separating efficient AI organizations from those bleeding money, it’s the end of the wall between modeling teams and infrastructure teams. “Optimization has to be a key focus area when you are doing AI,” Animesh said. “I think one thing which has worked out really well is that the silos have been broken.” In the old model, ML engineers would develop a model, hand it over, and expect infrastructure to serve it in production. That approach doesn’t survive contact with the economics of modern AI. Model characteristics such as size, memory footprint, and inference behavior must directly inform infrastructure decisions, and that dialogue can take months. Val observed that this remains the exception, not the rule: “I still see firewalls between researchers and infrastructure operations folks. We know Cohere, for example, domestically was one lab which did the fusion of those two teams. DeepSeek famously published their papers about doing that. So it's really a cool trend to see.” The co-design imperative extends beyond ML and infra all the way to storage and networking. Traditionally, AI infrastructure has been built on a purely contractual model: Storage delivers a capability, and the AI team builds on top. LinkedIn broke that pattern. When data movement became a GPU bottleneck, the AI infrastructure team worked directly with the storage team to co-design a data layer optimized for AI workloads because getting data to GPUs faster translates directly to less idle compute. As Animesh put it: “Data is my biggest bottleneck, and I need to get data as fast as possible to the GPUs so we are not keeping the GPU idle.” **The takeaway: **If your ML engineers and infrastructure engineers aren't in structured, ongoing collaboration vs. ticket-based handoffs, you are leaving substantial GPU efficiency on the table. ## Memory tiering and KV cache optimization: Where GPU efficiency is won or lost For teams running inference at scale, memory hierarchy is where the efficiency battle is fought. HBM on the GPU is fast but finite, so the question becomes how intelligently you use it and how far down the memory tier you actually need to go. LinkedIn's initial hypothesis was that production workloads would require full tiering: HBM to CPU DRAM to SSD-based storage. In practice, aggressive KV cache optimization has kept them from needing external storage at all. “I think so far, we have seen that most of the optimizations using KV cache and HBM have been successful, right?” Animesh said. “At most, we are defaulting to the CPU memory on the GPU nodes” without going to SSDs. External storage introduces real complexity: GPU-direct communication requirements, increased network traffic, and latency overhead. As Val noted, “Memory tier performance is essential, right? And especially as GPUs get faster and pre-fills get faster, you still have a high bar to meet in terms of KV cache offloading.” **The takeaway: **Exhaust KV cache optimization before architecting for storage tiering. The ceiling is likely higher than your initial assumptions. 👉 Read [this article](/learn/ai-ml/what-is-kv-cache/) for more on the importance of KV cache optimization. ## The playbook: What enterprise AI teams should do now LinkedIn's infrastructure journey maps to a clear set of principles any scaling AI organization can apply: - **Treat GPUs as pets, not cattle. **Design around their architecture from the start. - **Build resiliency in, not on. **Failure at GPU scale is operational reality, not an edge case. - **Unite the modeling and infrastructure teams. **Co-design is a competitive advantage. - **Extend co-design to storage and networking. **Data bottlenecks create GPU efficiency problems. - **Optimize KV cache aggressively. **Don’t assume you need storage tiering. - **Train large, infer lean. **LinkedIn compresses 100B+ parameter training models down to ~7-8B for production inference via distillation, pruning, and quantization. The organizations that internalize these principles now won't just run AI more cheaply. They'll run it more reliably and at a scale that compounds over time. 👉 For more insights from this conversation, be sure to watch the [full-length video](/resources/video/linkedins-ai-infrastructure-secrets-for-1-2b-users/).