llm-d 1.0 is an open-source, Kubernetes-native distributed inference stack for running large language models at production scale. Co-founded by Google, Red Hat, and IBM, it is a CNCF Sandbox project built by more than 500 contributors from over 25 organizations. It works with vLLM and SGLang, and supports NVIDIA, AMD, and Intel GPUs as well as Google Cloud TPUs.

Why does llm-d matter for agentic AI workloads?

Generic load balancers scatter requests across replicas, destroying prefix cache state and forcing expensive recomputation on every turn of a multi-step agent conversation. llm-d routes each request to the replica that already holds the agent's context, avoiding that cost. IBM Research achieved a 98% prefix-cache hit rate and reduced serving costs approximately 11.5 times compared to a proprietary stack.

How much does it cost to run models on llm-d?

IBM Research self-hosts frontier open models on llm-d for up to 3,000 concurrent coding agents on 544 NVIDIA H100 GPUs, reaching roughly $0.11 per million tokens. That is approximately 11.5 times cheaper than equivalent proprietary serving infrastructure for the same workload.

Who is already running llm-d in production?

Google Cloud (Vertex AI), AWS SageMaker HyperPod, IBM Research, Tesla, Red Hat, Mistral AI, JetBrains (behind its Junie coding agent), DigitalOcean, Oracle Cloud, and Prime Intellect all run llm-d in production. Red Hat ships it as the distributed inference stack in Red Hat AI on OpenShift.

What is new in llm-d v1.0 compared to v0.x?

v1.0 adds stable v1 APIs, multi-tenant quality-of-service routing via llm-d Flow Control, zero-downtime rolling upgrades via DisaggregatedSet, agent-aware session routing with native Responses API state, and native PyTorch model serving on Google Cloud TPUs via TorchTPU.

llm-d 1.0: The Open Inference Stack Built for AI Agents

On October 20, 2026, at PyTorch Conference in San Francisco, Google, Red Hat, and IBM announced llm-d 1.0, a milestone that matters for every enterprise team running or planning to run AI agents in production.

llm-d is a free, open-source, Kubernetes-native stack for distributed LLM inference. A CNCF Sandbox project co-founded by Google, Red Hat, and IBM, it counts more than 500 contributors from over 25 organizations. Version 1.0 ships the first stable API, multi-tenant quality-of-service, zero-downtime upgrade tooling, and native support for agent workloads. It is already in production at Google Cloud, AWS, IBM Research, Tesla, Mistral AI, JetBrains, and others.

For teams building agentic AI, this is the infrastructure layer that makes self-hosted open models economically viable at scale.

Why Inference Infrastructure Breaks Under Agent Load

Most inference infrastructure was designed for chat: one prompt in, one response out. Agents are different. A single task can generate twenty or more tool calls, each building on context from previous turns. A naive load balancer scatters those calls across different replicas, each of which must recompute the shared prefix from scratch. At scale, that recomputation becomes a dominant cost.

IBM Research documented the gap directly. Running coding agents on generic infrastructure versus llm-d, the team measured a 98% prefix-cache hit rate on llm-d. Blended serving costs dropped to approximately $0.11 per million tokens, around 11.5 times cheaper than a comparable proprietary stack for the same workload.

Google Cloud observed the same dynamic in Vertex AI production. Its serving infrastructure, built on the llm-d Endpoint Picker, doubled its prefix-cache hit rate from 35% to 70% and cut median time-to-first-token by more than 35%.

This is the core problem llm-d solves: turning a cluster of independent model servers into a coordinated serving pool that understands agent session context.

What llm-d 1.0 Actually Does

llm-d sits between your Kubernetes cluster and your model servers (vLLM or SGLang). It adds an intelligent routing and scheduling layer that is aware of inference dynamics. The key capabilities:

Prefix-cache-aware routing. Requests from the same agent session are routed to the replica already holding the session’s KV cache, avoiding prefix recomputation and cutting both latency and compute cost.

KV cache offloading. When GPU HBM fills up, llm-d offloads KV cache to host CPU memory and, on cloud deployments, to remote storage tiers such as Google Cloud Managed Lustre. Long-context workloads that would otherwise run out of memory continue uninterrupted.

Prefill/decode disaggregation. The compute-heavy prefill phase and the memory-bandwidth-intensive decode phase are handled by separate pools, improving GPU utilization for mixed traffic.

Multi-tenant quality of service. v1.0 adds priority and fairness-aware routing via llm-d Flow Control. Interactive turns get priority; background agent loops and RL rollouts are queued to avoid stalling users sharing the same fleet.

Zero-downtime upgrades. The new DisaggregatedSet controller orchestrates rolling updates across prefill and decode pools without dropping active sessions or losing KV cache state.

Agent-aware session routing. Turn-priority scheduling and native Responses API state keep a session’s context on the same replicas across multiple calls, removing a major source of latency variance in long-horizon tasks.

Production Numbers

The table below summarizes reported results from production deployments, as stated in the v1.0 announcement materials from participating organizations:

Organization	Workload	Key Result
Google Cloud (Vertex AI)	Production inference serving	Prefix cache hit rate 35% to 70%; TTFT reduced 35%+
IBM Research	3,000 concurrent coding agents, 544 H100 GPUs	Cost: ~$0.11/M tokens; 11.5x cheaper than proprietary
Tesla + Red Hat	Llama 3.1 70B on AMD MI300X	3x output tokens/sec; 2x TTFT reduction
AWS SageMaker HyperPod	Disaggregated inference	Up to 70% more tokens/sec vs standard vLLM
Red Hat benchmark	GLM-5.2 (753B) on NVIDIA	6.6M output tokens/min peak; 3,000 agents, zero preemptions

Sources: llm-d v1.0 release post, Google Cloud blog, IBM Newsroom, Red Hat blog.

What This Means for Enterprise AI Teams

Three implications stand out beyond the benchmarks.

The agentic cost problem is solved for self-hosters. The biggest barrier to running open-weight models for agentic workloads was infrastructure complexity. Proprietary serving was expensive; rolling your own distributed routing was months of engineering. llm-d 1.0 gives teams a production-grade solution they can deploy on any major cloud or on-premises, in weeks rather than quarters. For context on recent open-weight model options suited to agentic workflows, see DeepSeek V4.1 Flash-1017 and what its agentic benchmark numbers mean for enterprise buyers.

Vendor lock-in is now a choice, not a constraint. llm-d supports NVIDIA, AMD, and Intel GPUs and Google Cloud TPUs, all behind the same API. DigitalOcean runs the trillion-parameter Kimi K3 model across AMD and NVIDIA hardware in a single serving pool. Teams that have been tied to proprietary clouds because of inference complexity now have a portable alternative with a stable open-source governance structure.

The open standard wins when large companies commit to it. llm-d is not a research project or a startup bet. Google runs it in Vertex AI production. AWS runs it on SageMaker HyperPod. Red Hat ships it in Red Hat AI on OpenShift with validated deployment guides for Azure and CoreWeave. When enterprises assess whether an open-source project will survive and be maintained, this level of commercial backing provides meaningful assurance. The CNCF Sandbox governance means no single contributor can capture the standard or change the license.

Hardware Coverage

One underreported aspect of llm-d 1.0 is its accelerator breadth. The project supports:

NVIDIA GPUs: validated on A100, H100, H200, and Blackwell series; deployed by Google Cloud, IBM, AWS, and Tesla
AMD GPUs: validated on MI300X with RDMA; deployed by Oracle Cloud and Tesla
Intel GPUs: supported via upstream vLLM integration
Google Cloud TPUs: native PyTorch model serving via TorchTPU, validated on TPU 7x (Ironwood); prefix-aware routing cut median TTFT by more than 85% on agentic coding workloads in benchmarks

For enterprises that have already invested in on-premises AMD or Intel compute, or that use TPUs through Google Cloud, this coverage removes a practical blocker to deploying competitive open models.

How to Get Started

llm-d 1.0 is available today. The llm-d.ai documentation covers paths from a single node to a production fleet. The project’s GitHub is at github.com/llm-d. Each major cloud partner published validated Helm charts and deployment guides alongside the v1.0 release.

For teams not ready to self-host, the same capabilities are available through managed offerings: GKE Inference Gateway on Google Cloud, SageMaker HyperPod on AWS, and Red Hat AI on OpenShift.

The IBM Research benchmark provides the most directly useful reference for enterprise evaluation: 3,000 concurrent coding agents, 98% prefix-cache hit rate, $0.11 per million tokens. If that workload profile matches what you are building, the ROI case for moving to llm-d is clear.

Building a production agentic AI system requires more than inference infrastructure. Governance, orchestration, and multi-agent coordination are equally important. For a framework on those questions, see how enterprises approach multi-agent system governance and deployment. If you want to work through the architecture for your specific environment, our team at Enera can help.