Google launches llm-d 1.0 for distributed AI inference

Google and its partners have released llm-d 1.0, an open-source software stack for distributed large language model inference on Kubernetes. More than 500 contributors from 25 organisations back the project under the Cloud Native Computing Foundation sandbox.

The release formalises a system designed to help companies run AI model-serving workloads across clusters of model servers, rather than on single machines or behind standard load balancers. Co-founded by Google, Red Hat, IBM and other contributors, it links model-serving tools from the PyTorch ecosystem with Kubernetes infrastructure used in production environments.

llm-d 1.0 targets a shift in AI inference traffic from simple one-prompt interactions to multi-step agent workloads and reinforcement learning rollouts. These workloads can arrive in bursts, reuse long prompt prefixes and compete with interactive requests, creating delays and duplicate memory use when traffic is spread inefficiently across servers.

The stack includes routing that tracks where a session's cached context is already stored, a tiered key-value cache system, and flow control designed to separate latency-sensitive requests from background workloads. It can route requests across vLLM and SGLang deployments, as well as mixed fleets of cloud TPUs and GPUs, through a single endpoint.

What is included

Version 1.0 introduces a set of stable application programming interfaces and a request path tested across multiple clouds and accelerator types. It also adds higher-availability routing and upgrade tools intended to avoid dropping active sessions during software updates.

One new orchestration feature is DisaggregatedSet, which manages separate pools for the prefill and decode functions used during model inference. This allows rolling updates to move matching versions together while letting each phase scale independently.

The release also adds an asynchronous processor for queue-based workloads. The component can pull requests from services including Google Cloud Pub/Sub or Redis, then route them alongside real-time traffic while using strict priority lanes and quotas to limit interference with interactive use.

Another feature is support for time-slicing on Google Cloud TPU and GPU infrastructure. Google says this lets operators use otherwise idle periods during reinforcement learning workloads to run asynchronous inference jobs, raising accelerator utilisation from about 40% to more than 70% in tested scenarios.

Cloud integration

Google is tying the software closely to its cloud infrastructure, particularly Google Kubernetes Engine. GKE Inference Gateway can run the llm-d router as a managed service and extend routing across clusters and regions, while autoscaling tools such as KEDA, Horizontal Pod Autoscaler and Kueue can manage queue depth, saturation and quota on shared accelerators.

The system also supports key-value cache offloading through several memory layers, including host memory, solid-state storage and Google Cloud Managed Lustre. The aim is to reduce the risk of accelerator memory being filled by repeated or long-context workloads.

Google cited several benchmark results for the software and related services. In one coding workload on Ironwood TPU, prefix-aware routing and host-memory offload reduced median time-to-first-token by more than 85% and increased total throughput by 2.4 times, compared with a standard Kubernetes Service.

Google also pointed to production use in the Gemini platform, formerly Vertex AI, where GKE Inference Gateway reportedly doubled prefix cache hit rates from 35% to 70%. The company said that change cut time-to-first-token by more than 35% for coding agents and reduced P95 time-to-first-token by more than 50% under bursty chat traffic.

In another published benchmark, Google said a 17,000-node multi-region deployment serving a mixture-of-experts model across the US and Europe increased throughput by 2.9 times across three clusters, with a 99.9% success rate and less than 1% routing overhead. A separate benchmark on an A3 Mega cluster serving a 70 billion-parameter model showed about a 40% improvement in time-to-first-token and a 30% reduction in end-to-end latency for 50,000-token prompts when using multi-connector cache offloading to Managed Lustre in addition to host memory.

User backing

The project has also attracted public support from users and contributors operating large-scale AI infrastructure.

"At Snap, we are integrating llm-d into our production AI infrastructure to facilitate high-performance inference at scale. By employing prefix-cache-aware routing, we have achieved prefix cache hit rates ranging up to 75-80%. We appreciate the open-source nature of llm-d, as it enables seamless integration with our Envoy-based service mesh," said Vinay Kola, Senior Manager, Software Engineering, Snap.

"At ZenteiQ, we use GKE Inference Gateway powered by llm-d to accelerate synthetic data generation for our sovereign scientific foundation models. By leveraging load-aware and prefix-cache-aware routing, we have achieved over 4× higher aggregate inference throughput, increasing daily generated token capacity from 1.47 billion to 5.97 billion tokens per 1,000 TPU v6e accelerators. The open-source foundation of llm-d, integrated with GKE, enables us to efficiently serve multiple model families at scale without developing and maintaining a custom inference scheduler," said Ganesan.

"Serving frontier models across interactive applications and high-volume APIs means handling steep traffic spikes without letting bursty workloads degrade latency for priority customers. With llm-d's prefix-cache-aware routing, flow control and disaggregated serving support, we can enforce strict priority bands across our commercial tiers, automatically holding or shedding lower-priority requests when pools saturate so paid traffic meets our latency SLOs. Mistral is a proud supporter of open-source software, and we will continue to contribute back to llm-d and DisaggregatedSet projects," said Felardos.