IBM, Google Launch llm-d 1.0 Framework for Enterprise AI Inference at 10x Lower Cost

IBM and Tech Giants Release Production-Ready AI Inference Framework

IBM, Red Hat, Google Cloud, CoreWeave and NVIDIA announced the release of llm-d 1.0, marking a significant milestone for organizations seeking to run large language models on their own infrastructure1

1

2

2

. The distributed AI inference framework, announced ahead of the PyTorch Conference North America in San Jose, now includes operational controls designed to help enterprises deploy and scale large language model inference reliably across computing infrastructure. Built for Kubernetes, the open-source project addresses the growing challenge of managing AI inference costs while maintaining control over hardware and cloud platform choices1

1

.

Cost Savings and Hardware Efficiency Drive Enterprise Adoption

The llm-d framework delivers substantial cost savings for running AI models in-house, with IBM and Red Hat demonstrating 5-10x lower costs compared to serving the same open model through hosted services1

1

. In one demonstration, researchers ran 3,000 software-engineering agents concurrently on self-hosted infrastructure at these reduced costs. IBM Research reported even more impressive results when using llm-d with open frontier models for quantum error correction hardware, achieving a blended serving cost of approximately $0.11 per million tokens with a prefix-cache hit rate of approximately 98%2

2

. Ruchir Puri, IBM Research's chief scientist and IBM Fellow, noted that the framework helped achieve an 11.8x efficiency advantage in computationally demanding problems, with version 1.0 extending these savings through enhanced operational readiness1

1

.

Multi-Tenant Quality of Service and Zero-Downtime Operations

Version 1.0 introduces critical enterprise capabilities including multi-tenant quality of service, zero-downtime operations with router high availability and hot standby, and rolling updates2

2

. The release adds flow controls and prioritization designed to prevent bursts of agent activity from overwhelming services requiring fast responses, such as coding assistants1

1

. Carlos Costa, distinguished engineer at IBM Research and project co-founder, explained that the release "adds the maturity to run it like any other critical service: a stable application programming interface, multiple tenants with competing priorities sharing the same infrastructure and upgrades that don't interrupt operations"1

1

. These operational controls address what happens after deployment, handling changing traffic, hardware failures, software updates and competing users while maintaining acceptable response times.

Optimized for Agentic Workloads and Context Reuse

The llm-d framework includes specialized features for agentic workloads that repeatedly access and reuse large amounts of context while supporting parallel AI agents2

2

. A routing system directs requests toward replicas that already hold relevant cached context, while another capability moves cached information among GPU memory, CPU memory and storage1

1

. IBM cited a recent agentic coding test where the framework's router captured 97% of available cache reuse. The framework can keep requests directed to appropriate infrastructure across multiple turns, which matters because agents repeatedly consult the same code, documents, instructions and conversation history while completing assignments. In a separate demonstration, researchers deployed GLM 5.2, an approximately 753-billion-parameter model, on NVIDIA GPUs, delivering over 6.6 million output tokens per minute at peak while serving up to 3,000 concurrent coding agents2

2

.

Broad Industry Support and Production Deployments

The open-source project, now governed by the Cloud Native Computing Foundation as a sandbox project, includes more than 500 contributors from over 25 organizations2

2

. Beyond founding members IBM, Red Hat, Google, CoreWeave and NVIDIA, contributors include AMD, Cisco Systems, Hugging Face, Intel and Mistral AI1

1

. Organizations currently using llm-d include DigitalOcean for Serverless Inference and Managed Agents serving trillions of tokens, JetBrains running it in production for its Junie Lite plan, and deployments at Mistral, Roblox and Cohere2

2

. Google Cloud announced integration of llm-d 1.0 into Google Kubernetes Engine across its Google TPU and NVIDIA GPU portfolio. The software supports multiple model servers including vLLM and SGLang, offering capabilities such as prefix-cache and load-aware routing, KV cache offloading and prefill/decode disaggregation. Stephen Watt, vice president in Red Hat's chief technology officer's office, emphasized that the framework "delivers high-performance, distributed inference with the reliability and scale enterprises need within an operational model they already know"1

1

.