DeepSeek Open-Sources DeepInfer Inference Tool, Pressuring OpenAI and Anthropic on AI Costs

DeepSeek Releases Production Inference Framework as Open Source

DeepSeek has open-sourced DeepInfer, the inference framework powering its large-scale production services, in a move that could reshape how businesses approach AI deployment costs1

1

2

2

. The Hangzhou-based startup announced the release on X and published the code on GitHub under the Apache 2.0 license3

3

. "We're sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable," DeepSeek stated4

4

. The timing matters because AI bills have ballooned across enterprises, forcing companies like Accenture to cap monthly Claude spending at $29 per employee after previously pushing aggressive AI adoption1

1

.

Technical Capabilities and Hardware Support

DeepInfer currently supports the DeepSeek-V4.1-Flash model and runs on both Nvidia SM100 GPUs and Huawei Ascend 950 NPUs3

3

. The inference framework bundles optimized kernels with distributed inference methods that split work across multiple chips2

2

. Key features include speculative decoding, a technique that drafts several tokens ahead and verifies them in one pass, plus asynchronous scheduling that overlaps CPU batch scheduling with device execution3

3

. The framework was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM3

3

. The vLLM project thanked DeepSeek on X for crediting the open-source community and committed to improving its own support for V4.1-Flash using ideas from DeepInfer2

2

.

Cost Implications for Cloud Providers and Enterprises

The release reveals production techniques that AI companies typically guard closely, creating opportunities for cloud providers and hosting companies running DeepSeek's AI models to reduce their own operational expenses3

3

. DeepSeek's V4.1-Flash is priced at just $0.035 per million input tokens and $0.12 per million output tokens, according to OpenRouter1

1

. In stark contrast, OpenAI's GPT-6 Astra costs $10 and $50 respectively1

1

. Inference represents one of the most computationally expensive phases in the AI lifecycle, with a McKinsey & Company report finding that by decade's end, the majority of data center compute will go towards inference rather than training new models1

1

. Open-weight models now capture 23% of spending on Vercel's AI Gateway2

2

.

Limitations and Implementation Requirements

DeepInfer provides offline inference but leaves out several components of a complete inference service3

3

. Users must supply their own KV cache system, handle prefill and decode disaggregation, and implement load balancing across data-parallel replicas2

2

3

3

. The framework requires Linux, GCC 13.0 or later, and Python 3.12, plus CUDA 13 on Nvidia hardware or CANN 9.2 on Ascend chips3

3

. DeepSeek measured its published benchmarks on four SM100 GPUs for processing prompts and 64 for generating output, though no independent party has yet verified DeepInfer's efficiency gains2

2

3

3

. Future DeepSeek models will be added to the framework as they become publicly available1

1

.

Regulatory Response and Market Dynamics

The US Department of Homeland Security issued an order one day before DeepInfer's release, banning federal civilian agencies from acquiring DeepSeek products and requiring removal of the company's technology from government IT systems4

4

. The order cited multiple threat vectors associated with the Chinese company but does not prohibit private companies or organizations from using DeepSeek's tools1

1

. Despite regulatory headwinds, the GitHub repository attracted more than 8,000 stars by Tuesday afternoon2

2

. The growing embrace of Chinese open-source models throughout US enterprises reflects mounting pressure on American AI companies to offer frontier tools without runaway inference costs that cause customers to switch providers1

1

. Outside developers might study DeepInfer's code and adapt its techniques for their own systems, potentially accelerating innovation across the broader AI ecosystem3

3

.