DeepSeek open-sources DeepInfer inference framework

DeepSeek has released DeepInfer, the inference framework the company says carries its large-scale production services, under Apache 2.0 at deepseek-ai/DeepInfer. Alongside it ships deepseek-recipe, a library that reproduces how the official DeepSeek API turns requests into token sequences, so self-hosted deployments behave the same way.

What ships in the box

The repository lists a stack of in-house kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect. On top sit data parallelism for attention, balanced context parallelism, and expert parallelism for the MoE layers with load balancing (EPLB), driven by an asynchronous batch scheduler DeepSeek labels "zero-overhead."

Speculative decoding runs through DSpark, paired with confidence-scheduled verification and a bounded replay path for sliding-window attention. For engineering teams, two less flashy items may matter most: batch-invariant inference and deterministic sampling, meaning a request returns the same output regardless of which other requests it gets batched with. Structured output is supported via regex, JSON schema and EBNF grammars.

A narrow hardware window

The framework targets Nvidia SM100-generation GPUs and Huawei Ascend 950 NPUs, with Linux, GCC 13.0 or newer, Python 3.12, and CUDA 13 or CANN 9.2. SiliconANGLE reports that Nvidia Hopper and earlier architectures are not supported, which leaves anyone running an existing H100 fleet outside the supported path for now.

The benchmark charts in the repository are measured on 4 SM100 GPUs for prefill and 64 GPUs for decode, each inside a single scale-up domain. The figures themselves appear only in those charts rather than in the prose, so throughput cannot be recomputed from the README alone.

What is deliberately missing

DeepSeek itself spells out what a complete inference service needs but this release does not contain: a KV cache system, disaggregation of VE, prefill and decode, and load balancing across data-parallel replicas. SiliconANGLE reports those pieces are wired into the company's internal infrastructure. Anyone taking DeepInfer to production has to supply them.

Reading the move

SiliconANGLE characterizes DeepInfer as having begun as an early fork of vLLM; the-decoder puts it as the framework building on vLLM. DeepSeek's stated reason for not publishing the full engine sooner was bandwidth — a small research team whose focus is building better models. At the time we checked, the repository showed roughly 14,100 stars, 1,300 forks and 120 open pull requests.

Several widely cited numbers come from reporting rather than the README: SiliconANGLE puts one cluster at 155 trillion tokens per day and off-peak revenue at $38 per GPU per hour, with speculative decoding adding 4 to 7 percent throughput, prefill computation falling by nearly 50 percent, and draft-token verification trimmed by about 30 percent. We could not locate those figures in the repository text we read, so they remain attributed to that source and are not independently verified here.

Timing

Per SiliconANGLE, the release landed one day after the U.S. Department of Homeland Security ordered federal civilian agencies to strip DeepSeek products from their systems. The-decoder frames the move within U.S.–China technology competition. DeepSeek's own framing is narrower: helping the community cut compute costs and energy use, and making AI more affordable.