Three days after the United States banned its models from federal civilian agencies, DeepSeek open sourced the framework that serves them at home. DeepInfer is the company’s in-house inference engine, open source under Apache-2.0, powering DeepSeek-V4.1-Flash., the 552B-parameter 16B-active model, at production scale. The announcement read “We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable,” and landed at 7,900 likes within hours.
What is actually in the release
The honest envelope first, because it shapes who this is for: the release covers offline inference only. The README’s list of what is missing names the rest of a real serving stack: a KV cache system, prefill-decode disaggregation, load balancing across replicas. Most engine functionality is already implemented, they write, but this is a serving kernel, not a service.
Inside that envelope, it is dense: the company’s own optimized kernels (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang ops), speculative decoding with a twist they call confidence-scheduled verification (draft tokens are scheduled for verification by confidence estimates; the benchmark runs nextn=3 with 75% per-token acceptance, tau 3.25), decoder SWA bounded replay that shrinks prefill work to a 140-token suffix of the 131,072-token context window, an async scheduler on by default, and two features most stacks cannot promise: batch-invariant inference (same results regardless of who shares your batch) and deterministic sampling (identical inputs, fixed seed, identical outputs, both NVIDIA-only today).
Structured output support (regex, JSON schema, EBNF grammar) ships at the sampler level.
The numbers
Two runs of numbers matter here, and they answer two different questions.
The published benchmarks answer “how fast”: DeepSeek shipped reproduction scripts with the charts, which puts this above most launch-day claims. Prefill runs at nearly 90k non-cached input tokens per second per GPU at 750 ms steps on 4 SM100 cards; decode at 26.3k output tokens/s/GPU with 256 requests per rank, trading down to about 330 tokens/s per user with 8 requests per rank at the interactivity end. Config lines read EP4DP4 and EP64DP64 over V4.1 Flash’s 384-and-change experts, running the 1017 checkpoint as the default.
The unverified-but-plausible number answers “how much”: a Zhihu post the announcement thread points to says a single cluster serving V4.1 Flash moves 155 trillion tokens per day and collects 38 dollars per card per hour, at off-peak pricing. HN’s top reaction called the figures “a nuclear bomb.” Two honest caveats: DeepSeek has not confirmed which card the dollar figure assumes (the thread’s most-upvoted objection), and off-peak pricing is the cheap end of the demand curve. But even discounted, the arithmetic says the same thing as the profit-estimator spreadsheets linked in the same thread: inference on the best MoE hardware is one of the profitable places in AI, and DeepSeek just open sourced the software that captures it.
The two hardware tracks
The supported-hardware line is short and louder than it looks: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs (CUDA 13, CANN 9.2). A first-party serving stack compiled for two supply chains at once is a geopolitical artifact as much as an engineering one: the Chinese-market fork of AI infrastructure is no longer hypothetical, it has a build script. The lineage section credits vLLM as the primary ancestor, with SGLang and TensorRT-LLM as inspiration, which makes DeepInfer the highest-profile vLLM fork to reach production at scale.
For local readers
The local angle needs the honest note: SM100 is Blackwell data-center silicon, Ascend 950 is a Chinese NPU, and neither is a desktop. Nothing in this release runs on your workstation, and our fit tables for V4.1 Flash (207 GB at 3-bit, 384 GB pooled minimum) still route through llama.cpp, vLLM, and MLX. What DeepInfer changes for local users is indirect: the reference for how the 1017 checkpoint behaves at scale is now open, which means the community stacks that DO run locally can diff their performance against the vendor’s own numbers instead of press claims.
Our engines catalog carries DeepInfer as row 22 as of today (api_openai_compatible: false: no server layer shipped), three days after the DHS order made the same lab a procurement problem for US civilian agencies. Both facts in one week say the same thing: DeepSeek is racing to make every serving dollar of its moat reproducible by anyone, wherever the hardware comes from.
Sources: DeepSeek: Introducing DeepInfer - HN thread (369 points) - Zhihu production-scale post (via the thread) - Our DeepSeek ban coverage
Related on this site: Twenty-one ways to run a model locally: the engine catalog - DeepSeek V4.1 Flash (model page) - DeepSeek banned from US civilian IT