2026-10-20
DeepInfer is not a local-AI engine and was never meant to be one. It runs one model (DeepSeek-V4.1-Flash, 552B + 196B Engram, ~476 GiB on disk) on two chips (NVIDIA SM100, Huawei Ascend 950), and its economics only appear at 64 GPUs per decode instance. Nothing in the release lowers the hardware floor for running DeepSeek locally; that floor is set by the model, not the engine.
If you run models locally, the release matters in two indirect ways. It is the reference implementation of V4.1-Flash, so the engines you actually use (vLLM, llama.cpp, SGLang) will copy its kernels and tricks over the next few months. And it documents, with production numbers, which optimisations pay off and which are datacentre-only, which is useful when deciding what to turn on.
If you run vLLM, DeepInfer is a vLLM fork, and vLLM already serves V4.1-Flash from day 0 with the same DSpark speculative decoding, adaptive verification, EPLB and prefill/decode disaggregation. What DeepInfer adds over vLLM is a C++ scheduler, hand-fused TileLang kernels, Ascend support and DeepSeek's own tuning. What it lacks is everything else: other models, other GPUs, an HTTP server, a KV cache store, load balancing. Staying on vLLM is the right default; DeepInfer is worth reading, not running, unless you are standing up a V4.1-Flash cluster on B200-class hardware and want DeepSeek's exact numbers.
If you run consumer or workstation Blackwell (RTX 5090, RTX PRO 6000, DGX Spark), the comparison point is not upstream vLLM but Local Inference Lab's FlashInfer + b12x backend and their vLLM fork. b12x is DeepInfer's sm_120/sm_121 counterpart: hand-written kernels for the same V4.1 op set (DSA indexer, compressed sparse MLA, Engram, hyper-connections, MXFP4 MoE, DSpark) tuned for PCIe boxes instead of NVLink racks. It already runs V4.1-Flash at TP4 on four RTX PRO 6000s with the Engram table in host RAM or on SSD, which upstream vLLM's recipes do not reach. So for that hardware class DeepInfer's tricks did not need porting; the port shipped first.
DeepInfer is the engine behind DeepSeek's own API, open-sourced on 20 October 2026 as an offline inference release: a Python LLM class you call with a list of prompts, not a server. The deployment it assumes is spelled out in DeepSeek's deployment Q&A and in the code's defaults.
| Assumption | Value | Where it shows |
|---|---|---|
| One model | DeepSeek-V4.1-Flash only | Head dims asserted to (128, 512); every kernel specialised by hand |
| Two chips | NVIDIA SM100 (B200/B300/GB200/GB300), Huawei Ascend 950 | DEEPINFER_ACCELERATOR=nvidia|ascend; CUDA 13 or CANN 9.2 |
| Decode scale | 64–128 GPUs per instance ("Wide-EP"), one scale-up domain | README benchmarks on 64 SM100; Q&A: "use as large an EP as the cards allow" |
| Prefill scale | 8-GPU boxes at EP2–8 | Q&A: prefill-to-decode GPU ratio about 1:1; efficient minimum unit 64 P + 64 D |
| No tensor parallelism | Attention replicated per rank, experts sharded | model_parallel_size = context_parallel_size; Q&A: "TP and CP not yet used" |
| External prefix cache | A separate SSD cluster (3FS, 10+2 erasure coding, ~100 PiB, 3–7 day TTL) | No hash prefix cache in the engine; hit_tokens is an input |
| External load balancing | Prefill, decode, vision-encoder and expert balancers, not released | README: "required for a complete service, not included" |
| Configuration | ~60 environment variables | deepinfer/environ.py |
The headline numbers from the Q&A: one cluster serves 155 trillion tokens a day; speculative decoding with confidence-scheduled verification cuts verified drafts per step from 5.0 to 3.2–4.4 for +4–7% throughput; expert load balancing holds the slowest rank within 5–9% of the mean; recomputing only the last 128 tokens of sliding-window state at a cache hit nearly halves prefill compute.
All of this is tuned for a workload of ~60 concurrent requests per GPU with a 94%+ prefix-cache hit rate across a fleet. That is the opposite corner of the design space from a workstation serving one to eight users.
No, on two counts: the engine will not start on consumer silicon, and the model it serves does not fit.
The engine. DeepInfer requires compute capability 10.0 (SM100). RTX 50-series cards are sm_120, DGX Spark / GB10 is sm_121, and the vendored kernels (FlashMLA, DeepGEMM, DeepEP, DeepSelect) are built for SM100's tensor-memory and TMA features, not the consumer Blackwell variants. There is no AMD, Apple or Intel path. The build also pins CUDA 13, Python 3.12, a patched NCCL and flash-attn-4.
The model. V4.1-Flash is 552B backbone parameters plus 196B of Engram tables, about 476 GiB on disk at its native MXFP4/MXFP8 precision. vLLM's own recipe puts the minimum at 614 GB of GPU memory: one 8×H200 node, or one GB200 NVL4 tray. The smallest single-box configuration anyone publishes is a DGX Station GB300 at TP1 with the Engram tables (189 GiB) and 60 GiB of experts offloaded to pinned host memory, and that is a 784 GB unified-memory machine.
| Hardware | Memory | DeepInfer | V4.1-Flash via upstream vLLM | V4.1-Flash via LIL vLLM + FlashInfer/b12x | Earlier V4-Flash (284B) via llama.cpp |
|---|---|---|---|---|---|
| RTX 5090 / RTX PRO 6000 | 32 / 96 GB | No (sm_120) | No | No; one card is far below the 476 GiB checkpoint | No (needs ~110 GB at 3-bit) |
| DGX Spark / GB10 | 128 GB unified | No (sm_121) | No | No for V4.1; b12x targets GB10 and runs V4-Flash | Yes at 3-bit, slow; a 2-node pair is the usual recipe |
| Strix Halo (Ryzen AI Max) | 128 GB unified | No (AMD) | No | No (NVIDIA only) | Yes at 3-bit (110–135 GB with KV), ~5–10 tok/s class |
| Mac Studio 512 GB | 512 GB unified | No (Apple) | No | No (NVIDIA only) | Yes at 4–8-bit |
| 4× RTX PRO 6000 workstation | 384 GB | No (sm_120) | No published recipe below 614 GB | Yes: TP4, Engram table (189 GiB) in pinned host RAM or on SSD, 1M context; LIL's tested configuration | Yes |
| DGX Station GB300 | 784 GB unified | Untested; SM100-class | Yes, TP1 with host offload | Not targeted (SM100-class) | Yes |
| 8× H200 / 4× B200 | 1.1 TB / 768 GB | H200 no (SM90); B200 yes | Yes | Not targeted | Yes |
The practical consequence: below a 4× RTX PRO 6000 workstation, "running DeepSeek locally" means the earlier 284B V4-Flash in GGUF at 2–4 bits, where Unsloth's recommendations are 92–172 GB of combined RAM + VRAM depending on quant and whether the DSpark drafter is loaded. As of 20 October no established GGUF of V4.1-Flash exists; the 552B + 196B size makes a sub-200 GB quant unlikely to be usable. At 384 GB of PCIe-attached GPU memory, the LIL fork is the only path to V4.1-Flash; b12x's README marks the host-RAM Engram path unsafe after a host OOM and advises SSD storage, and publishes correctness results (16/16 on their bench) but no throughput numbers.
The one local-adjacent feature in DeepInfer, Engram tables served from pinned host RAM over RDMA, is the same idea b12x ships for PCIe (host-resident Engram, no RDMA) and vLLM ships as Engram CPU offload. On a unified-memory box the question doesn't arise.
| Axis | DeepInfer | vLLM (upstream) | LIL vLLM + FlashInfer/b12x | llama.cpp |
|---|---|---|---|---|
| Models | DeepSeek-V4.1-Flash only | Hundreds; V4.1-Flash from day 0 | Upstream vLLM's set plus b12x paths for V4.1-Flash, V4-Flash, GLM-5.3, Qwen3.8-Flash-Next | Hundreds via GGUF; V4-Flash yes, V4.1-Flash not yet |
| Hardware | SM100, Ascend 950 | NVIDIA sm_80+, AMD Instinct, Intel, TPU, Ascend via plugin | SM120/SM121 only: RTX 5090, RTX PRO 6000, DGX Spark | Anything: CUDA, ROCm, Metal, Vulkan, CPU |
| Smallest V4-family box | 4× B200 | 8× H200 / 4× B200 (V4.1-Flash); DGX Station GB300 at TP1 | 4× RTX PRO 6000 at TP4 (V4.1-Flash, Engram on host/SSD); GB10 for V4-Flash | 128 GB unified memory (V4-Flash at 3-bit) |
| Serving surface | Offline LLM.generate only; no HTTP server |
OpenAI-compatible server, router, Dynamo/llm-d integration | Same as vLLM (it is a fork), plus --load-format b12x loader and Docker images |
OpenAI-compatible llama-server |
| Scheduler | C++, zero-overhead async scheduling, DP-rank voting | Python V1 engine with async scheduling | vLLM's | Single-process, simpler batching |
| Speculative decoding | DSpark with confidence-scheduled variable verification (+4–7% at 60/rank) | DSpark with adaptive verification (NVIDIA only), EAGLE, n-gram | vLLM's DSpark path with b12x MTP/DSpark sequence ops | DSpark drafter (--spec-type draft-dspark), ~1.9× at draft 3 |
| Parallelism | Attention-DP + MoE-EP + balanced CP; no TP | TP, PP, DP+EP, DCP, EPLB | TP over PCIe with compressed all-reduce (132 B per 128 values, 48% fewer wire bytes); moe.ep_moe; comm.roce for multi-node |
Layer split / row split across GPUs or RPC nodes |
| Prefix cache | None in-engine; external 3FS store | In-engine hash prefix cache + LMCache/KV connectors | vLLM's | In-process slot cache, prompt cache files |
| Prefill/decode split | Hooks only; the layer is not released | NIXL-based 1P1D, Dynamo recipes | vLLM's | No |
| Determinism | Batch-invariant mode + seedless counter-hash sampling | VLLM_BATCH_INVARIANT=1 (beta) + seeded sampling |
vLLM's; b12x kernels not in the batch-invariant set | Seeded sampling; single-batch so invariant by construction |
| Structured output | xgrammar, compiled off the hot path | xgrammar, guidance, outlines | vLLM's | GBNF grammars, JSON schema |
| Kernels | TileLang + DeepGEMM/FlashMLA/DeepEP/DeepSelect, hand-fused for SM100 | Triton, FlashInfer, CUTLASS, DeepGEMM, FlashMLA; V4 fusions landed April–October | CuTe DSL + Triton, hand-fused for SM120/121: DSA indexer, compressed sparse MLA, blockscaled GEMM, fused MoE, Engram/PLE, hyper-connection gate fusion | ggml, hand-written per backend |
| Quantisation | Native MXFP4 experts, MXFP8 weights, FP4/FP8 KV | Same native formats; FP8 KV, MXFP4 indexer on Blackwell | NVFP4, MXFP4, MXFP8, block-FP8 GEMM; FP8 paged KV; W4A16 and W4A8 MoE | 1–8-bit GGUF, K-quants, imatrix |
| Config | ~60 env vars | CLI flags + env vars | vLLM's, plus b12x preparation sessions that benchmark fused vs separate candidates at load | CLI flags |
| Status | Production at DeepSeek; v0.1.0 public | Stable | Experimental: off backend="auto" unless FLASHINFER_ALLOW_EXPERIMENTAL_AUTO_BACKENDS=1; "not intended for production"; correctness results only |
Stable |
| Who it is for | DeepSeek, and clusters copying DeepSeek | Anyone serving at scale on datacentre GPUs | Workstation Blackwell owners who want the V4-family at full size | Anyone on one box, including unified memory |
Three rows deserve a note. The speculative-decoding row is the clearest case of DeepInfer's work already flowing downstream: vLLM shipped DSpark with adaptive verification and llama.cpp shipped the DSpark drafter before DeepInfer itself was public, both working from DeepSeek's kernel libraries and model card. The kernels row shows DeepInfer and b12x as mirror images: the same op set hand-fused twice, once for NVLink-connected SM100 and once for PCIe-connected SM120, and b12x's hyper-connection gate fusion (merged 16 October: +5% prefill throughput on GB10, decode within noise) landed before DeepInfer's mega_mhc was visible. And the scheduler row is where DeepInfer is ahead of every vLLM-derived engine in a way that may matter at small batch: its scheduler never waits on the GPU, even with speculation on, where vLLM's async scheduling still has sync points. At one to eight users on one GPU that gap is a few percent of step time; at 60 per rank it is the whole margin.
Six ideas in DeepInfer help at one to eight users on one box, and most are already available or arriving in the engines you would actually run.
| Idea | What it does for a local user | vLLM today | llama.cpp today |
|---|---|---|---|
| Speculative decoding with variable verification | At low concurrency the GPU is idle most of each step; drafting 5 tokens and verifying only the confident ones is the single biggest decode win (1.5–2×) | DSpark + adaptive verification on NVIDIA; EAGLE and n-gram for other models | DSpark drafter; fixed draft count, ~1.9× at 3 |
| Scheduling that never waits on the GPU | Removes the CPU bubble between steps; matters most when batches are small and steps are short | Async scheduling in the V1 engine; some sync points remain | Not applicable at batch 1 |
| Sliding-window bounded replay | At a cache hit, rebuild only the last 128 tokens of window state instead of re-running prefill; makes long agentic sessions with 90%+ prefix reuse cheap | Prefix cache keeps block KV; window-state recompute is model-specific and landing per model | Prompt cache keeps full state; no partial replay |
| Seedless deterministic sampling | Same seed + same position = same token, regardless of batch composition, restarts or which replica served it; makes evals and RL rollouts reproducible | Seeded RNG per request; VLLM_BATCH_INVARIANT=1 (beta) for batch independence |
Seeded; invariant at batch 1 |
| Structured output compiled off the hot path | Grammar compiled before admission so the first token is not delayed | xgrammar with compile cache; same approach | GBNF compiled at request start |
| Fused norm → RoPE → quantise → KV write | Decode on a memory-bound GPU is dominated by small kernels; fusing the per-layer tail removes several launches per layer | V4 fusions landed with 1.4–20× kernel-level gains in the April post; more following | Per-backend fused ops, slower to land |
Two DeepInfer details are worth copying into any local setup even without engine changes. Pin NCCL (or any collective library) to a fixed number of SMs if you run tensor parallel on two or more cards; DeepInfer uses 32 of 148, and letting collectives grab the whole GPU is the common cause of jitter in multi-GPU workstations. And keep a measured per-batch-size latency table: DeepInfer times each CUDA-graph bucket at start and caches it on disk for an hour; the same table tells you which concurrency your box can take before per-user speed collapses, and it is what drives its speculative-decoding budget.
For sm_120/121 owners the "already have it" column is mostly b12x rather than upstream vLLM: the Engram host path, the fused norm/RoPE/quantise tails, the DSA indexer and compressed sparse MLA kernels, and the hyper-connection fusion all exist there today. What b12x does not have is DeepInfer's scheduler, confidence-scheduled verification, or any published throughput figure; those are the parts still worth reading DeepInfer for.
What local users should not expect: DeepInfer's 5.0 → 3.2 draft reduction and its +4–7% throughput are measured at 60 concurrent requests per GPU. At one user the verification budget is never the constraint; a fixed draft length of 3–5 is as good.
Five of DeepInfer's headline features only pay off with dozens of GPUs, and copying them onto a workstation costs more than it returns.
- Wide expert parallelism. Spreading 384 experts over 64 GPUs cuts the per-step weight bytes each GPU reads, which is the point. On one box every expert is already local; the all-to-all is pure overhead. On a unified-memory machine the "experts offloaded to host" path is the same memory anyway.
- Expert load balancing (EPLB). Its job is to stop one GPU in 64 from being the straggler. With one GPU there is no straggler. With four, routing imbalance shows up as a few percent, and the cost of live weight shuffles between steps is larger than that.
- Balanced context parallelism. Splits one long prompt's attention across GPUs with equal FLOPs per rank. Useful when a 1M-token prefill must finish in seconds across an 8-GPU box; at home the prompt is served by one GPU and the cache hit does the work.
- An external KV store. DeepSeek keeps 14 TB of cache per GPU on an SSD cluster so any prefill machine can hit any prefix. The equivalent for one box is the engine's own prefix cache plus host RAM; the win comes from the hit rate, which an agentic session gives you for free.
- Prefill/decode disaggregation and DP load balancing. Separate fleets for prefill and decode let each be sized and tuned alone. The minimum efficient unit DeepSeek quotes is 64 + 64 GPUs. Below that, the KV transfer between roles costs more than the specialisation saves.
Two more are engineering choices, not scale effects, and are simply worse for a local operator: ~60 environment variables with invariants enforced by assertion at start, and no HTTP server (the released entry point is an offline Python LLM class).
| Situation | Do | Don't |
|---|---|---|
| Single consumer GPU (RTX 5090, 4090) | Stay on llama.cpp or vLLM with models that fit; V4-family is out of reach | Wait for a DeepInfer build; none is coming for sm_120 |
| Unified-memory box (DGX Spark, Strix Halo, Mac) | llama.cpp with V4-Flash (284B) at 3-bit and the DSpark drafter; expect 5–15 tok/s | Expect V4.1-Flash; 552B + 196B does not fit at any usable quant |
| Multi-GPU workstation (2–4× RTX PRO 6000, 192–384 GB) | LIL vLLM fork with FlashInfer/b12x for the V4 family (V4.1-Flash at TP4 with Engram on SSD); upstream vLLM for everything else, NCCL pinned to a fixed SM count, VLLM_BATCH_INVARIANT=1 if you run evals |
Enable EPLB or P/D split; neither pays below a rack. Treat b12x as production; its own README says not to |
| DGX Station GB300 (784 GB unified) | vLLM recipe at TP1 with Engram and expert host offload is the only published V4.1-Flash single-box path | DeepInfer; it is SM100-class but untested outside DeepSeek's cluster shapes |
| vLLM shop on H100/H200 | Keep vLLM; DeepInfer does not support SM90 at all | Fork or port DeepInfer kernels; they target SM100 tensor-memory features |
| vLLM shop on B200/B300/GB200 serving V4.1-Flash | Benchmark vLLM's recipe (DSpark adaptive verification, EPLB, NIXL P/D) first; read DeepInfer's scheduler and Q&A for DeepSeek's exact settings (prefill EP2–8, decode EP64–128, 4 and 1 redundant experts, 1,000/10,000-step rearrangement) | Switch wholesale; you lose the HTTP server, KV store, router and every other model |
| Ascend 950 fleet | DeepInfer is one of two production-grade V4.1-Flash paths (vllm-ascend is the other) | Assume feature parity; several DeepInfer kernels are CUDA-only and skip on Ascend |
| Building or tuning your own engine | Read csrc/engine/scheduler.hpp, mtp_varlen.hpp, worker/decode_perf_cache.py and the Q&A PDF; the design rationale is in the Chinese comments |
Treat the tile sizes as tuned; they are hard-coded heuristics for one GPU |
The one-line version: local AI did not change on 20 October; the reference implementation of the largest open model did, and the engines local users already run, upstream vLLM and llama.cpp for most people and the b12x stack for workstation Blackwell, had most of it earlier.
DeepInfer repository (v0.1.0, snapshot read 20 Oct 2026) and doc/DeepSeek V4.1 Flash 推理部署 Q&A.pdf; vLLM recipe for DeepSeek-V4.1-Flash; vLLM DeepSeek V4 post, 24 Apr 2026; vLLM batch invariance docs; vLLM Ascend V4.1-Flash tutorial; Unsloth DeepSeek V4 guide (GGUF sizes, memory table, llama.cpp DSpark flags); V4-Flash GGUF on 128 GB machines; b12x README (op set, targets, TP4 V4.1 profile, Engram host/SSD note); LIL flashinfer PR #34 (hyper-connection gate fusion, GB10 numbers); LIL vllm PR #1008 (B12X from FlashInfer 0.7.1, RTX PRO 6000 test set). Consumer-GPU figures (5–15 tok/s on unified memory) are approximate, from community reports rather than a single benchmark.