Naadir Jeewa randomvariable

DeepInfer for local AI vs vLLM

2026-10-20

Short answer

DeepInfer is not a local-AI engine and was never meant to be one. It runs one model (DeepSeek-V4.1-Flash, 552B + 196B Engram, ~476 GiB on disk) on two chips (NVIDIA SM100, Huawei Ascend 950), and its economics only appear at 64 GPUs per decode instance. Nothing in the release lowers the hardware floor for running DeepSeek locally; that floor is set by the model, not the engine.

If you run models locally, the release matters in two indirect ways. It is the reference implementation of V4.1-Flash, so the engines you actually use (vLLM, llama.cpp, SGLang) will copy its kernels and tricks over the next few months. And it documents, with production numbers, which optimisations pay off and which are datacentre-only, which is useful when deciding what to turn on.

CAKE on Consumer Blackwell: Evidence, Kernel Replacement Map, and Reproduction Design

Date: 2026-10-13. Status: research and proposed design, not an implemented compiler or a qualified kernel cutover.

Executive decision

CAKE's compiler-agent methodology applies to SM120 and SM121. Public FlashInfer contributions already demonstrate consumer-specific generated schedules. Datacenter Blackwell kernels cannot be enabled by changing an architecture string: consumer hardware has warp mma.sync and register accumulators, but lacks tcgen05, tensor memory (TMEM), WGMMA, and two-CTA MMA.

Use family-targeted sm_120f objects where all required instructions are family-supported. Keep operator admission and performance tuning separate from cubin compatibility. An instruction-compatible GB10 is not automatically qualified for an RTX-tuned operator.

Qwen3.8 Flash Next on LiL/vLLM: why the decode step is host-bound, and what LMCache does and does not do to it

Date: 2026-10-07 Container under analysis: ghcr.io/local-inference-lab/vllm@sha256:629d097ce4f3ac2ceea86bffb174fec94f5c7fe9bb0a6dbc70e9a05e30b9e54a Reported symptom: decode throughput on RTX PRO 6000 Blackwell PCIe under WSL2 is far below the published recipe numbers; an Nsight CUDA timeline shows large GPU-idle holes between CUDA-graph replays in the MTP verifier loop.

Status of claims: everything marked verified was produced by running the shipped image or by reading the source files that the shipped image contains. The per-step timing table is the end user's own capture and is not independently reproduced here — no SM120 device was available on the analysis host.

DGX Spark: 64 KiB pages, THP and what the measurements establish

This note separates three changes that are often discussed together: kernel base-page size, huge-page policy and memory reserves. They affect different parts of Linux memory management. They must be measured separately.

Measurements below come from four GB10 hosts running two live, two-host vLLM groups on 2 October 2026. Host labels are anonymous. No isolated inference benchmark was completed.

The short version

GB10 (NVIDIA DGX Spark) arm64 IOMMU notes: v6.17 → v7.0

Audience: someone running inference on a GB10 (DGX Spark) who wants to know what changed in the arm64 IOMMU stack between v6.17 and v7.0, what it means for that part, and what to set. §1 is a plain-language explainer for the stock 4 KiB kernel. §§2–6 are measured on one GB10 host. §§7–15 are the arm64 IOMMU changelog the GB10 analysis depends on, reduced to the parts that reach this hardware.

Source of truth: local Linus tree at ~/go/src/git.kernel.org/pub/scm/linux (HEAD = 93f51579e7df, tag v7.3-rc4, 2026-09-20). Every commit hash below was resolved

Qwen3.8-Flash-Next: serving the 100 GB per-layer embedding table

Identity

The model the community calls Qwen3.8-Flash-Next lands in transformers as model_type = "qwen4_exp", under src/transformers/models/qwen4_exp/, with @auto_docstring(checkpoint="Qwen/Qwen4-Exp"). It is a VLM (Qwen4ExpForConditionalGeneration) over a hybrid text stack: linear-attention layers, MoE, hyper-connections, and Qwen sparse attention (QSA) on the layers that indexer_n_heads enables.

Did the Moral Foundations Theory Group Publish a Post-COVID Retrospective on the Purity Foundation? A Slop Report

TL;DR

No. The original Moral Foundations Theory (MFT) authors—Jonathan Haidt and Jesse Graham—have not published a formal scholarly retrospective or reassessment of the Purity/Sanctity foundation in light of COVID-19. The closest thing on record is an informal, unpublished remark by Haidt in a 2024 Conversations with Tyler interview, where—asked directly whether COVID changed his view—he argued the pandemic "wasn't about germs" but about government control, so it neither cleanly validated nor revised his purity work.
The MFT group did publish a major theory revision in this window—Atari, Haidt, Graham et al. (2023), "Morality Beyond the WEIRD," JPSP (the MFQ-2, which splits Fairness into Equality and Proportionality)—but it is a cross-cultural measurement overhaul built on 25 populations, not a COVID/purity retrospective. The Liberty/Oppression foundation was added ea

	# Laguna-S-2.1-NVFP4 on NVIDIA DGX Spark (GB10, sm_121a, arm64)
	# 2-node TENSOR-PARALLEL (TP=2) vLLM serving at NATIVE 1M context, one engine
	# sharded across two GB10 nodes over ConnectX-7 RoCE, with DFlash speculative
	# decoding and the FlashInfer B12X NVFP4 MoE backend. Fronted by an llm-d
	# Endpoint Picker (EPP) constrained to rank0.
	#
	# ─────────────────────────────────────────────────────────────────────────────
	# WHY TP=2 (vs 2 independent replicas):
	# One vLLM engine tensor-sharded across 2 DGX Spark nodes (1 GPU/node) gives a
	# single UNIFIED KV pool and shards the weights (~35 GiB/node vs ~70 GiB on a

	#!/bin/bash

	declare -A seenNamespaces
	declare -A containerInfo

	# Ensure ipcs command is available
	ipcsPath=$(which ipcs)
	if [ -z "$ipcsPath" ]; then
	echo "ipcs command could not be found"
	exit

	apiVersion: v1
	data: ${CNI_RESOURCES}
	kind: ConfigMap
	metadata:
	name: cni-${CLUSTER_NAME}-crs-0
	namespace: default
	---
	apiVersion: addons.cluster.x-k8s.io/v1alpha4
	kind: ClusterResourceSet
	metadata: