stock-stack-gaps-1017.md 5.0 KB
Stock stacks vs ours — DeepSeek-V4.1-Flash-1017 (2026-10-23)
Focus since 2026-10-23: deepseek-ai/DeepSeek-V4.1-Flash-1017 — the
production model. Everything else is secondary. This note records where our
pinned engines sit against upstream and which upstream work matters for 1017
on four Sparks.
Where we are
| lane | engine build | upstream | gap |
|---|---|---|---|
production dsv41-flash-1017-4x-sglang |
aidendle94/sparkrun-sglang-dsv41-gb10@sha256:f617afd8… (2026-09-29), sglang 0.0.0.dev1+gda64c5cbb, torch 2.13+cu130 |
SGLang v0.5.22 (2026-10-19) | ~3 weeks of DSpark/HiSparse work |
glm53-mxfp4-fp8-4x-r39 (our clone-at-pin) |
jovian-judgement:r39-glm53, vLLM 0.26.1rc0+glm53.r39.vllm9e5d1793 |
vLLM 0.32.0 (2026-10-18) | six minor releases, plus the fork's own b12x |
| eugr nightly (staged on dgx5–7) | eugr/spark-vllm-b12x:nightly-20261015, vLLM main 8244d1c44 |
vLLM main | essentially current |
What upstream added that bears on 1017
vLLM 0.32.0 (release notes, 2026-10-18):
- DeepSeek-V4 and V4.1: decoder-side SWA bounded replay (#58132) — replay
layers captured into CUDA graphs and trimmed in PIECEWISE graphs
(
compilation_config). This is the mechanism that lets a bounded-window model replay long prefixes instead of recomputing them; it is exactly the "long prefill is our pain" problem we hit on the GLM lane. - NVFP4 KV cache on SM12x (#46963) and with XQA decode on SM12x (#60452): a 4× smaller KV page on GB10 than fp8 → more context per pin. Worth probing on 1017 first: it changes the context/token economics more than any kernel.
- Speculative decoding: fused multi-step draft decode on FlashInfer trtllm-gen (#58371), in-graph metric collection; GLM-5.3 gets sparse-MLA index conversion reuse (#59464) and fused multi-step draft decode (#57443).
- Context parallelism: decode requests sharded across PCP ranks (#52162),
block-interleaved DCP (#52162 area) — the DCP that our r39 fork refused with
DCP not supportedmay exist upstream now. - HiSparse: per-request residency cache (up to 58% at high concurrency, #60083), lower scheduler overhead, GPU-resident contexts.
- Startup/memory: shared-memory tensor arena on by default (#51207).
- Fixes: silent CUTLASS NVFP4/MXFP4 MoE corruption under EP (#59042) — that one is a correctness bug in the exact kernel family our mxfp4 lanes use.
SGLang v0.5.22 (2026-10-19):
- DeepSeek-V4.1: up to 2.3× more throughput per GPU, plus DSpark on new hardware; DSpark support (#41623) with a run of correctness fixes: mask uncommitted slots before commit-inject cache lookups (#42624), fold draft sampling for DSv4.1 heads (#42987), one step interface for markov heads (#43620/#43621), top-p mask capture for DFlash/DSpark (#34201).
- HiSparse with speculative decoding (multi-step KV swapping, #37771) — long-context serving with the KV cache in CPU memory, which is a different answer to our GB10 memory wall than TP8.
- Hybrid-linear models no longer resume from wrong mamba states when spec + structured output are combined (relevant to our agentic benchmarks).
FLARE (different runtime, same model)
NVIDIA/flare-local-inference serves the same checkpoint from prepared CUDA
graphs in a Rust runtime; first boot was ~2.4× slower on decode than our SGLang
lane with nothing tuned. It moves several commits a day — see
flare-bringup.md
and scripts/flare-upstream-watch.sh. The 2026-10-23 batch added
Preset DeepSeek V4.1 packed GPU speculation, adaptive verification scoring
alpha, resume preparation checkpoints, Serve and prepare over hosts cabled
as a chain and a MiniMax-M3.1-Flash recipe with packed DSpark speculation.
MiniMax-M3.1-Flash (new on the NAS)
MiniMaxAI/MiniMax-M3.1-Flash — 252.4 GB, 72 files, downloaded 2026-10-23.
MiniMaxM3SparseForCausalLM (plus a vision tower: minimax_m3_vl), 60 layers,
hidden 6144, 64 heads, MoE top-4, 1M context, mxfp8 weights (block [1,32]).
At TP4 that is ~63 GB per rank — comfortable on four Sparks, unlike the GLM-5.3
full lane's 99 GiB. Candidate for the next "big model that actually fits" slot;
FLARE upstream already has the recipe with packed DSpark speculation.
The queue that follows from this
- 1017 on stock vLLM 0.32 (image already staged): boot TP4, measure
pp/tg, then probe
--kv-cache-dtype nvfp4on SM121 — the single biggest potential context win. - SGLang 0.5.22 for the production lane: our base is 2026-09-29; the DSpark fixes since then are correctness fixes for our exact configuration.
- b12x A4 prefill into our clone-at-pin build (+10% prefill claimed, todo #23) — prefill is what limits usable prompt length on GB10.
- FLARE tuning on 1017 (scheduler policy, concurrency, adaptive verification) to see whether the Rust runtime can close the 2.4× gap.
- MiniMax-M3.1-Flash: serve it once, measure, and only then compare it with 1017 on quality.