Startseite Erkunden Hilfe
Anmelden
sm121
/
launcher
2
0
Fork 0
Dateien Issues 0 Pull-Requests 0 Wiki
Struktur: 72358e130c
Branches Tags
launcher
/
specs
/
tech-architecture
/
stock-stack-gaps-1017.md

stock-stack-gaps-1017.md 5.0 KB
Verlauf Originalformat

Stock stacks vs ours — DeepSeek-V4.1-Flash-1017 (2026-10-23)

Focus since 2026-10-23: deepseek-ai/DeepSeek-V4.1-Flash-1017 — the production model. Everything else is secondary. This note records where our pinned engines sit against upstream and which upstream work matters for 1017 on four Sparks.

Where we are

lane engine build upstream gap
production dsv41-flash-1017-4x-sglang aidendle94/sparkrun-sglang-dsv41-gb10@sha256:f617afd8… (2026-09-29), sglang 0.0.0.dev1+gda64c5cbb, torch 2.13+cu130 SGLang v0.5.22 (2026-10-19) ~3 weeks of DSpark/HiSparse work
glm53-mxfp4-fp8-4x-r39 (our clone-at-pin) jovian-judgement:r39-glm53, vLLM 0.26.1rc0+glm53.r39.vllm9e5d1793 vLLM 0.32.0 (2026-10-18) six minor releases, plus the fork's own b12x
eugr nightly (staged on dgx5–7) eugr/spark-vllm-b12x:nightly-20261015, vLLM main 8244d1c44 vLLM main essentially current

What upstream added that bears on 1017

vLLM 0.32.0 (release notes, 2026-10-18):

  • DeepSeek-V4 and V4.1: decoder-side SWA bounded replay (#58132) — replay layers captured into CUDA graphs and trimmed in PIECEWISE graphs (compilation_config). This is the mechanism that lets a bounded-window model replay long prefixes instead of recomputing them; it is exactly the "long prefill is our pain" problem we hit on the GLM lane.
  • NVFP4 KV cache on SM12x (#46963) and with XQA decode on SM12x (#60452): a 4× smaller KV page on GB10 than fp8 → more context per pin. Worth probing on 1017 first: it changes the context/token economics more than any kernel.
  • Speculative decoding: fused multi-step draft decode on FlashInfer trtllm-gen (#58371), in-graph metric collection; GLM-5.3 gets sparse-MLA index conversion reuse (#59464) and fused multi-step draft decode (#57443).
  • Context parallelism: decode requests sharded across PCP ranks (#52162), block-interleaved DCP (#52162 area) — the DCP that our r39 fork refused with DCP not supported may exist upstream now.
  • HiSparse: per-request residency cache (up to 58% at high concurrency, #60083), lower scheduler overhead, GPU-resident contexts.
  • Startup/memory: shared-memory tensor arena on by default (#51207).
  • Fixes: silent CUTLASS NVFP4/MXFP4 MoE corruption under EP (#59042) — that one is a correctness bug in the exact kernel family our mxfp4 lanes use.

SGLang v0.5.22 (2026-10-19):

  • DeepSeek-V4.1: up to 2.3× more throughput per GPU, plus DSpark on new hardware; DSpark support (#41623) with a run of correctness fixes: mask uncommitted slots before commit-inject cache lookups (#42624), fold draft sampling for DSv4.1 heads (#42987), one step interface for markov heads (#43620/#43621), top-p mask capture for DFlash/DSpark (#34201).
  • HiSparse with speculative decoding (multi-step KV swapping, #37771) — long-context serving with the KV cache in CPU memory, which is a different answer to our GB10 memory wall than TP8.
  • Hybrid-linear models no longer resume from wrong mamba states when spec + structured output are combined (relevant to our agentic benchmarks).

FLARE (different runtime, same model)

NVIDIA/flare-local-inference serves the same checkpoint from prepared CUDA graphs in a Rust runtime; first boot was ~2.4× slower on decode than our SGLang lane with nothing tuned. It moves several commits a day — see flare-bringup.md and scripts/flare-upstream-watch.sh. The 2026-10-23 batch added Preset DeepSeek V4.1 packed GPU speculation, adaptive verification scoring alpha, resume preparation checkpoints, Serve and prepare over hosts cabled as a chain and a MiniMax-M3.1-Flash recipe with packed DSpark speculation.

MiniMax-M3.1-Flash (new on the NAS)

MiniMaxAI/MiniMax-M3.1-Flash — 252.4 GB, 72 files, downloaded 2026-10-23. MiniMaxM3SparseForCausalLM (plus a vision tower: minimax_m3_vl), 60 layers, hidden 6144, 64 heads, MoE top-4, 1M context, mxfp8 weights (block [1,32]). At TP4 that is ~63 GB per rank — comfortable on four Sparks, unlike the GLM-5.3 full lane's 99 GiB. Candidate for the next "big model that actually fits" slot; FLARE upstream already has the recipe with packed DSpark speculation.

The queue that follows from this

  1. 1017 on stock vLLM 0.32 (image already staged): boot TP4, measure pp/tg, then probe --kv-cache-dtype nvfp4 on SM121 — the single biggest potential context win.
  2. SGLang 0.5.22 for the production lane: our base is 2026-09-29; the DSpark fixes since then are correctness fixes for our exact configuration.
  3. b12x A4 prefill into our clone-at-pin build (+10% prefill claimed, todo #23) — prefill is what limits usable prompt length on GB10.
  4. FLARE tuning on 1017 (scheduler policy, concurrency, adaptive verification) to see whether the Rust runtime can close the 2.4× gap.
  5. MiniMax-M3.1-Flash: serve it once, measure, and only then compare it with 1017 on quality.