launcher/specs/tech-architecture/upstream-watch-2026-10-17.md

Upstream watch — 2026-10-17 (night-shift research pass)

Pass across vLLM releases, the NVIDIA forums, and HF model cards, feeding the night-shift agenda (history/night-shifts/2026-10-17-night-plan.md).

vLLM release status — base stays v0.31.0

• Latest release is still v0.31.0 (2026-10-05). v0.32.0 is only at rc2 (rc0/rc1/rc2 tags, no stable). No re-pin of the kindling base yet.
• Keep watching for v0.32.0 stable (would trigger the base re-pin + overlay rebase + flashinfer bump, bench-gated vs the 95/100 kindling 2x anchor).

New NVIDIA-official FP4 checkpoints (all need the b12x container)

NVIDIA shipped a family of "…-FP4-Local" QAD checkpoints quantized with nvidia-modelopt v0.47.0 via quantization-aware distillation (QAD). Their NVFP4 scale factors use a new lossless-compression format that upstream vLLM does NOT yet support — you MUST run the eugr/spark-vllm-b12x container (b12x backend + --load-format b12x), not stock vLLM. Digest-pinned eugr/spark-vllm-b12x@sha256:03a95654…bb3595 (arm64, built 2026-10-15).

• nvidia/GLM-5.3-Flash-FP4-Local — official NVFP4 GLM-5.3-Flash. 2x DGX Spark, 500K ctx, MTP n=5 (moe_backend:humming, attention_backend:B12X), FP8 KV (~2M tokens), --attention-backend B12X --block-size 256 --moe/--linear/--load-format b12x, --reasoning-parser glm45 --tool-call-parser glm47, GMU 0.87. Card quality 91.3% / 92.4%. This is the official NVFP4 counterpart to our kindling 2x lane (our own r35 build) — a direct A/B vs the 95/100 anchor. Recipe: glm53-flash-fp4-local-2x (recipes d5276db). NOTE: 2x → needs 2 free Sparks; 2nodes-dac is busy with the live kindling serve.
• nvidia/Qwen3.8-Flash-Next-FP4-Local — 1x Spark, b12x, MTP n=4, FP8 KV, 262K, GMU 0.8, VLLM_ENGINE_READY_TIMEOUT_S=3600 (b12x autotune > 600 s). Forum hardmode 89/100, ~49 t/s decode. Recipe qwen38-flash-next-fp4-local.
• nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 (75 GB, 1x Spark, VL, 1M ctx, LiLiCorr dflash-7 drafter, mamba flags, qwen3_coder/nemotron_v3) and Nemotron-3.5-Bolt-74B-A9B-24GB (24 GB, QAD 2-bit experts, 1x Spark, ultra_v3 parser plugin). Recipes nemotron-35-super-vl-120b-nvfp4, nemotron-35-bolt-74b-24gb. Forum: Super-VL 90/100, Bolt 88/100.

GB10 hardware gotchas (both directly bit us this session)

• Stuck GPU clock (forum 386060, confirms 376039/376239/366944/…): GB10 SM clock can stick at ~660 MHz (max 3003, app clock 2418) at ~16 W with throttle reasons 0x0; nvidia-smi -lgc is accepted but ignored and a reboot does not clear it. Symptom = anomalously low throughput that looks like "hot state" but is a hard clock cap. Remedy (community, not NVIDIA- documented): power off, unplug AC + USB, hold the power button ~10 s, wait a few minutes, reconnect (one report cleared to 2080 MHz / 77 TFLOPS). → When a lane underperforms, check SM clock under load (expect ~2000–2400, not ~660) before blaming the recipe; a smart-plug cycle may not be enough (needs the hold-power-button drain).
• GPU/unified memory is NOT cgroup-contained on GB10 (forum 386057): with the 580 driver nvidia-smi reports Memory-Usage: Not Supported on the iGPU, and docker --memory / MemoryMax= do not cap cudaMalloc/unified memory. When the unified pool is exhausted the whole host stalls (not just the container). This is exactly why the Ling fp4 converter OOM-killed on dgx1 while the GLM-5.3 kindling 2x serve held ~119/121 GB. → Never run a heavy host job (conversion, big load) on a node that has a live serve. Use a clean node (we moved to dgx5, 117 GB free). The reporter backported 615's memacct to 580 for per-cgroup GPU-memory accounting (not adopted; noted).

Other forum items (noted, not lanes)

• Muse Spark Open 816B on 2 Sparks (386058): llama.cpp RPC over 200 GbE (RoCEv2), UD-IQ1_M 194.6 GiB / 6 shards, 32K ctx, q8_0 K/V, 6.8–7.0 tok/s single request. Two nodes is the floor (one can't hold weights; a third buys only context). Batch-only, llama.cpp stack — not a vLLM lane.
• DGPP thread 383406 at 108 posts (1 new since the 10-16 pass; no material change). Our pin 3ad63d6b…, HEAD observed d9311e72… — bump pending a supervised re-bench.

Muse Spark Open 816B (meta-models/Muse-Spark-Open-816B-A42B-NVFP4 + DFlash2)

Downloaded to NAS (that download is what saturated nas.lan cold-read to ~1.3 MB/s for an hour on 10-16 — NOT our cluster). Both repos present: models--meta-models--Muse-Spark-Open-816B-A42B-NVFP4 (497 GB, 127 shards, snapshot 4239e4db…) + models--nvidia--Muse-Spark-Open-816B-A42B-DFlash2 (8.2 GB draft). Also incidental models--meta-models--Muse-Glimmer-30B.

Facts (config + card): 816B total / 42B active, natively multimodal omni (text+image+video+audio → text; MuseSparkForConditionalGeneration, model_type: muse_spark, auto_map: None). text_config: 62 layers, hidden 8192, 8 experts/token, moe_intermediate 4096, ctx 1,048,576 (1M), vocab 202048. NVFP4 group_size 16, exclude layers.0., layers.61., self_attn, lm_head. Card: NVFP4 needs ≥700 GB VRAM (BF16 2 TB); frameworks SGLang/vLLM/TRT-LLM/ Unsloth/llama.cpp; reasoning_effort low/med/high/xhigh (default xhigh); Temp 1.0 / top_p 1.0 / top_k off. DFlash2 draft = DFlash2DraftModel, qwen3-based, 5 layers.

Feasibility on GB10 — BLOCKED (same class as Step-5):

• MuseSparkForConditionalGeneration is NOT in vLLM v0.31.0 nor main — the registry only has MuseGlimmer* (the 30B model); only muse_glimmer.py exists.
• The single MuseSpark PR (#62064, open) targets Ampere (sm80) + AutoRound/ GPTQ INT4 experts — a different quant (not our NVFP4) and arch (not SM121). Even merged it would not run this NVFP4 checkpoint on GB10.
• 497 GB weights is tight even on 8 Sparks (8×119 = 952 GB; 497 weights + KV + multimodal encoders + activations); vision/audio towers + any sparse-attention kernels have no SM121 build (Step-5 DSA precedent).
• Multimodal omni → our tool-eval-bench is text-only; quality gate partial. → Watch item, not a night-shift lane. Re-check when a vLLM PR lands MuseSpark with NVFP4 + SM121 (and a DFlash2-for-MuseSpark target). Track #62064.

Runnable Muse-family alternative NOW: meta-models/Muse-Glimmer-30B IS supported in vLLM v0.31.0 (MuseGlimmerForCausalLM / MuseGlimmerForConditionalGeneration

• DFlashMuseGlimmerAssistantModel speculator via qwen3_dflash) and is already on NAS. If we want a Muse lane this shift, Glimmer-30B (single Spark) is the one that actually loads — not the 816B.

SGLang is NOT a workaround today (but is the promising track)

We are not vLLM-locked — sm121 has engine = "sglang" first-class. Checked SGLang (latest release v0.5.21, 2026-10-02):

• MuseSpark: not in a release, not in main (models/muse_spark.py = 404; only muse_glimmer.py). BUT the monolithic PR #44530 is being landed as a 5-layer series: #44664 (Docker) merged, #44665 (kernels) merged (456231e2), #44666 (srt model layer, base of 4/5), #44667 (ROCm, 4/5) open,
  • a 5/5 CUDA/deploy layer. So the model layer is in-flight, not merged. A SGLang cookbook for Muse Spark Open is already merged (#44654). → SGLang will likely run Muse-816B before vLLM does (vLLM's only MuseSpark PR is sm80+INT4). Watch the #44530 series; retry SGLang once muse_spark.py lands in a release.
• Step-5: also not in SGLang — SGLang has step3_vl, step3p5, step3p7 (Step3 family) but no Step5 arch. Same blocker as vLLM.
• SGLang DOES have bailing_moe_v3.py (Ling/BailingMoeV3) — a real alternative engine to A/B our Ling fp8/fp4 serve against vLLM (not a blocker, an option).

eugr/spark-vllm-b12x nightly checked — still no MuseSpark / Step5

Pulled the digest-pinned b12x image (sha256:03a95654…) and dumped its registry: it is a nightly vLLM 0.1.dev21830+g8244d1c44 built 2026-10-15 (ahead of v0.31.0), 391 archs. Yet:

• MUSE: only MuseGlimmer* — no MuseSparkForConditionalGeneration.
• STEP: Step1, Step3Text, Step3p5, StepVL, Step3VL, Step3p7, Step3p5MTP — no Step5ForConditionalGeneration. So Muse-816B and Step-5 are unsupported across vLLM stable, vLLM main, the b12x nightly, AND SGLang — the b12x build is not ahead on these archs. Confirms the SGLang #44530 series is the earliest path for Muse-816B.
• Useful find in the b12x nightly: DSpark speculators Glm53DSparkForCausalLM, Qwen3DSparkModel, Qwen3OmniDSparkModel, K3DSparkModel, Gemma4DSparkModel, DSparkV41DraftModel — a DSpark spec-decode path for GLM-5.3 (and others) available in the b12x image; candidate for our kindling/GLM lanes.

Ling-3.1-flash fp8→fp4 (our own converter — in progress)

No official 3.1-flash fp4 exists (inclusionAI has fp4 only for 3.0). We wrote a converter (fp8-block routed experts → MXFP4, "even" scale mode, byte-exact vs vLLM reference_mxfp4.py dq). Prototype validated (my dq == reference dq, relerr 0.0000; ~10.8% mxfp4 loss vs bf16; experts ~53% of fp8 bytes). Full 54-shard conversion running on dgx5 → NAS models--inclusionAI--Ling-3.1-flash-fp4. Target ~110 GB → one Spark (vs 534 GB fp8 → 8). Serve + bench next. Format + math spec: specs/tech-architecture/upstream-watch-2026-10-16.md ("making 3.1-flash smaller").