# Upstream watch — 2026-10-17 (night-shift research pass)

Pass across vLLM releases, the NVIDIA forums, and HF model cards, feeding the
night-shift agenda (`history/night-shifts/2026-10-17-night-plan.md`).

## vLLM release status — base stays v0.31.0
- Latest **release** is still **v0.31.0** (2026-10-05). v0.32.0 is only at
  **rc2** (rc0/rc1/rc2 tags, no stable). No re-pin of the kindling base yet.
- Keep watching for v0.32.0 stable (would trigger the base re-pin + overlay
  rebase + flashinfer bump, bench-gated vs the 95/100 kindling 2x anchor).

## New NVIDIA-official FP4 checkpoints (all need the b12x container)
NVIDIA shipped a family of "…-FP4-Local" QAD checkpoints quantized with
**nvidia-modelopt v0.47.0** via **quantization-aware distillation (QAD)**. Their
NVFP4 scale factors use a **new lossless-compression format that upstream vLLM
does NOT yet support** — you MUST run the `eugr/spark-vllm-b12x` container
(b12x backend + `--load-format b12x`), not stock vLLM. Digest-pinned
`eugr/spark-vllm-b12x@sha256:03a95654…bb3595` (arm64, built 2026-10-15).

- **nvidia/GLM-5.3-Flash-FP4-Local** — official NVFP4 GLM-5.3-Flash. **2x DGX
  Spark**, 500K ctx, MTP n=5 (`moe_backend:humming, attention_backend:B12X`),
  FP8 KV (~2M tokens), `--attention-backend B12X --block-size 256
  --moe/--linear/--load-format b12x`, `--reasoning-parser glm45
  --tool-call-parser glm47`, GMU 0.87. Card quality 91.3% / 92.4%. This is the
  official NVFP4 counterpart to our kindling 2x lane (our own r35 build) — a
  direct A/B vs the 95/100 anchor. Recipe: `glm53-flash-fp4-local-2x` (recipes
  `d5276db`). NOTE: 2x → needs 2 free Sparks; 2nodes-dac is busy with the live
  kindling serve.
- **nvidia/Qwen3.8-Flash-Next-FP4-Local** — 1x Spark, b12x, MTP n=4, FP8 KV,
  262K, GMU 0.8, `VLLM_ENGINE_READY_TIMEOUT_S=3600` (b12x autotune > 600 s).
  Forum hardmode 89/100, ~49 t/s decode. Recipe `qwen38-flash-next-fp4-local`.
- **nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4** (75 GB, 1x Spark, VL,
  1M ctx, LiLiCorr dflash-7 drafter, mamba flags, `qwen3_coder`/`nemotron_v3`)
  and **Nemotron-3.5-Bolt-74B-A9B-24GB** (24 GB, QAD 2-bit experts, 1x Spark,
  `ultra_v3` parser plugin). Recipes `nemotron-35-super-vl-120b-nvfp4`,
  `nemotron-35-bolt-74b-24gb`. Forum: Super-VL 90/100, Bolt 88/100.

## GB10 hardware gotchas (both directly bit us this session)
- **Stuck GPU clock (forum 386060, confirms 376039/376239/366944/…)**: GB10 SM
  clock can stick at ~660 MHz (max 3003, app clock 2418) at ~16 W with
  **throttle reasons 0x0**; `nvidia-smi -lgc` is accepted but ignored and a
  **reboot does not clear it**. Symptom = anomalously low throughput that looks
  like "hot state" but is a hard clock cap. **Remedy** (community, not NVIDIA-
  documented): power off, **unplug AC + USB**, hold the power button ~10 s, wait
  a few minutes, reconnect (one report cleared to 2080 MHz / 77 TFLOPS). → When
  a lane underperforms, check SM clock under load (expect ~2000–2400, not ~660)
  before blaming the recipe; a smart-plug cycle may not be enough (needs the
  hold-power-button drain).
- **GPU/unified memory is NOT cgroup-contained on GB10 (forum 386057)**: with
  the 580 driver `nvidia-smi` reports `Memory-Usage: Not Supported` on the iGPU,
  and `docker --memory` / `MemoryMax=` do **not** cap `cudaMalloc`/unified
  memory. When the unified pool is exhausted the **whole host stalls** (not just
  the container). This is exactly why the Ling fp4 converter OOM-killed on dgx1
  while the GLM-5.3 kindling 2x serve held ~119/121 GB. → **Never run a heavy
  host job (conversion, big load) on a node that has a live serve.** Use a clean
  node (we moved to dgx5, 117 GB free). The reporter backported 615's `memacct`
  to 580 for per-cgroup GPU-memory accounting (not adopted; noted).

## Other forum items (noted, not lanes)
- **Muse Spark Open 816B on 2 Sparks (386058)**: llama.cpp RPC over 200 GbE
  (RoCEv2), UD-IQ1_M 194.6 GiB / 6 shards, 32K ctx, q8_0 K/V, **6.8–7.0 tok/s**
  single request. Two nodes is the floor (one can't hold weights; a third buys
  only context). Batch-only, llama.cpp stack — not a vLLM lane.
- DGPP thread 383406 at 108 posts (1 new since the 10-16 pass; no material
  change). Our pin `3ad63d6b…`, HEAD observed `d9311e72…` — bump pending a
  supervised re-bench.

## Muse Spark Open 816B (meta-models/Muse-Spark-Open-816B-A42B-NVFP4 + DFlash2)
Downloaded to NAS (that download is what saturated `nas.lan` cold-read to ~1.3 MB/s
for an hour on 10-16 — NOT our cluster). Both repos present:
`models--meta-models--Muse-Spark-Open-816B-A42B-NVFP4` (497 GB, 127 shards,
snapshot `4239e4db…`) + `models--nvidia--Muse-Spark-Open-816B-A42B-DFlash2`
(8.2 GB draft). Also incidental `models--meta-models--Muse-Glimmer-30B`.

**Facts (config + card):** 816B total / 42B active, **natively multimodal omni**
(text+image+video+audio → text; `MuseSparkForConditionalGeneration`,
`model_type: muse_spark`, `auto_map: None`). text_config: 62 layers, hidden 8192,
8 experts/token, moe_intermediate 4096, **ctx 1,048,576 (1M)**, vocab 202048.
NVFP4 `group_size 16`, exclude `layers.0.`, `layers.61.`, `self_attn`, `lm_head`.
Card: **NVFP4 needs ≥700 GB VRAM** (BF16 2 TB); frameworks SGLang/vLLM/TRT-LLM/
Unsloth/llama.cpp; `reasoning_effort` low/med/high/xhigh (default **xhigh**);
Temp 1.0 / top_p 1.0 / top_k off. DFlash2 draft = `DFlash2DraftModel`, qwen3-based,
5 layers.

**Feasibility on GB10 — BLOCKED (same class as Step-5):**
- `MuseSparkForConditionalGeneration` is **NOT in vLLM v0.31.0 nor main** — the
  registry only has `MuseGlimmer*` (the 30B model); only `muse_glimmer.py` exists.
- The single MuseSpark PR (**#62064, open**) targets **Ampere (sm80) + AutoRound/
  GPTQ INT4 experts** — a different quant (not our NVFP4) and arch (not SM121).
  Even merged it would not run this NVFP4 checkpoint on GB10.
- 497 GB weights is tight even on 8 Sparks (8×119 = 952 GB; 497 weights + KV +
  multimodal encoders + activations); vision/audio towers + any sparse-attention
  kernels have no SM121 build (Step-5 DSA precedent).
- Multimodal omni → our `tool-eval-bench` is text-only; quality gate partial.
→ **Watch item, not a night-shift lane.** Re-check when a vLLM PR lands MuseSpark
with NVFP4 + SM121 (and a DFlash2-for-MuseSpark target). Track #62064.

**Runnable Muse-family alternative NOW:** `meta-models/Muse-Glimmer-30B` IS
supported in vLLM v0.31.0 (`MuseGlimmerForCausalLM` / `MuseGlimmerForConditionalGeneration`
+ `DFlashMuseGlimmerAssistantModel` speculator via `qwen3_dflash`) and is already
on NAS. If we want a Muse lane this shift, Glimmer-30B (single Spark) is the one
that actually loads — not the 816B.

### SGLang is NOT a workaround today (but is the promising track)
We are not vLLM-locked — sm121 has `engine = "sglang"` first-class. Checked
SGLang (latest release **v0.5.21**, 2026-10-02):
- **MuseSpark: not in a release, not in `main`** (`models/muse_spark.py` = 404;
  only `muse_glimmer.py`). BUT the monolithic PR **#44530 is being landed as a
  5-layer series**: #44664 (Docker) **merged**, #44665 (kernels) **merged**
  (`456231e2`), #44666 (srt model layer, base of 4/5), #44667 (ROCm, 4/5) open,
  + a 5/5 CUDA/deploy layer. So the model layer is in-flight, not merged. A
  SGLang **cookbook for Muse Spark Open** is already merged (#44654). → SGLang
  will likely run Muse-816B **before** vLLM does (vLLM's only MuseSpark PR is
  sm80+INT4). **Watch the #44530 series; retry SGLang once `muse_spark.py` lands
  in a release.**
- **Step-5: also not in SGLang** — SGLang has `step3_vl`, `step3p5`, `step3p7`
  (Step3 family) but no Step5 arch. Same blocker as vLLM.
- SGLang DOES have `bailing_moe_v3.py` (Ling/BailingMoeV3) — a real alternative
  engine to A/B our Ling fp8/fp4 serve against vLLM (not a blocker, an option).

### eugr/spark-vllm-b12x nightly checked — still no MuseSpark / Step5
Pulled the digest-pinned b12x image (`sha256:03a95654…`) and dumped its registry:
it is a **nightly vLLM `0.1.dev21830+g8244d1c44` built 2026-10-15** (ahead of
v0.31.0), **391 archs**. Yet:
- **MUSE**: only `MuseGlimmer*` — **no `MuseSparkForConditionalGeneration`**.
- **STEP**: `Step1`, `Step3Text`, `Step3p5`, `StepVL`, `Step3VL`, `Step3p7`,
  `Step3p5MTP` — **no `Step5ForConditionalGeneration`**.
So Muse-816B and Step-5 are unsupported across vLLM stable, vLLM main, the b12x
nightly, AND SGLang — the b12x build is not ahead on these archs. Confirms the
SGLang #44530 series is the earliest path for Muse-816B.
- Useful find in the b12x nightly: DSpark speculators `Glm53DSparkForCausalLM`,
  `Qwen3DSparkModel`, `Qwen3OmniDSparkModel`, `K3DSparkModel`, `Gemma4DSparkModel`,
  `DSparkV41DraftModel` — a DSpark spec-decode path for GLM-5.3 (and others)
  available in the b12x image; candidate for our kindling/GLM lanes.

## Ling-3.1-flash fp8→fp4 (our own converter — in progress)
No official 3.1-flash fp4 exists (inclusionAI has fp4 only for 3.0). We wrote a
converter (fp8-block routed experts → MXFP4, "even" scale mode, byte-exact vs
vLLM `reference_mxfp4.py` dq). Prototype validated (my dq == reference dq,
relerr 0.0000; ~10.8% mxfp4 loss vs bf16; experts ~53% of fp8 bytes). Full
54-shard conversion running on dgx5 → NAS
`models--inclusionAI--Ling-3.1-flash-fp4`. Target ~110 GB → **one Spark**
(vs 534 GB fp8 → 8). Serve + bench next. Format + math spec:
`specs/tech-architecture/upstream-watch-2026-10-16.md` ("making 3.1-flash
smaller").