upstream-watch + shift log 10-16: live model probes — Ling fp8 feasible on GB10, Step-5 SM121 DSA blocker; entrypoint=[vllm,serve] gap noted

history/night-shifts/2026-10-16.md

@@ -78,3 +78,30 @@ with `runlock close`. Verified no containers on either cluster.
 nodes (dgx4 already down), then HA `turn_off` for all 8 node switches +
 `sw6_mt` + `sw7_fan`. Post-check health: **all 8 nodes offline, switch off, fan
 off — cluster fully powered down.**
+
+## 5. Model probes (operator push-back: the weights were already on the NAS)
+
+The operator pointed out Ling + Step-5 were already downloaded — I had wrongly
+cited "download" as the blocker. Confirmed on the NAS HF cache (`hf_cache_dir`,
+the default serve cache): Ling fp8 **534 GB**, Step-5 NVFP4 **368 GB**, both
+complete snapshots. Powered the 8x cluster back on and served both.
+
+- **Ling-3.1-flash fp8 — feasible on GB10.** Rewrote the recipe to the fp8
+  original (`inclusionAI/Ling-3.1-flash`, 8x TP8, stock `vllm/vllm-openai:v0.31.0`,
+  `entrypoint=[vllm,serve]` — `build_tail` needs it or the model lands in
+  `--entrypoint`). The launcher auto-synced the image from dgx1 to the 7 peers
+  over the fabric (no manual docker). All 8 ranks loaded 534 GB over NFS in
+  ~40 min, reached readiness, generated. **First bench invalid**: 94/97 scenarios
+  excluded (thinking-on default → 120 s/turn timeouts); a `--no-think` direct run
+  then showed tool scenarios 400 (`tool_choice=auto` needs
+  `--enable-auto-tool-choice --tool-call-parser`). So: serves + generates, but a
+  real score needs tool-call config + thinking control (follow-up).
+- **Step-5-Preview-NVFP4 — ruled out on GB10.** Dedicated image
+  `vllm/vllm-openai:step-5-preview` resolves the arch + modelopt_fp4, but the
+  engine aborts: `Step5 DSA requires SM90 or SM103; got DeviceCapability(12,1)`.
+  GB10 is SM121 — the sparse-attention kernel isn't built for it. Hard blocker.
+
+Fixed a launcher gap en route: vllm recipes must set `entrypoint=[vllm,serve]`
+(`build_tail` only prepends `recipe.entrypoint`; empty → the model becomes the
+`--entrypoint`). Recorded in both recipes + the findings spec. Stopped both
+serves, closed ledgers, powered the cluster back off.

specs/tech-architecture/upstream-watch-2026-10-16.md

@@ -91,6 +91,29 @@ gates on deployment prereqs the launcher does NOT own:
   of truth for the exact env format. Candidate launcher follow-up: a
   `[dgpp].roce_devices` recipe field → emit `DGPP_ROCE_DEVICES` in the `.env`.
 
+## Live 8x probe results (ran overnight 2026-10-16, cluster powered back on)
+
+Both models were already on the NAS HF cache (Ling fp8 534 GB, Step-5 NVFP4
+368 GB) — the download was never the blocker (corrected). Served on 8nodes-switch:
+
+- **Ling-3.1-flash fp8 — FEASIBLE on GB10.** Stock `vllm/vllm-openai:v0.31.0`
+  (arch built-in), 8 ranks TP8, 534 GB loaded over NFS in ~40 min, reached
+  readiness on :30000, generated correctly. vLLM auto-selected FP8 MoE
+  (DeepGEMM/TRITON) + TRITON_MLA + FLASH_ATTN prefill; no UMA boot gate at
+  max-model-len 65536 / GMU 0.78. **A valid bench score needs serving config**:
+  `--enable-auto-tool-choice --tool-call-parser <bailing>` (tool scenarios
+  otherwise 400 on `tool_choice=auto`) + thinking control (the model's default
+  long reasoning blows the 120 s/turn timeout → 94/97 scenarios excluded, so the
+  first 67/100 is not a real quality number). Follow-up: find the bailing
+  tool-call parser, re-serve with tool config, bench with `--no-think`.
+- **Step-5-Preview-NVFP4 — NOT runnable on GB10 (hard blocker).** The dedicated
+  image resolves `Step5ForConditionalGeneration` + `modelopt_fp4`, but the engine
+  aborts at model init: `NotImplementedError: Step5 DSA requires SM90 or SM103;
+  got DeviceCapability(major=12, minor=1)`. GB10 is SM121 (CC 12.1); Step-5's
+  DeepSeek Sparse Attention kernel is built only for SM90 (Hopper) / SM103
+  (B-series Blackwell). Not a config issue — ruled out on Spark until upstream
+  ships an SM121 DSA path.
+
 ## Not done overnight (deferred, supervised)
 
 - Ling 572B-fp8 8x lane (689 GB + all 8 Sparks) and Step-5 8x GB10 probe