launcher/history/night-shifts/2026-10-16.md

Night shift — 2026-10-16 (autonomous)

Plan: 2026-10-16-night-plan.md. Full autonomy. Cluster powered on at start (dgx1/2 on 2nodes-dac, dgx5-8 on dgx58-switch running MiMo 4x; dgx3 idle, dgx4 off).

1. DGPP w2 provisioning — end-to-end test (the main deliverable)

Ran sm121 serve --recipe glm53-flash-nvfp4-fp8-dgpp-2x --cluster 2nodes-dac as the first live test of on-demand DGPP provisioning. It surfaced four real launcher bugs, all fixed + unit-tested (make check green):

1. tee logging — the inline cmd && ... && echo DONE 2>&1 | tee log piped only the final echo, so the whole build log was lost on failure. Now the build script is written to a file (heredoc) and run as bash <file> 2>&1 | tee <log>.
2. install needs a config — dgpp-cluster install <tarball> requires --config <json>. The generated cluster JSON is now staged on every node before provisioning.
3. install is head-coordinated — it installs on rank 0 AND SSHes to peers; running it on every node double-installed the peers ("release already exists"). Provisioning now runs on rank 0 only.
4. DGPP_NODES is space-separated (was comma); .env staged on every node (install validates DGPP_NODES per node). Plus idempotent install: a re-serve at the same pin tolerates "already exists" via releases | grep <rev-abbrev>.

Result: provisioning completes and hands off to dgpp-cluster up, which runs DGPP's own preflight. The launcher side is validated end-to-end. Commits: launcher 89013fb (fixes), 28f9097 (spec).

w2 does not reach serving — DGPP's preflight gates on deployment prereqs the launcher does NOT own: (a) the HawkBearPig/GLM-5.3-Flash-NVFP4-FP8 checkpoint is not in the node HF cache (scripts/download_model.py --config <file> on rank 0), and (b) multi-fabric hosts need DGPP_ROCE_DEVICES set explicitly ("the transport supports at most two lanes"). Both are DGPP-side deployment steps; deferred (a ~GLM-5.3-class model download + fabric tuning before a power-off is wasteful/unsupervised). DGPP installed cleanly on dgx1/dgx2 (marker on the head).

2. The two queued models — feasibility (findings spec)

specs/tech-architecture/upstream-watch-2026-10-16.md:

• Ling-3.1-flash = BailingMoeV3ForCausalLM — vLLM v0.31.0 already registers it + BailingMoeV3MTPModel (no nightly needed; the old "needs nightly" note is stale). BUT the day-0 NVFP4 cut cbfight/...-NVFP4-lossless8 is an empty repo (reserved, not populated). Only loadable Ling is the 572B fp8 original (~689 GB → 8 Sparks). Prep recipe authored: recipes/ling-31-flash-nvfp4/ (4x, stock v0.31.0, --trust-remote-code) — validate 0 errors, dry-run OK. Serve deferred until a real small cut lands.
• Step-5-Preview-NVFP4 = Step5ForConditionalGeneration (step5_vl) — a vision-language model, 604B/~473 GB. vLLM v0.31.0 has StepVLForConditional Generation but NOT Step5ForConditionalGeneration → needs the dedicated image vllm/vllm-openai:step-5-preview. GB10-unverified (kernels, UMA gate). Prep recipe recipes/step5-preview-nvfp4/ (8x) — validate 0 errors, dry-run OK. Live GB10 probe deferred (473 GB + 8 Sparks, supervised).

3. Repo / forum mining

• vLLM: v0.31.0 still latest stable; v0.32.0 only rc0/rc1/rc2. Base current.
• DGPP HEAD moved to d9311e72 (our pin 3ad63d6b). Forum post #106 (the HEAD release notes): adds dense Qwen3.8-27B, three quant options across Qwen + full GLM-5.3, DGPP_NO_SWAP=1 (systemd scope, no swap — the GB10 unified-memory fix), 1s decode-gap target. Strong pin-bump candidate (supervised: read changelog + re-bench).
• gr8 HEAD 3df902dc (active). tool-eval-bench latest tag = v2.8.0 (our pin).
• Community 2x DGPP (post #73): GX10, driver 580.173.02, CUDA 13.0.88, one ConnectX-7 RoCE lane (200G, GID 3) via switch — matches our fabric.

4. End of shift

Stopped MiMo 4x (sm121 stop --cluster dgx58-switch → all containers stopped, ledger closed). 2nodes-dac was already clean (DGPP w2 preflight failed, no containers); killed the lingering Mac serve procs and closed the DGPP w2 ledger with runlock close. Verified no containers on either cluster.

sm121 power off --cluster 8nodes-switch: SSH shutdown now on the reachable nodes (dgx4 already down), then HA turn_off for all 8 node switches + sw6_mt + sw7_fan. Post-check health: all 8 nodes offline, switch off, fan off — cluster fully powered down.

5. Model probes (operator push-back: the weights were already on the NAS)

The operator pointed out Ling + Step-5 were already downloaded — I had wrongly cited "download" as the blocker. Confirmed on the NAS HF cache (hf_cache_dir, the default serve cache): Ling fp8 534 GB, Step-5 NVFP4 368 GB, both complete snapshots. Powered the 8x cluster back on and served both.

• Ling-3.1-flash fp8 — feasible on GB10. Rewrote the recipe to the fp8 original (inclusionAI/Ling-3.1-flash, 8x TP8, stock vllm/vllm-openai:v0.31.0, entrypoint=[vllm,serve] — build_tail needs it or the model lands in --entrypoint). The launcher auto-synced the image from dgx1 to the 7 peers over the fabric (no manual docker). All 8 ranks loaded 534 GB over NFS in ~40 min, reached readiness, generated. First bench invalid: 94/97 scenarios excluded (thinking-on default → 120 s/turn timeouts); a --no-think direct run then showed tool scenarios 400 (tool_choice=auto needs --enable-auto-tool-choice --tool-call-parser). So: serves + generates, but a real score needs tool-call config + thinking control (follow-up).
• Step-5-Preview-NVFP4 — ruled out on GB10. Dedicated image vllm/vllm-openai:step-5-preview resolves the arch + modelopt_fp4, but the engine aborts: Step5 DSA requires SM90 or SM103; got DeviceCapability(12,1). GB10 is SM121 — the sparse-attention kernel isn't built for it. Hard blocker.

Fixed a launcher gap en route: vllm recipes must set entrypoint=[vllm,serve] (build_tail only prepends recipe.entrypoint; empty → the model becomes the --entrypoint). Recorded in both recipes + the findings spec. Stopped both serves, closed ledgers, powered the cluster back off.