# Night shift — 2026-10-16 (autonomous) Plan: `2026-10-16-night-plan.md`. Full autonomy. Cluster powered on at start (dgx1/2 on 2nodes-dac, dgx5-8 on dgx58-switch running MiMo 4x; dgx3 idle, dgx4 off). ## 1. DGPP w2 provisioning — end-to-end test (the main deliverable) Ran `sm121 serve --recipe glm53-flash-nvfp4-fp8-dgpp-2x --cluster 2nodes-dac` as the first live test of on-demand DGPP provisioning. It surfaced **four real launcher bugs**, all fixed + unit-tested (`make check` green): 1. **tee logging** — the inline `cmd && ... && echo DONE 2>&1 | tee log` piped only the final echo, so the whole build log was lost on failure. Now the build script is written to a file (heredoc) and run as `bash 2>&1 | tee `. 2. **install needs a config** — `dgpp-cluster install ` requires `--config `. The generated cluster JSON is now staged on every node before provisioning. 3. **install is head-coordinated** — it installs on rank 0 AND SSHes to peers; running it on every node double-installed the peers ("release already exists"). Provisioning now runs on **rank 0 only**. 4. **DGPP_NODES is space-separated** (was comma); `.env` staged on every node (install validates `DGPP_NODES` per node). Plus **idempotent install**: a re-serve at the same pin tolerates "already exists" via `releases | grep `. Result: provisioning completes and hands off to `dgpp-cluster up`, which runs DGPP's own preflight. The launcher side is **validated end-to-end**. Commits: launcher `89013fb` (fixes), `28f9097` (spec). **w2 does not reach serving** — DGPP's preflight gates on deployment prereqs the launcher does NOT own: (a) the `HawkBearPig/GLM-5.3-Flash-NVFP4-FP8` checkpoint is not in the node HF cache (`scripts/download_model.py --config ` on rank 0), and (b) multi-fabric hosts need `DGPP_ROCE_DEVICES` set explicitly ("the transport supports at most two lanes"). Both are DGPP-side deployment steps; deferred (a ~GLM-5.3-class model download + fabric tuning before a power-off is wasteful/unsupervised). DGPP installed cleanly on dgx1/dgx2 (marker on the head). ## 2. The two queued models — feasibility (findings spec) `specs/tech-architecture/upstream-watch-2026-10-16.md`: - **Ling-3.1-flash** = `BailingMoeV3ForCausalLM` — **vLLM v0.31.0 already registers it + `BailingMoeV3MTPModel`** (no nightly needed; the old "needs nightly" note is stale). BUT the day-0 NVFP4 cut `cbfight/...-NVFP4-lossless8` is an **empty repo** (reserved, not populated). Only loadable Ling is the 572B fp8 original (~689 GB → 8 Sparks). Prep recipe authored: `recipes/ling-31-flash-nvfp4/` (4x, stock v0.31.0, `--trust-remote-code`) — validate 0 errors, dry-run OK. Serve deferred until a real small cut lands. - **Step-5-Preview-NVFP4** = `Step5ForConditionalGeneration` (`step5_vl`) — a **vision-language** model, 604B/~473 GB. vLLM v0.31.0 has `StepVLForConditional Generation` but NOT `Step5ForConditionalGeneration` → needs the dedicated image `vllm/vllm-openai:step-5-preview`. GB10-unverified (kernels, UMA gate). Prep recipe `recipes/step5-preview-nvfp4/` (8x) — validate 0 errors, dry-run OK. Live GB10 probe deferred (473 GB + 8 Sparks, supervised). ## 3. Repo / forum mining - vLLM: v0.31.0 still latest stable; v0.32.0 only rc0/rc1/rc2. Base current. - **DGPP HEAD moved** to `d9311e72` (our pin `3ad63d6b`). Forum post #106 (the HEAD release notes): adds **dense Qwen3.8-27B**, **three quant options across Qwen + full GLM-5.3**, `DGPP_NO_SWAP=1` (systemd scope, no swap — the GB10 unified-memory fix), 1s decode-gap target. **Strong pin-bump candidate** (supervised: read changelog + re-bench). - gr8 HEAD `3df902dc` (active). tool-eval-bench latest tag = v2.8.0 (our pin). - Community 2x DGPP (post #73): GX10, driver 580.173.02, CUDA 13.0.88, one ConnectX-7 RoCE lane (200G, GID 3) via switch — matches our fabric. ## 4. End of shift Stopped MiMo 4x (`sm121 stop --cluster dgx58-switch` → all containers stopped, ledger closed). 2nodes-dac was already clean (DGPP w2 preflight failed, no containers); killed the lingering Mac serve procs and closed the DGPP w2 ledger with `runlock close`. Verified no containers on either cluster. `sm121 power off --cluster 8nodes-switch`: SSH `shutdown now` on the reachable nodes (dgx4 already down), then HA `turn_off` for all 8 node switches + `sw6_mt` + `sw7_fan`. Post-check health: **all 8 nodes offline, switch off, fan off — cluster fully powered down.** ## 5. Model probes (operator push-back: the weights were already on the NAS) The operator pointed out Ling + Step-5 were already downloaded — I had wrongly cited "download" as the blocker. Confirmed on the NAS HF cache (`hf_cache_dir`, the default serve cache): Ling fp8 **534 GB**, Step-5 NVFP4 **368 GB**, both complete snapshots. Powered the 8x cluster back on and served both. - **Ling-3.1-flash fp8 — feasible on GB10.** Rewrote the recipe to the fp8 original (`inclusionAI/Ling-3.1-flash`, 8x TP8, stock `vllm/vllm-openai:v0.31.0`, `entrypoint=[vllm,serve]` — `build_tail` needs it or the model lands in `--entrypoint`). The launcher auto-synced the image from dgx1 to the 7 peers over the fabric (no manual docker). All 8 ranks loaded 534 GB over NFS in ~40 min, reached readiness, generated. **First bench invalid**: 94/97 scenarios excluded (thinking-on default → 120 s/turn timeouts); a `--no-think` direct run then showed tool scenarios 400 (`tool_choice=auto` needs `--enable-auto-tool-choice --tool-call-parser`). So: serves + generates, but a real score needs tool-call config + thinking control (follow-up). - **Step-5-Preview-NVFP4 — ruled out on GB10.** Dedicated image `vllm/vllm-openai:step-5-preview` resolves the arch + modelopt_fp4, but the engine aborts: `Step5 DSA requires SM90 or SM103; got DeviceCapability(12,1)`. GB10 is SM121 — the sparse-attention kernel isn't built for it. Hard blocker. Fixed a launcher gap en route: vllm recipes must set `entrypoint=[vllm,serve]` (`build_tail` only prepends `recipe.entrypoint`; empty → the model becomes the `--entrypoint`). Recorded in both recipes + the findings spec. Stopped both serves, closed ledgers, powered the cluster back off.