|
@@ -0,0 +1,80 @@
|
|
|
|
|
+# Night shift — 2026-10-16 (autonomous)
|
|
|
|
|
+
|
|
|
|
|
+Plan: `2026-10-16-night-plan.md`. Full autonomy. Cluster powered on at start
|
|
|
|
|
+(dgx1/2 on 2nodes-dac, dgx5-8 on dgx58-switch running MiMo 4x; dgx3 idle,
|
|
|
|
|
+dgx4 off).
|
|
|
|
|
+
|
|
|
|
|
+## 1. DGPP w2 provisioning — end-to-end test (the main deliverable)
|
|
|
|
|
+
|
|
|
|
|
+Ran `sm121 serve --recipe glm53-flash-nvfp4-fp8-dgpp-2x --cluster 2nodes-dac`
|
|
|
|
|
+as the first live test of on-demand DGPP provisioning. It surfaced **four real
|
|
|
|
|
+launcher bugs**, all fixed + unit-tested (`make check` green):
|
|
|
|
|
+
|
|
|
|
|
+1. **tee logging** — the inline `cmd && ... && echo DONE 2>&1 | tee log` piped
|
|
|
|
|
+ only the final echo, so the whole build log was lost on failure. Now the
|
|
|
|
|
+ build script is written to a file (heredoc) and run as
|
|
|
|
|
+ `bash <file> 2>&1 | tee <log>`.
|
|
|
|
|
+2. **install needs a config** — `dgpp-cluster install <tarball>` requires
|
|
|
|
|
+ `--config <json>`. The generated cluster JSON is now staged on every node
|
|
|
|
|
+ before provisioning.
|
|
|
|
|
+3. **install is head-coordinated** — it installs on rank 0 AND SSHes to peers;
|
|
|
|
|
+ running it on every node double-installed the peers ("release already
|
|
|
|
|
+ exists"). Provisioning now runs on **rank 0 only**.
|
|
|
|
|
+4. **DGPP_NODES is space-separated** (was comma); `.env` staged on every node
|
|
|
|
|
+ (install validates `DGPP_NODES` per node). Plus **idempotent install**: a
|
|
|
|
|
+ re-serve at the same pin tolerates "already exists" via
|
|
|
|
|
+ `releases | grep <rev-abbrev>`.
|
|
|
|
|
+
|
|
|
|
|
+Result: provisioning completes and hands off to `dgpp-cluster up`, which runs
|
|
|
|
|
+DGPP's own preflight. The launcher side is **validated end-to-end**. Commits:
|
|
|
|
|
+launcher `89013fb` (fixes), `28f9097` (spec).
|
|
|
|
|
+
|
|
|
|
|
+**w2 does not reach serving** — DGPP's preflight gates on deployment prereqs the
|
|
|
|
|
+launcher does NOT own: (a) the `HawkBearPig/GLM-5.3-Flash-NVFP4-FP8` checkpoint
|
|
|
|
|
+is not in the node HF cache (`scripts/download_model.py --config <file>` on rank
|
|
|
|
|
+0), and (b) multi-fabric hosts need `DGPP_ROCE_DEVICES` set explicitly ("the
|
|
|
|
|
+transport supports at most two lanes"). Both are DGPP-side deployment steps;
|
|
|
|
|
+deferred (a ~GLM-5.3-class model download + fabric tuning before a power-off is
|
|
|
|
|
+wasteful/unsupervised). DGPP installed cleanly on dgx1/dgx2 (marker on the head).
|
|
|
|
|
+
|
|
|
|
|
+## 2. The two queued models — feasibility (findings spec)
|
|
|
|
|
+
|
|
|
|
|
+`specs/tech-architecture/upstream-watch-2026-10-16.md`:
|
|
|
|
|
+
|
|
|
|
|
+- **Ling-3.1-flash** = `BailingMoeV3ForCausalLM` — **vLLM v0.31.0 already
|
|
|
|
|
+ registers it + `BailingMoeV3MTPModel`** (no nightly needed; the old "needs
|
|
|
|
|
+ nightly" note is stale). BUT the day-0 NVFP4 cut `cbfight/...-NVFP4-lossless8`
|
|
|
|
|
+ is an **empty repo** (reserved, not populated). Only loadable Ling is the 572B
|
|
|
|
|
+ fp8 original (~689 GB → 8 Sparks). Prep recipe authored:
|
|
|
|
|
+ `recipes/ling-31-flash-nvfp4/` (4x, stock v0.31.0, `--trust-remote-code`) —
|
|
|
|
|
+ validate 0 errors, dry-run OK. Serve deferred until a real small cut lands.
|
|
|
|
|
+- **Step-5-Preview-NVFP4** = `Step5ForConditionalGeneration` (`step5_vl`) — a
|
|
|
|
|
+ **vision-language** model, 604B/~473 GB. vLLM v0.31.0 has `StepVLForConditional
|
|
|
|
|
+ Generation` but NOT `Step5ForConditionalGeneration` → needs the dedicated
|
|
|
|
|
+ image `vllm/vllm-openai:step-5-preview`. GB10-unverified (kernels, UMA gate).
|
|
|
|
|
+ Prep recipe `recipes/step5-preview-nvfp4/` (8x) — validate 0 errors, dry-run
|
|
|
|
|
+ OK. Live GB10 probe deferred (473 GB + 8 Sparks, supervised).
|
|
|
|
|
+
|
|
|
|
|
+## 3. Repo / forum mining
|
|
|
|
|
+
|
|
|
|
|
+- vLLM: v0.31.0 still latest stable; v0.32.0 only rc0/rc1/rc2. Base current.
|
|
|
|
|
+- **DGPP HEAD moved** to `d9311e72` (our pin `3ad63d6b`). Forum post #106 (the
|
|
|
|
|
+ HEAD release notes): adds **dense Qwen3.8-27B**, **three quant options across
|
|
|
|
|
+ Qwen + full GLM-5.3**, `DGPP_NO_SWAP=1` (systemd scope, no swap — the GB10
|
|
|
|
|
+ unified-memory fix), 1s decode-gap target. **Strong pin-bump candidate**
|
|
|
|
|
+ (supervised: read changelog + re-bench).
|
|
|
|
|
+- gr8 HEAD `3df902dc` (active). tool-eval-bench latest tag = v2.8.0 (our pin).
|
|
|
|
|
+- Community 2x DGPP (post #73): GX10, driver 580.173.02, CUDA 13.0.88, one
|
|
|
|
|
+ ConnectX-7 RoCE lane (200G, GID 3) via switch — matches our fabric.
|
|
|
|
|
+
|
|
|
|
|
+## 4. End of shift
|
|
|
|
|
+
|
|
|
|
|
+Stopped MiMo 4x (`sm121 stop --cluster dgx58-switch` → all containers stopped,
|
|
|
|
|
+ledger closed). 2nodes-dac was already clean (DGPP w2 preflight failed, no
|
|
|
|
|
+containers); killed the lingering Mac serve procs and closed the DGPP w2 ledger
|
|
|
|
|
+with `runlock close`. Verified no containers on either cluster.
|
|
|
|
|
+
|
|
|
|
|
+`sm121 power off --cluster 8nodes-switch`: SSH `shutdown now` on the reachable
|
|
|
|
|
+nodes (dgx4 already down), then HA `turn_off` for all 8 node switches +
|
|
|
|
|
+`sw6_mt` + `sw7_fan`. Post-check health: **all 8 nodes offline, switch off, fan
|
|
|
|
|
+off — cluster fully powered down.**
|
