|
|
@@ -0,0 +1,71 @@
|
|
|
+# Shift 2026-09-25 (day) — MiMo-V2.6-Pro-RL 8× Spark bring-up
|
|
|
+
|
|
|
+## Context
|
|
|
+
|
|
|
+User: new model to try on 8 nodes — rhys101's SGLang variant of the NVIDIA
|
|
|
+forum thread 384064 (github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8, build a14).
|
|
|
+Also: user reverted dgx1's HF symlink to the NAS (NAS is back, 7.4T with data),
|
|
|
+and wiped all docker images from dgx1.
|
|
|
+
|
|
|
+## Power + state survey
|
|
|
+
|
|
|
+- All 8 nodes powered on (`sm121 power on --cluster 8nodes-switch`, HA relays
|
|
|
+ incl. sw6_mt for the MikroTik). NAS read speed single-stream ~171 MB/s.
|
|
|
+- Symlink matrix: dgx1 → NAS (user-fixed); dgx2–4 → hf-local (as before);
|
|
|
+ dgx5–8 → NAS (never changed). `--hf-cache` flag (new, `51b7404`) makes the
|
|
|
+ symlink question moot for a given serve.
|
|
|
+- Images: dgx1 wiped (only sglang dev-qwen38-next-local left); dgx2 the
|
|
|
+ richest (r35/r38/r39/lab/karmic); dgx5–8 stale (r27/r35/r36/r37 + dsv41).
|
|
|
+
|
|
|
+## MiMo bring-up (rhys101 repo @ 2f3e5fd, build a14)
|
|
|
+
|
|
|
+- Model already on the NAS: `XiaomiMiMo/MiMo-V2.6-Pro-RL` @ `73875d00` (the
|
|
|
+ exact pinned revision), 146 top-level files, `dflash/` subfolder present.
|
|
|
+ Not on any node locally (535 GB/rank would not fit on dgx1–3/5 anyway).
|
|
|
+- Vendored the build tree into `docker-images/mimo26-spark/` (Dockerfile +
|
|
|
+ 12 patches + third_party/b12x + runtime + fp8 tile tables + versions.env
|
|
|
+ pinning the base digest + versions.lock.json). `d5663ec`.
|
|
|
+- Recipe `mimo26-pro-rl-8x`: sglang, TP8 (launcher-generated via
|
|
|
+ `distributed_backend = "mp"`), all of the repo's SERVE_ENV flags as `args`,
|
|
|
+ `{{dflash}}` path ref for the draft weights, env: SGLANG8_ROCE_ALLREDUCE +
|
|
|
+ MIMO26_* + NCCL extras.
|
|
|
+- **TOML trap hit (again):** `args = [...]` written after `[[extra_mounts]]`
|
|
|
+ parses *into* the extra_mounts table — args silently dropped from the
|
|
|
+ docker command. Reordered (bare keys first). Lesson: in recipe TOML, bare
|
|
|
+ keys always precede table headers.
|
|
|
+- Launcher change: dry-run `--speculative-draft-model-path` now shows the
|
|
|
+ raw `{{name}}` placeholder (the old "descriptive path" was substituted
|
|
|
+ into the dry-run command, misleading); `substitute_path_refs` skips a
|
|
|
+ name==path pair. Removed dead `path_ref_placeholder`.
|
|
|
+- Build on dgx4.lan: 14m20s, label `5b92eae1`. Note: `--build-node dgx4`
|
|
|
+ does NOT work — node names carry the `.lan` suffix (`dgx4.lan`).
|
|
|
+- Sync dgx4→7 peers sequential, ~4.5–6 min each (~35 min total).
|
|
|
+- Serve with `--hf-cache /mnt/nas/ai/huggingface --keep`: all 8 containers up
|
|
|
+ at 10:44. Model loads **from the NAS per rank** (535 GB each, ~110 MB/s
|
|
|
+ per rank, ~1h+ wall). The launcher's 1800 s readiness wait timed out as
|
|
|
+ expected (run closed at 1801 s in the ledger — cosmetic; containers stay
|
|
|
+ with `--keep`). API came up on its own.
|
|
|
+
|
|
|
+## Smoke (all pass)
|
|
|
+
|
|
|
+- Text + reasoning split: reasoning/content channels clean, `mimo` parsers OK
|
|
|
+- Tool call: `get_weather({"city":"Paris"})` exact
|
|
|
+- Vision: red/blue 64×32 test → "Red" (correct; image_tokens 8)
|
|
|
+- Decode: **~50 tok/s C1** (512 tok, two runs: 50.0 / 49.6) — consistent with
|
|
|
+ the repo's a13 C1 per-category range (prose 40.3 … format 99.5, mean 69.1)
|
|
|
+
|
|
|
+## Next / open
|
|
|
+
|
|
|
+- Run ledger closed (1801 s timeout); `sm121 stop` still works via labels.
|
|
|
+- Candidate follow-ups: node-to-node fan-out of the model for fast re-boots
|
|
|
+ (fits only on dgx4/6/7/8 + tight elsewhere); bench (repo's mimobench.py vs
|
|
|
+ our tool-eval-bench); C8 aggregate.
|
|
|
+
|
|
|
+## Ops notes
|
|
|
+
|
|
|
+- `--build-node` takes the full node name (`dgx4.lan`), not the bare host.
|
|
|
+- A serve that outgrows the 1800 s readiness wait fails the launcher but,
|
|
|
+ with `--keep`, leaves the containers running and the API functional.
|
|
|
+- py-spy (pip in-container) + `cat /proc/<pid>/io` is the fast way to tell
|
|
|
+ "loading slowly" from "stuck" — check the WORKER pid (sglang::scheduler_TP0),
|
|
|
+ not the parent (pid 54) which only reads config.
|
