Shift 2026-09-25 (day) — MiMo-V2.6-Pro-RL 8× Spark bring-up

Context

User: new model to try on 8 nodes — rhys101's SGLang variant of the NVIDIA forum thread 384064 (github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8, build a14). Also: user reverted dgx1's HF symlink to the NAS (NAS is back, 7.4T with data), and wiped all docker images from dgx1.

Power + state survey

All 8 nodes powered on (sm121 power on --cluster 8nodes-switch, HA relays incl. sw6_mt for the MikroTik). NAS read speed single-stream ~171 MB/s.
Symlink matrix: dgx1 → NAS (user-fixed); dgx2–4 → hf-local (as before); dgx5–8 → NAS (never changed). --hf-cache flag (new, 51b7404) makes the symlink question moot for a given serve.
Images: dgx1 wiped (only sglang dev-qwen38-next-local left); dgx2 the richest (r35/r38/r39/lab/karmic); dgx5–8 stale (r27/r35/r36/r37 + dsv41).

MiMo bring-up (rhys101 repo @ `2f3e5fd`, build a14)

Model already on the NAS: XiaomiMiMo/MiMo-V2.6-Pro-RL @ 73875d00 (the exact pinned revision), 146 top-level files, dflash/ subfolder present. Not on any node locally (535 GB/rank would not fit on dgx1–3/5 anyway).
Vendored the build tree into docker-images/mimo26-spark/ (Dockerfile + 12 patches + third_party/b12x + runtime + fp8 tile tables + versions.env pinning the base digest + versions.lock.json). d5663ec.
Recipe mimo26-pro-rl-8x: sglang, TP8 (launcher-generated via distributed_backend = "mp"), all of the repo's SERVE_ENV flags as args, {{dflash}} path ref for the draft weights, env: SGLANG8_ROCEALLREDUCE + MIMO26* + NCCL extras.
TOML trap hit (again): args = [...] written after [[extra_mounts]] parses into the extra_mounts table — args silently dropped from the docker command. Reordered (bare keys first). Lesson: in recipe TOML, bare keys always precede table headers.
Launcher change: dry-run --speculative-draft-model-path now shows the raw {{name}} placeholder (the old "descriptive path" was substituted into the dry-run command, misleading); substitute_path_refs skips a name==path pair. Removed dead path_ref_placeholder.
Build on dgx4.lan: 14m20s, label 5b92eae1. Note: --build-node dgx4 does NOT work — node names carry the .lan suffix (dgx4.lan).
Sync dgx4→7 peers sequential, ~4.5–6 min each (~35 min total).
Serve with --hf-cache /mnt/nas/ai/huggingface --keep: all 8 containers up at 10:44. Model loads from the NAS per rank (535 GB each, ~110 MB/s per rank, ~1h+ wall). The launcher's 1800 s readiness wait timed out as expected (run closed at 1801 s in the ledger — cosmetic; containers stay with --keep). API came up on its own.

Smoke (all pass)

Text + reasoning split: reasoning/content channels clean, mimo parsers OK
Tool call: get_weather({"city":"Paris"}) exact
Vision: red/blue 64×32 test → "Red" (correct; image_tokens 8)
Decode: ~50 tok/s C1 (512 tok, two runs: 50.0 / 49.6) — consistent with the repo's a13 C1 per-category range (prose 40.3 … format 99.5, mean 69.1)

Bench (tool-eval-bench v2.7.0, seed 42, hardmode, perf, c1,2) — 11:54 UTC

165/176 (94/100 quality, ★★★★★) — on par with the agent-tuned GLM-5.3 anchors (164–165). Fails: TC-03 (implicit tool need, wrong_args), TC-68 (schema violation, invalid JSON — the recurring one on GLM too). Partials: TC-21/30/39/52/55/57/60 (mostly "did the work, missed a tool-use nuance" + the two injection-repro cases). Note: GLM's recurring TC-43/45/51 all PASS here.

Perf (pp2048/tg128): pp 1,687–1,940 t/s; tg c1 50–54 t/s, c2 65–69 t/s (per-stream); TTFT 1.3 s (d0) … 9.6 s (d8192 c2). Consistent with the repo's a13 numbers (prefill 1,880 at 5K; C1 mean 69.1 across long 512-tok generations — ours is 128-tok, DFlash-adaptive, so slightly lower tg is expected).

Next / open

Run ledger closed (1801 s timeout); sm121 stop still works via labels.
Candidate follow-ups: node-to-node fan-out of the model for fast re-boots (fits only on dgx4/6/7/8 + tight elsewhere); bench (repo's mimobench.py vs our tool-eval-bench); C8 aggregate.

Ops notes

--build-node takes the full node name (dgx4.lan), not the bare host.
A serve that outgrows the 1800 s readiness wait fails the launcher but, with --keep, leaves the containers running and the API functional.
py-spy (pip in-container) + cat /proc/<pid>/io is the fast way to tell "loading slowly" from "stuck" — check the WORKER pid (sglang::scheduler_TP0), not the parent (pid 54) which only reads config.

2026-09-25.md 4.5 KB Cronologia Originale