name: serve-boot-triage

description: Fast triage and recovery of sm121 serve launches — a container that died, a node missing its container, or an engine that never reaches readiness. Log preservation (--keep), detached serve, per-node state matrix, ledger hygiene, GB10 UMA memory gate, --speculative-config quoting. Use whenever a serve boot fails or a node has no container, before waiting 10+ minutes.

Serve boot triage

Goal: within ~5 minutes of a failed boot, know (a) which node and (b) why, with the crash log in hand — instead of a dead --rm container and a guess.

1. Log preservation (the #1 time sink)

Default serve runs docker run --rm: a crashed container is gone WITH ITS LOGS. For any first / unproven launch:

sm121 serve ... --keep — containers are NOT auto-removed on stop or crash; sm121 logs --recipe R --node N.lan then works on the corpse.
Cleanup afterwards: sm121 stop stops but keeps (--keep semantics); remove the leftovers with docker rm per node (note them in the report).

2. Detached serve (the harness kills long foreground calls)

A long foreground sm121 serve gets "Command aborted" by the tool harness; the process dies mid-launch → "containers on 2 of 4 nodes", lost logs, unclear ledger. Pattern (short call, returns immediately):

setsid nohup ~/.cargo/bin/sm121 serve --recipe R --cluster C --no-wait --keep \
  > /tmp/serve-R.log 2>&1 < /dev/null & sleep 2; echo started

--no-wait skips the 30-min readiness poll; do your own polling (crashes are caught sooner anyway). Poll with SHORT calls: tail /tmp/serve-R.log + per-node docker ps -a --filter name=<recipe>.

3. Per-node state matrix (first thing, ~10 s)

for ip in <ssh_ip from config.toml>; do
  ssh -i ~/.ssh/id_ed25519_shared -o IdentitiesOnly=yes kilork@$ip \
    'docker ps -a --filter name=<recipe-prefix> --format "{{.Names}} {{.Status}}"'
done

State	Meaning	Action
Up on all	boot in progress (weights from NAS 25–40 min; first autotune +15–40)	poll API: `curl -s http://<head_ib_ip>:30000/v1/models`
Up on some, ABSENT (not even in `ps -a`) on others	serve process died first (aborted call) OR image missing	check `docker images` on the absent node; then full clean re-serve (stop → fresh detached) or surgical `sm121 serve --recipe R --cluster C --node <n.lan> --no-wait --keep` (only if the ledger run is closed)
`Exited (2)` (with `--keep`)	engine crashed	`sm121 logs --recipe R --node <n.lan>` → last lines are the cause
Up on all, no API after 45+ min	NCCL hang or slow NFS	tail each node's logs: NCCL hang = all quiet at the same step; check `df`/NFS on the head

4. Hard rules (learned the hard way)

Node IPs only from config.toml (ssh_ip), never from memory. A wrong IP looks exactly like "the node hung" (ssh timeout/refused). Current (2026-10-02): dgx1 88.14, dgx2 88.10, dgx3 88.50, dgx4 88.51, dgx5 88.43, dgx6 88.44, dgx7 88.54, dgx8 88.55; ib 192.168.200.x; DAC (dgx1/2) 192.168.100.x.
Multi-line remote scripts: base64 them (printf '%s' "$S" | base64 | ssh ... "bash -c 'base64 -d | bash'") — nested quotes in local double-quotes + remote zsh single-quotes break.
sm121 runlock list — an open run = a serve that never closed; sm121 runlock close <recipe> after manual cleanup.
Do NOT duplicate launcher built-ins: drop_caches (config field, default true — already runs before every serve), config-level before_serve.
sm121 -d serve ... (dry run) renders the exact per-node commands — use it to predict, not to execute (its shell escaping is not paste-able).

4b. Long-running local work (benches, builds) — move to a node

The tool harness reaps long foreground calls (~10 min) and detached local processes (nohup/setsid die with the call). Benches run 1.5–2 h. Pattern:

Ship the launcher to a cluster node and build it there (the Mac binary is Mach-O; nodes are aarch64 Linux): rustup one-liner (minimal profile, ~1 min) + cargo build --release (deps ~5 min, incremental ~15 s).
Ship the minimal repo subset — src Cargo.toml Cargo.lock locales host-scripts (~400 KB). NEVER tar the whole repo: target/ is tens of GB.
Fix ssh_key_path in the node's config.toml (Mac path → node key, ~/.ssh/id_ed25519_shared).
Install the harness on the node: uv installer + uv tool install "tool-eval-bench[perf] @ git+...@<pinned-rev>"; add ~/.local/bin to PATH inside the tmux command (non-interactive sessions don't source the installer's profile).
Run under tmux new -d -s <name>; poll with short calls: tail -c 2000 /tmp/<name>.log | tr \\r \\n | tail.
--tokenizer must be a host-side path readable by the bench user (the NAS path, not the container-side /root/.cache/... — containers are root, the host user is not).
Reports land in the node's recipes/<lane>/reports/; copy them back to the Mac repo and commit (same stem).

5. Failure patterns (GB10 / kindling era)

UMA memory gate (vLLM #48140): Free memory on device cuda:0 (106.53/121.63 GiB) ... less than desired GPU memory utilization (0.88, 107.03 GiB) — random node, <1 GiB margin; the check reads MemFree at worker init; weight/NFS page cache tips it. Built-in drop_caches covers the common case; recurrence → recipe before_serve drop_caches (on_failure = "warn") or GMU −0.02.
--speculative-config quoting: the TOML value must be """ + newline + '{JSON}' + """ with the closing """ on the same line as the JSON (mimo-recipe form). A trailing newline before """ → the launcher keeps the apostrophes → vLLM cannot be converted to <function loads>. A doubled --speculative-config (recipe args + transcribed env EXTRA) → vLLM takes the last one = garbage.
X/gdm running on a node = extra GBs + a display context: headless = sudo systemctl set-default multi-user.target && sudo systemctl stop gdm (dgx1–5 already headless; dgx6/7/8 switched 2026-10-02).
First launch after an image build = autotune candidate runs (394 on the 8x b12x stack) + NAS weight read: budget 60–90 min total; don't declare failure early.
The engine's own preflight (kindling entrypoint) is bypassed in B-mode; their preflight checks (fabric, GID) are the first things to replicate manually if a boot fails without a clear vLLM traceback.

SKILL.md 6.3 KB Riwayat Mentahan