|
|
@@ -0,0 +1,91 @@
|
|
|
+---
|
|
|
+name: serve-boot-triage
|
|
|
+description: Fast triage and recovery of sm121 serve launches — a container that died, a node missing its container, or an engine that never reaches readiness. Log preservation (--keep), detached serve, per-node state matrix, ledger hygiene, GB10 UMA memory gate, --speculative-config quoting. Use whenever a serve boot fails or a node has no container, before waiting 10+ minutes.
|
|
|
+---
|
|
|
+
|
|
|
+# Serve boot triage
|
|
|
+
|
|
|
+Goal: within ~5 minutes of a failed boot, know (a) which node and (b) why,
|
|
|
+with the crash log in hand — instead of a dead `--rm` container and a guess.
|
|
|
+
|
|
|
+## 1. Log preservation (the #1 time sink)
|
|
|
+
|
|
|
+Default `serve` runs `docker run --rm`: a crashed container is gone WITH ITS
|
|
|
+LOGS. For any first / unproven launch:
|
|
|
+
|
|
|
+- `sm121 serve ... --keep` — containers are NOT auto-removed on stop or
|
|
|
+ crash; `sm121 logs --recipe R --node N.lan` then works on the corpse.
|
|
|
+- Cleanup afterwards: `sm121 stop` stops but keeps (`--keep` semantics);
|
|
|
+ remove the leftovers with `docker rm` per node (note them in the report).
|
|
|
+
|
|
|
+## 2. Detached serve (the harness kills long foreground calls)
|
|
|
+
|
|
|
+A long foreground `sm121 serve` gets "Command aborted" by the tool harness;
|
|
|
+the process dies mid-launch → "containers on 2 of 4 nodes", lost logs,
|
|
|
+unclear ledger. Pattern (short call, returns immediately):
|
|
|
+
|
|
|
+```
|
|
|
+setsid nohup ~/.cargo/bin/sm121 serve --recipe R --cluster C --no-wait --keep \
|
|
|
+ > /tmp/serve-R.log 2>&1 < /dev/null & sleep 2; echo started
|
|
|
+```
|
|
|
+
|
|
|
+`--no-wait` skips the 30-min readiness poll; do your own polling (crashes are
|
|
|
+caught sooner anyway). Poll with SHORT calls: `tail /tmp/serve-R.log` +
|
|
|
+per-node `docker ps -a --filter name=<recipe>`.
|
|
|
+
|
|
|
+## 3. Per-node state matrix (first thing, ~10 s)
|
|
|
+
|
|
|
+```
|
|
|
+for ip in <ssh_ip from config.toml>; do
|
|
|
+ ssh -i ~/.ssh/id_ed25519_shared -o IdentitiesOnly=yes kilork@$ip \
|
|
|
+ 'docker ps -a --filter name=<recipe-prefix> --format "{{.Names}} {{.Status}}"'
|
|
|
+done
|
|
|
+```
|
|
|
+
|
|
|
+| State | Meaning | Action |
|
|
|
+|---|---|---|
|
|
|
+| Up on all | boot in progress (weights from NAS 25–40 min; first autotune +15–40) | poll API: `curl -s http://<head_ib_ip>:30000/v1/models` |
|
|
|
+| Up on some, ABSENT (not even in `ps -a`) on others | serve process died first (aborted call) OR image missing | check `docker images` on the absent node; then full clean re-serve (stop → fresh detached) or surgical `sm121 serve --recipe R --cluster C --node <n.lan> --no-wait --keep` (only if the ledger run is closed) |
|
|
|
+| `Exited (2)` (with `--keep`) | engine crashed | `sm121 logs --recipe R --node <n.lan>` → last lines are the cause |
|
|
|
+| Up on all, no API after 45+ min | NCCL hang or slow NFS | tail each node's logs: NCCL hang = all quiet at the same step; check `df`/NFS on the head |
|
|
|
+
|
|
|
+## 4. Hard rules (learned the hard way)
|
|
|
+
|
|
|
+- **Node IPs only from `config.toml` (`ssh_ip`), never from memory.** A wrong
|
|
|
+ IP looks exactly like "the node hung" (ssh timeout/refused). Current
|
|
|
+ (2026-10-02): dgx1 88.14, dgx2 **88.10**, dgx3 **88.50**, dgx4 **88.51**,
|
|
|
+ dgx5 88.43, dgx6 88.44, dgx7 88.54, dgx8 88.55; ib 192.168.200.x; DAC
|
|
|
+ (dgx1/2) 192.168.100.x.
|
|
|
+- Multi-line remote scripts: base64 them
|
|
|
+ (`printf '%s' "$S" | base64 | ssh ... "bash -c 'base64 -d | bash'"`) —
|
|
|
+ nested quotes in local double-quotes + remote zsh single-quotes break.
|
|
|
+- `sm121 runlock list` — an open run = a serve that never closed;
|
|
|
+ `sm121 runlock close <recipe>` after manual cleanup.
|
|
|
+- Do NOT duplicate launcher built-ins: `drop_caches` (config field, default
|
|
|
+ `true` — already runs before every serve), config-level `before_serve`.
|
|
|
+- `sm121 -d serve ...` (dry run) renders the exact per-node commands — use it
|
|
|
+ to predict, not to execute (its shell escaping is not paste-able).
|
|
|
+
|
|
|
+## 5. Failure patterns (GB10 / kindling era)
|
|
|
+
|
|
|
+- **UMA memory gate** (vLLM #48140):
|
|
|
+ `Free memory on device cuda:0 (106.53/121.63 GiB) ... less than desired GPU
|
|
|
+ memory utilization (0.88, 107.03 GiB)` — random node, <1 GiB margin; the
|
|
|
+ check reads MemFree at worker init; weight/NFS page cache tips it. Built-in
|
|
|
+ drop_caches covers the common case; recurrence → recipe `before_serve`
|
|
|
+ drop_caches (`on_failure = "warn"`) or GMU −0.02.
|
|
|
+- **`--speculative-config` quoting**: the TOML value must be
|
|
|
+ `"""` + newline + `'{JSON}'` + `"""` with the closing `"""` **on the same
|
|
|
+ line** as the JSON (mimo-recipe form). A trailing newline before `"""` →
|
|
|
+ the launcher keeps the apostrophes → vLLM `cannot be converted to
|
|
|
+ <function loads>`. A doubled `--speculative-config` (recipe args +
|
|
|
+ transcribed env EXTRA) → vLLM takes the last one = garbage.
|
|
|
+- X/gdm running on a node = extra GBs + a display context: headless =
|
|
|
+ `sudo systemctl set-default multi-user.target && sudo systemctl stop gdm`
|
|
|
+ (dgx1–5 already headless; dgx6/7/8 switched 2026-10-02).
|
|
|
+- First launch after an image build = autotune candidate runs (394 on the 8x
|
|
|
+ b12x stack) + NAS weight read: budget 60–90 min total; don't declare
|
|
|
+ failure early.
|
|
|
+- The engine's own preflight (kindling entrypoint) is bypassed in B-mode;
|
|
|
+ their preflight checks (fabric, GID) are the first things to replicate
|
|
|
+ manually if a boot fails without a clear vLLM traceback.
|
