---
name: serve-boot-triage
description: Fast triage and recovery of sm121 serve launches — a container that died, a node missing its container, or an engine that never reaches readiness. Log preservation (--keep), detached serve, per-node state matrix, ledger hygiene, GB10 UMA memory gate, --speculative-config quoting. Use whenever a serve boot fails or a node has no container, before waiting 10+ minutes.
---

# Serve boot triage

Goal: within ~5 minutes of a failed boot, know (a) which node and (b) why,
with the crash log in hand — instead of a dead `--rm` container and a guess.

## 1. Log preservation (the #1 time sink)

Default `serve` runs `docker run --rm`: a crashed container is gone WITH ITS
LOGS. For any first / unproven launch:

- `sm121 serve ... --keep` — containers are NOT auto-removed on stop or
  crash; `sm121 logs --recipe R --node N.lan` then works on the corpse.
- Cleanup afterwards: `sm121 stop` stops but keeps (`--keep` semantics);
  remove the leftovers with `docker rm` per node (note them in the report).

## 2. Detached serve (the harness kills long foreground calls)

A long foreground `sm121 serve` gets "Command aborted" by the tool harness;
the process dies mid-launch → "containers on 2 of 4 nodes", lost logs,
unclear ledger. Pattern (short call, returns immediately):

```
setsid nohup ~/.cargo/bin/sm121 serve --recipe R --cluster C --no-wait --keep \
  > /tmp/serve-R.log 2>&1 < /dev/null & sleep 2; echo started
```

`--no-wait` skips the 30-min readiness poll; do your own polling (crashes are
caught sooner anyway). Poll with SHORT calls: `tail /tmp/serve-R.log` +
per-node `docker ps -a --filter name=<recipe>`.

## 3. Per-node state matrix (first thing, ~10 s)

```
for ip in <ssh_ip from config.toml>; do
  ssh -i ~/.ssh/id_ed25519_shared -o IdentitiesOnly=yes kilork@$ip \
    'docker ps -a --filter name=<recipe-prefix> --format "{{.Names}} {{.Status}}"'
done
```

| State | Meaning | Action |
|---|---|---|
| Up on all | boot in progress (weights from NAS 25–40 min; first autotune +15–40) | poll API: `curl -s http://<head_ib_ip>:30000/v1/models` |
| Up on some, ABSENT (not even in `ps -a`) on others | serve process died first (aborted call) OR image missing | check `docker images` on the absent node; then full clean re-serve (stop → fresh detached) or surgical `sm121 serve --recipe R --cluster C --node <n.lan> --no-wait --keep` (only if the ledger run is closed) |
| `Exited (2)` (with `--keep`) | engine crashed | `sm121 logs --recipe R --node <n.lan>` → last lines are the cause |
| Up on all, no API after 45+ min | NCCL hang or slow NFS | tail each node's logs: NCCL hang = all quiet at the same step; check `df`/NFS on the head |

## 4. Hard rules (learned the hard way)

- **Node IPs only from `config.toml` (`ssh_ip`), never from memory.** A wrong
  IP looks exactly like "the node hung" (ssh timeout/refused). Current
  (2026-10-02): dgx1 88.14, dgx2 **88.10**, dgx3 **88.50**, dgx4 **88.51**,
  dgx5 88.43, dgx6 88.44, dgx7 88.54, dgx8 88.55; ib 192.168.200.x; DAC
  (dgx1/2) 192.168.100.x.
- Multi-line remote scripts: base64 them
  (`printf '%s' "$S" | base64 | ssh ... "bash -c 'base64 -d | bash'"`) —
  nested quotes in local double-quotes + remote zsh single-quotes break.
- `sm121 runlock list` — an open run = a serve that never closed;
  `sm121 runlock close <recipe>` after manual cleanup.
- Do NOT duplicate launcher built-ins: `drop_caches` (config field, default
  `true` — already runs before every serve), config-level `before_serve`.
- `sm121 -d serve ...` (dry run) renders the exact per-node commands — use it
  to predict, not to execute (its shell escaping is not paste-able).

## 4b. Long-running local work (benches, builds) — move to a node

The tool harness reaps long foreground calls (~10 min) and detached local
processes (`nohup`/`setsid` die with the call). Benches run 1.5–2 h. Pattern:

1. Ship the launcher to a cluster node and build it there (the Mac binary is
   Mach-O; nodes are aarch64 Linux): `rustup` one-liner (minimal profile,
   ~1 min) + `cargo build --release` (deps ~5 min, incremental ~15 s).
2. Ship the minimal repo subset — `src Cargo.toml Cargo.lock locales
   host-scripts` (~400 KB). NEVER tar the whole repo: `target/` is tens of GB.
3. Fix `ssh_key_path` in the node's `config.toml` (Mac path → node key,
   `~/.ssh/id_ed25519_shared`).
4. Install the harness on the node: `uv` installer +
   `uv tool install "tool-eval-bench[perf] @ git+...@<pinned-rev>"`; add
   `~/.local/bin` to PATH inside the tmux command (non-interactive sessions
   don't source the installer's profile).
5. Run under `tmux new -d -s <name>`; poll with short calls:
   `tail -c 2000 /tmp/<name>.log | tr \\r \\n | tail`.
6. `--tokenizer` must be a **host-side** path readable by the bench user
   (the NAS path, not the container-side `/root/.cache/...` — containers are
   root, the host user is not).
7. Reports land in the node's `recipes/<lane>/reports/`; copy them back to
   the Mac repo and commit (same stem).

## 5. Failure patterns (GB10 / kindling era)

- **UMA memory gate** (vLLM #48140):
  `Free memory on device cuda:0 (106.53/121.63 GiB) ... less than desired GPU
  memory utilization (0.88, 107.03 GiB)` — random node, <1 GiB margin; the
  check reads MemFree at worker init; weight/NFS page cache tips it. Built-in
  drop_caches covers the common case; recurrence → recipe `before_serve`
  drop_caches (`on_failure = "warn"`) or GMU −0.02.
- **`--speculative-config` quoting**: the TOML value must be
  `"""` + newline + `'{JSON}'` + `"""` with the closing `"""` **on the same
  line** as the JSON (mimo-recipe form). A trailing newline before `"""` →
  the launcher keeps the apostrophes → vLLM `cannot be converted to
  <function loads>`. A doubled `--speculative-config` (recipe args +
  transcribed env EXTRA) → vLLM takes the last one = garbage.
- X/gdm running on a node = extra GBs + a display context: headless =
  `sudo systemctl set-default multi-user.target && sudo systemctl stop gdm`
  (dgx1–5 already headless; dgx6/7/8 switched 2026-10-02).
- First launch after an image build = autotune candidate runs (394 on the 8x
  b12x stack) + NAS weight read: budget 60–90 min total; don't declare
  failure early.
- The engine's own preflight (kindling entrypoint) is bypassed in B-mode;
  their preflight checks (fabric, GID) are the first things to replicate
  manually if a boot fails without a clear vLLM traceback.