|
|
@@ -66,6 +66,30 @@ done
|
|
|
- `sm121 -d serve ...` (dry run) renders the exact per-node commands — use it
|
|
|
to predict, not to execute (its shell escaping is not paste-able).
|
|
|
|
|
|
+## 4b. Long-running local work (benches, builds) — move to a node
|
|
|
+
|
|
|
+The tool harness reaps long foreground calls (~10 min) and detached local
|
|
|
+processes (`nohup`/`setsid` die with the call). Benches run 1.5–2 h. Pattern:
|
|
|
+
|
|
|
+1. Ship the launcher to a cluster node and build it there (the Mac binary is
|
|
|
+ Mach-O; nodes are aarch64 Linux): `rustup` one-liner (minimal profile,
|
|
|
+ ~1 min) + `cargo build --release` (deps ~5 min, incremental ~15 s).
|
|
|
+2. Ship the minimal repo subset — `src Cargo.toml Cargo.lock locales
|
|
|
+ host-scripts` (~400 KB). NEVER tar the whole repo: `target/` is tens of GB.
|
|
|
+3. Fix `ssh_key_path` in the node's `config.toml` (Mac path → node key,
|
|
|
+ `~/.ssh/id_ed25519_shared`).
|
|
|
+4. Install the harness on the node: `uv` installer +
|
|
|
+ `uv tool install "tool-eval-bench[perf] @ git+...@<pinned-rev>"`; add
|
|
|
+ `~/.local/bin` to PATH inside the tmux command (non-interactive sessions
|
|
|
+ don't source the installer's profile).
|
|
|
+5. Run under `tmux new -d -s <name>`; poll with short calls:
|
|
|
+ `tail -c 2000 /tmp/<name>.log | tr \\r \\n | tail`.
|
|
|
+6. `--tokenizer` must be a **host-side** path readable by the bench user
|
|
|
+ (the NAS path, not the container-side `/root/.cache/...` — containers are
|
|
|
+ root, the host user is not).
|
|
|
+7. Reports land in the node's `recipes/<lane>/reports/`; copy them back to
|
|
|
+ the Mac repo and commit (same stem).
|
|
|
+
|
|
|
## 5. Failure patterns (GB10 / kindling era)
|
|
|
|
|
|
- **UMA memory gate** (vLLM #48140):
|
