flare: prod-no4 is production - dgx4 fails sustained Engram duty even with the reader patch

Run 9 (2026-11-04 10:03 UTC): the patched build (main@585159c9 + PR #184)
served the cold-boot 157K prompt and the full 97-scenario bench, then died
at seq 2833389 ('rank 1 waiting rank 3') - the KingSpec drive in dgx4 fails
the sustained Engram read duty no matter the reader parallelism. Production
moves back to prod-no4 (dgx1+dgx2+dgx3+dgx8, orbit prod5); prod4b stays
configured for after the NVMe swap.

Operational lesson recorded: every rank must run the same build - a boot
with dgx8 still on the old 0.1.0 failed at rendezvous ('incompatible
protocol' / 'failed to fill whole buffer'). The patched build is installed
as the agent binary on dgx1-4 and dgx8 (old apt build saved as
*.orig-20261104); config drops the --executable overrides.

AGENTS.md

@@ -1134,8 +1134,9 @@ the detect command per node and renders `NCCL_IB_GID_INDEX=auto`.
 - Use `rg` (ripgrep) for all code search — plain `grep -rn` across this tree aborts/hangs and is unusable; `rg` is fast and skips `.git`/gitignored paths
 - `#![deny(clippy::wildcard_imports)]` — no `use foo::*;`
 - **Clippy warnings**: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do **not** add `#[allow(...)]` attributes. If the only viable fix is an allow attribute, consult the user for guidance.
-- **tool-eval-bench runs happen in a separate working dir, never in this repo** — its output (e.g. `data/benchmarks.sqlite`) is not for this repo; a stray run here pollutes the working tree. Campaign **reports** ARE committed, next to the lane they benchmark: `recipes/<name>/reports/<UTC-stamp>_<label>.md` (copied from the tool's runs dir) **plus the raw JSON** — pass `--json-file <abs repo path>/recipes/<name>/reports/<UTC-stamp>_<label>.json` in the run command (absolute path: the bench runs in a separate cwd; the flag also keeps stdout clean for the `.log`); same base name for both. Quality-gate semantics are version-sensitive (`--hardmode` = 97 scenarios on v3.0.0 and v2.8.0; was 92 on 2.7.1.dev25, 88 on 2.7.0) — see the bench header in `specs/tech-architecture/glm53-recipe-state.md`; the tool is installed from `SeraphimSerapis/tool-eval-bench` — current pin: **v3.0.0** (2026-10-26, commit `d85126e`; 69 standard / 97 Hard Mode). Do not compare v2.8.0 reports with v3.0.0 reports; re-baseline the anchor lane first.
-- **Harness version — do not `uv tool upgrade` casually.** `tool-eval-bench` is a uv tool (`~/.local/share/uv/tools/tool-eval-bench`) installed from `SeraphimSerapis/tool-eval-bench.git`; the receipt is pinned at tag **v3.0.0** (2026-10-26, commit `d85126e`; 69 standard / 97 Hard Mode; web UI, `card`, `--perf-preset long`, `--needle`). Prior pins: v2.8.0 (2026-10-14; 97 scenarios; the `91/100`, `88/100`, `95/100` reports are v2.8.0), 2.7.1.dev7 (92 scen), 2.7.1.dev25 (the 8x 95/100 anchor, 2026-10-12), v2.7.0 `544afb576b3` (2026-09-21, first stable of dev65). Reports stamp the harness version in the header — **A/B runs must share a harness version**; pools and evaluators differ across versions (88→92→97), and v3.0.0 changes TC-21/TC-89 plus structured-output retries, so cross-version deltas are not comparable. To move the pin: `uv tool install --reinstall --force "tool-eval-bench[perf] @ git+https://github.com/SeraphimSerapis/tool-eval-bench.git@<tag-or-rev>"` then re-baseline the anchor lane. Reasoning effort: engine defaults (recipe templates) — explicit low-effort runs are separate labeled datapoints, not comparable to the standard ones.
+- **tool-eval-bench runs happen in a separate working dir, never in this repo** — its output (e.g. `data/benchmarks.sqlite`) is not for this repo; a stray run here pollutes the working tree. Campaign **reports** ARE committed, next to the lane they benchmark: `recipes/<name>/reports/<UTC-stamp>_<label>.md` (copied from the tool's runs dir) **plus the raw JSON** — pass `--json-file <abs repo path>/recipes/<name>/reports/<UTC-stamp>_<label>.json` in the run command (absolute path: the bench runs in a separate cwd; the flag also keeps stdout clean for the `.log`); same base name for both. Quality-gate semantics are version-sensitive (`--hardmode` = 97 scenarios on v3.0.0 and v2.8.0; was 92 on 2.7.1.dev25, 88 on 2.7.0) — see the bench header in `specs/tech-architecture/glm53-recipe-state.md`; the tool is installed from `SeraphimSerapis/tool-eval-bench` — current pin: **v3.0.0** (2026-10-26, commit `d85126e`; 69 standard / 97 Hard Mode) — installed on **pluto** (the bench run host, see the next bullet); the Mac's local copy may lag and is not campaign-grade. Do not compare v2.8.0 reports with v3.0.0 reports; re-baseline the anchor lane first.
+- **Run the bench from `pluto` (`--ssh-host pluto.lan`), not from the Mac.** The launcher uploads the harness scratch + tokenizer there and pulls the report back; the Mac's own `tool-eval-bench` install is allowed to drift (it was **2.8.0** on 2026-11-04 while the pin is **v3.0.0**) and must never be used for campaign runs — pluto carries the pinned version. Check before a campaign: `ssh pluto.lan tool-eval-bench --version`. If pluto's pin ever drifts: `uv tool install --reinstall --force "tool-eval-bench[perf] @ git+https://github.com/SeraphimSerapis/tool-eval-bench.git@<tag-or-rev>"` on pluto, then re-baseline the anchor lane.
+- **Tokenizer for the FLARE lanes** (`deepseek-v4.1-flash` etc.): the harness runs offline and needs a local `tokenizer.json`. The master copy lives on **pluto** at `/home/kilork/scratch/dsv41-tokenizer/` — but `--tokenizer <path>` is resolved on the **launcher's machine** and uploaded from there, so first fetch it to a local scratch: `scp pluto.lan:/home/kilork/scratch/dsv41-tokenizer/tokenizer.json /tmp/dsv41-tokenizer/` and pass `--tokenizer /tmp/dsv41-tokenizer/tokenizer.json`. (The FLARE package itself does not expose a tokenizer.json to point at.) Reasoning effort: engine defaults (recipe templates) — explicit low-effort runs are separate labeled datapoints, not comparable to the standard ones.
 - **Bulk data moves node-to-node, never through the Mac** — the Mac→node link
   is ~32 MB/s (a 200 GB transfer via the Mac takes hours); node-to-node over the
   200G fabric does ~547 MB/s per stream. Node-to-node SSH works (the shared

specs/tech-architecture/flare-roce-poison.md

@@ -60,14 +60,16 @@ measured, on the production package and flags.
 | 5 | orbit `prod5` = dgx1+dgx2+dgx3+**dgx8**, no dgx4 | packaged `flare` | **clean**, 0→157,086 in ~61 s (×2) |
 | 6 | orbit `prod4b`, dgx4 = rank 3 | **main + PR #184** | **clean**, 0→157,086 in ~60 s (~2,600 t/s) — warm stand |
 | 8 | orbit `prod4b` (2026-11-04, **freshly rebooted** node) | **main + PR #184** | **poison** @ seq 93, prefill frozen at **token 1** — the cold-boot case survives the reader patch |
+| 9 | orbit `prod4b` (2026-11-04, after the full bench ≈2.8 M collectives) | **main + PR #184** | **poison** @ seq 2833389, `rank 1 waiting rank 3` 10:03 UTC — sustained load kills it too |
 | 7 | orbit `sparks` = dgx5–8 (2 runs, incl. after 45 min idle) | packaged `flare` | clean |
 
 Runs 4 and 6 matter most: with dgx4 demoted from rank 0 the failure
 **followed the node, not the rank**. Run 5 isolates it further: remove dgx4
 entirely and the packaged binary is clean. The wedging pair is always
-a peer waiting on **dgx4's** data. Run 8 adds the cold-boot dimension: the
-reader patch rescues a warm stand but not the first long request after a
-reboot — the drive stalls mid-burst inside the spin window at chunk 1.
+a peer waiting on **dgx4's** data. Runs 8 and 9 close the escape hatches:
+the reader patch rescues neither the cold-boot first chunk (seq 93,
+prefill=1) nor sustained duty (seq 2833389 after the full bench) — the
+drive stalls inside the spin window regardless of reader parallelism.
 
 ### Rank 0 metrics through one wedge (1 Hz, `prod4`, layout `ESP`)
 
@@ -128,18 +130,29 @@ first long request wedges at `sequence 93`, prefill frozen at token 1. Rank
 0's 1 Hz capture shows the first chunk's reads running hot (≈87 K reads/s,
 disk wait +2.7 s) and then the whole runtime dead within ~8 s — dgx4's drive
 stalls mid-burst inside the spin window, which the parallel reader raises in
-throughput but cannot bound in time. Operationally this means: after every
-reboot, a cold stand can lose its first long prompt even with the patch.
-The production lane therefore excludes dgx4 (`prod-no4`) until the drive is
-replaced; the `prod4b`+patch configuration is reserved for after the swap.
+throughput but cannot bound in time. **And not under sustained load** (run
+9): the same patched build served the cold-boot 157 K prompt, then the full
+97-scenario bench, then died at 10:03 UTC at `sequence 2833389` (`rank 1
+waiting rank 3`) — the drive fails the sustained Engram read duty no matter
+the reader. Operationally: after every reboot AND under sustained traffic,
+a stand with dgx4 loses its lane. The production lane therefore excludes
+dgx4 (`prod-no4`); the `prod4b`+patch configuration returns only after the
+NVMe swap.
 
 ## Production decision (2026-11-04)
 
 - Serving lane: **`prod-no4`** = dgx1+dgx2+dgx3+dgx8 (orbit `prod5`, rank 0
   = dgx1), recipe `dsv41-flash-1017-4x-flare-gb10-nodgx4`. Clean on the same
-  prompts that killed the dgx1–4 lane, with or without the reader patch.
-- `prod4b` + `executable` stays configured for **after the NVMe swap**
-  (Samsung PM9A3/PM9E1 M.2 2242 or Corsair MP700 Micro class; back up
+  prompts and bench load that killed the dgx1–4 lane.
+- The patched build (main@585159c9 + PR #184) is **installed as the agent
+  binary on dgx1–4 and dgx8** (`~/.local/lib/flare-agent/flare` +
+  `/usr/bin/flare`; the old apt build saved as `*.orig-20261104`).
+  **Every rank must run the same build**: a 10:06 UTC boot with dgx8 still
+  on the old 0.1.0 failed at rendezvous (`incompatible protocol` on rank 0,
+  `failed to fill whole buffer` on dgx8) — after any binary install, all
+  rank nodes get it before relaunch.
+- `prod4b` stays configured for **after the NVMe swap** (Samsung
+  PM9A3/PM9E1 M.2 2242 or Corsair MP700 Micro class; back up
   `~/.config/flare/orbit/controller.key` before the DGX OS reinstall to skip
   re-pairing).
 - The `prod4` orbit (rank 0 = dgx4) remains in `orbit.toml` for reference.

src/native/flare.rs

@@ -79,13 +79,19 @@ pub fn download_cmd(config: &Config, recipe: &Recipe) -> Result<String> {
     Ok(cmd)
 }
 
-/// `flare orbit run "<node>:<package>" --cluster <c> --name <n> … --detach`.
+/// `flare orbit run <package> --cluster <c> --name <n> … --detach`.
+///
+/// The package path is **controller-local** (`flare download` and the mirror
+/// both land on the controller), and the CLI stages it to rank zero itself —
+/// no `NODE:` prefix. Prefixing with the orbit's rank-zero name broke the
+/// prod-no4 lane (rank 0 = dgx8 holds only its slice; the file lives on
+/// dgx4), and when the controller *is* rank zero the plain path points at the
+/// same file anyway.
 pub fn run_cmd(
     config: &Config,
     recipe: &Recipe,
     our_cluster: &str,
     orbit_cluster: &str,
-    controller: &str,
     package: &str,
 ) -> Result<String> {
     let f = recipe
@@ -96,7 +102,7 @@ pub fn run_cmd(
     let mut cmd = format!(
         "{} orbit run {} --cluster {} --name {} --host 0.0.0.0 --port {} --detach",
         binary(config),
-        shq(&format!("{controller}:{package}")),
+        shq(package),
         shq(orbit_cluster),
         shq(&model_name),
         http_port(recipe),
@@ -345,8 +351,7 @@ pub async fn serve(
             crate::ssh::run_ssh_long_running(config, controller, cmd, true).await?;
         }
     }
-    let orbit_name = orbit_node_name(config, &cluster.name, controller);
-    let run = run_cmd(config, recipe, &cluster.name, &orbit, &orbit_name, &package)?;
+    let run = run_cmd(config, recipe, &cluster.name, &orbit, &package)?;
     crate::ssh::run_ssh_long_running(config, controller, run.clone(), dry_run).await?;
     if dry_run {
         println!("  [dry-run] {}: {run}", controller.name);

src/native/flare_tests.rs

@@ -34,14 +34,13 @@ fn cluster_extra_flags_reach_orbit_run() {
         &recipe(),
         "4nodes-switch",
         "prod4",
-        "gx10-dgx1",
         "/pkgs/x.safetensors",
     )
     .unwrap();
     assert!(cmd.contains("'--roce-ring'"), "{cmd}");
     assert!(!cmd.contains("--roce-ring") || cmd.matches("--roce-ring").count() == 1);
     // another cluster does not inherit it
-    let other = run_cmd(&c, &recipe(), "dgx58-switch", "sparks", "dgx5", "/p").unwrap();
+    let other = run_cmd(&c, &recipe(), "dgx58-switch", "sparks", "/p").unwrap();
     assert!(!other.contains("--roce-ring"), "{other}");
 }
 
@@ -52,14 +51,10 @@ fn run_passes_flags_verbatim_and_the_controller_package() {
         &recipe(),
         "dgx58-switch",
         "sparks",
-        "dgx5",
         "/pkgs/dsv41.safetensors",
     )
     .unwrap();
-    assert!(
-        cmd.contains("orbit run 'dgx5:/pkgs/dsv41.safetensors'"),
-        "{cmd}"
-    );
+    assert!(cmd.contains("orbit run '/pkgs/dsv41.safetensors'"), "{cmd}");
     assert!(cmd.contains("--cluster 'sparks'"), "{cmd}");
     assert!(cmd.contains("--name 'deepseek1017'"), "{cmd}");
     assert!(cmd.contains("--port 8000"), "{cmd}"); // recipe value wins
@@ -320,7 +315,6 @@ fn cluster_executable_override_reaches_orbit_run() {
         &recipe(),
         "4nodes-switch",
         "prod4b",
-        "gx10-dgx1",
         "/pkgs/x.safetensors",
     )
     .unwrap();
@@ -329,7 +323,7 @@ fn cluster_executable_override_reaches_orbit_run() {
         "{cmd}"
     );
     // another cluster keeps the packaged install
-    let other = run_cmd(&c, &recipe(), "dgx58-switch", "sparks", "dgx5", "/p").unwrap();
+    let other = run_cmd(&c, &recipe(), "dgx58-switch", "sparks", "/p").unwrap();
     assert!(!other.contains("--executable"), "{other}");
 }