flare: first-class --executable override + Engram-stall post-mortem

The prod FLARE lane died 3/3 on a 157K first-touch prefill with
'RoCE collective ... timed out ... the transport is poisoned'. It was not
the fabric: rank 0's /metrics showed stop-and-go prefill with disk wait
jumping +10.2s and ~24K random engram reads/s against a ~29K IOPS ceiling
on dgx4's KingSpec XG7000-2TB (the only non-ESL01 disk in the fleet).
Reordering the orbit moved the failure with the node; removing dgx4 was
clean with the packaged binary.

- config: [flare.clusters.<cluster>].executable (+ flat fallback) emits
  'orbit run --executable <path>' so a locally patched runtime serves a
  lane without touching the packaged agent install; absolute paths only
  (the CLI's cwd is on the controller). Tests for the per-cluster
  override, the isolation from other clusters, and the relative-path
  refusal.
- specs/tech-architecture/flare-roce-poison.md: evidence pack (run
  ledger, 1 Hz metrics, wire counters, what did not help), the
  pr184-on-main mitigation and its before/after numbers, and a
  paste-ready upstream issue.

AGENTS.md

@@ -428,6 +428,13 @@ disk holds the package), which is *not* our rank 0 once the orbit is re-paired
 or reordered. `orbit_nodes` is listed in the orbit's rank order and is matched
 by node name (`gx10-dgx4` ↔ `dgx4.lan`), never by our rank index.
 
+`executable` (per cluster, flat fallback) adds `orbit run --executable <path>`:
+serving ranks run **that** build instead of each node's packaged agent install,
+so a locally patched runtime serves a lane without touching `/usr/bin/flare`
+(absolute paths only — the CLI's cwd is on the controller). That is how the
+Engram disk-reader fix (#184 + main) runs on `4nodes-switch`; the failure it
+fixes is documented in `specs/tech-architecture/flare-roce-poison.md`.
+
 Two stores, both budgeted: `flare download` fills the **CLI store**
 (`paths.package_cache`), and `orbit run` then stages into each node's **agent
 cache** (`~/.cache/flare/packages/<sha>-<size>/`, *not* movable by

specs/tech-architecture/flare-roce-poison.md

@@ -0,0 +1,203 @@
+# FLARE `transport is poisoned`: an Engram disk stall, not a RoCE fault (2026-11-03)
+
+The prod FLARE lane on dgx1–4 (`dsv41-flash-1017-4x-flare-gb10-prod4`) died
+repeatedly with two messages that look like fabric failures:
+
+```
+tensor-parallel peer failed: RoCE collective on rank 2 timed out waiting for
+rank 0, HCA 0, at sequence 7122648; the transport is poisoned
+
+RDMA write to rank 3 failed: transport retry counter exceeded
+(vendor_err 0x81, seq 250342)
+```
+
+They are **not** fabric faults on this stand. The first is the collective
+spin-wait expiring because a peer rank is starved on **disk-backed Engram
+reads**; the second belongs to a different, rarer class (real QP retry
+exhaustion) and did not reproduce in any 2026-11-03 run. Everything below is
+measured, on the production package and flags.
+
+## Verdict
+
+- `RoCE collective … timed out … transport is poisoned` — the b12x/RoCEnante
+  GPU kernel polls a peer's stripe flag for `DEFAULT_SPIN_LIMIT = 20_000_000`
+  polls (≈20 s on GB10; compiled **into the prepared package**, not a
+  serve-time flag) and then poisons the transport. FLARE is fail-stop: no
+  retry, no reroute, restart every rank.
+- The stall is `flare_engram_disk_wait_nanoseconds_total` on the starved rank:
+  prefill advances in bursts, GPU idle 10–20 s at a time, and the slowest
+  reader still gets its data — the *peer's* spin limit expires first.
+- Root cause on this stand: **dgx4's SSD**. All four prod nodes run the
+  generic `ESL01TBTLCZ-27J2-TYN` 1 TB drive except dgx4, which carries a
+  **KingSpec XG7000-2TB 2242** (fw `GT0d9d63`, vid `0x1e4b`) — a budget
+  consumer drive, no first-party firmware tool, our firmware not even listed
+  in the public database (`SN13683` is what smarthdd knows for this model).
+  Sustained random-read ceiling measured on the wedge node: **≈29 K IOPS**
+  (4 threads, O_DIRECT 4 K, 10 s) vs ≈33 K on a healthy dgx5 — while a
+  first-touch prefill demanded **≈24 K reads/s**, i.e. 80–90 % of the drive.
+- **Mitigation that works today**: the unmerged upstream io_uring reader
+  (PR #184) on top of current `main`. Same hardware, same package, same
+  prompt: 3/3 wedges before, clean run after (numbers below).
+
+## Evidence
+
+### Run ledger (same package, same flags, 157 K-token first-touch prompt)
+
+| # | arrangement | runtime | result |
+|---|---|---|---|
+| 1 | orbit `prod4`, rank 0 = dgx4 | packaged `flare` | **poison** @ seq 7122648 (93 K tokens prefilled) |
+| 2 | orbit `prod4`, rank 0 = dgx4 | packaged `flare` | **poison** @ seq 2109 (frozen ~93 K) |
+| 3 | orbit `prod4`, rank 0 = dgx4 | packaged `flare` | **poison** @ seq 1284 (frozen ~61 K) |
+| 4 | orbit `prod4b`, rank 0 = **dgx1**, dgx4 = rank 3 | packaged `flare` | **poison** @ seq 1557 (`rank 1 waiting for rank 3`) |
+| 5 | orbit `prod5` = dgx1+dgx2+dgx3+**dgx8**, no dgx4 | packaged `flare` | **clean**, 0→157,086 in ~61 s |
+| 6 | orbit `prod4b`, dgx4 = rank 3 | **main + PR #184** | **clean**, 0→157,086 in ~60 s (~2,600 t/s) |
+| 7 | orbit `sparks` = dgx5–8 (2 runs, incl. after 45 min idle) | packaged `flare` | clean |
+
+Runs 4 and 6 matter most: with dgx4 demoted from rank 0 the failure
+**followed the node, not the rank**. Run 5 isolates it further: remove dgx4
+entirely and the packaged binary is clean. The wedging pair is always
+dgx2-or-dgx1 waiting on **dgx4's** data.
+
+### Rank 0 metrics through one wedge (1 Hz, `prod4`, layout `ESP`)
+
+| t | prefill | engram disk reads | disk wait (cum) |
+|---|---|---|---|
+| 20:52:55 | 52,993 | 2.32 M | 17.01 s |
+| 20:52:56–59 | **frozen** | +108 K reads | 17.01 s (frozen) |
+| 20:53:00–01 | **frozen** | slow | **27.17 s (+10.2 s in 2 s)** |
+| 20:53:02 | burst → 61,185 | +2.6 K | 27.17 s |
+| 20:53:03–08 | **frozen** | frozen | 27.17 s |
+| 20:53:09 | server dead — `poisoned`, seq 1284 | | |
+
+The disk queue backs up while the GPU starves; `engram_row_cache_misses`
+keeps climbing (69 % miss with the 1 GiB row cache the package recommends for
+unified-memory GB10).
+
+### Wire counters (all four nodes, sampled every 10 s across the wedges)
+
+`local_ack_timeout_err`, `req_transport_retries_exceeded`, `req_cqe_error`:
+**0 everywhere, all day**. `out_of_sequence` / `packet_seq_err` /
+`roce_adp_retrans` are non-zero but flat-rate and retried successfully
+(98.7 M RDMA write requests on dgx4). `rx_write_requests` freezes on all four
+ranks simultaneously at the wedge — writes stop because the *producer* stopped,
+not because the fabric dropped them. No mlx5/Xid events in the journal.
+
+### What did **not** help
+
+- `--embedding-table-cache-size 4GiB` (from 1 GiB): wedged **faster**
+  (froze ~35 K). More cached rows did not change the read burst pattern.
+- `--embedding-table-workers 4`: rejected —
+  `parallel embedding disk workers require Windows`. The Linux multi-lane
+  reader exists only as unmerged PR #184.
+- MTU / GID / link checks: MTU 9000 on both CX7 functions, RoCEv2 IPv4 GID at
+  index 3, static addresses (one /24 per host pair), `ping -M do -s 8972`
+  clean, SMART clean (`media_errors 0`, `num_err_log_entries 0`, 42 °C).
+- Rank-0 role / controller co-residency: disproved by runs 4 and 5.
+
+### The mitigation (our build)
+
+`pr184-on-main` = `NVIDIA/spark-local-inference` `main` (`44ec239`) +
+cherry-pick of PR #184 `4ed10e1b` ("Run parallel Engram disk workers on Linux
+with io_uring lanes"). Cherry-pick merges cleanly; 5 files, +508/−309, all in
+the runtime (`helios/src/engram/reader*`, `runtime_package/selection.rs`,
+`docs/cli-design.md`) — no package ABI change, the same prepared packages
+serve unchanged. Release build aarch64 on a Spark: ~50 s.
+
+Deployed with `orbit run --executable /home/kilork/flare-patched/flare`
+(launcher: `executable` in `[flare.clusters.<cluster>]`), packaged agent
+install untouched.
+
+Result (run 6): prefill 0 → 157,086 tokens in ~60 s, no burst gaps, total
+`engram_disk_wait` **48.7 s** — statistically identical to the healthy dgx5
+lane (48.4 s). dgx4's KingSpec drive is still the slowest disk in the fleet;
+the pipelined reader simply keeps it under the collective's patience.
+
+## Upstream material
+
+### Status in `NVIDIA/spark-local-inference` (was `flare-local-inference`)
+
+- Poison = fail-stop is **documented behaviour** (`docs/multi-host.md`,
+  `docs/roce-ring.md`): "A collective failure invalidates the serving run …
+  stop and restart all ranks together".
+- Open issue **#151** carries the same message text, but deterministic at
+  `sequence 1` with two rails on the `f1` functions; a commenter serves fine
+  on `f0`. Not this failure (ours is mid-run and needs no second rail).
+- Open issue **#78** documents disk-Engram first-touch prefill cost (28.6 s
+  @250 K, 61.3 s @500 K on a 4× RTX PRO box); cache budget 8→32 GiB does not
+  fix first touch.
+- PR **#184** (parallel Windows-style lanes via io_uring on Linux) was closed
+  unmerged on 2026-11-03 with "the filesystem was the real bottleneck" —
+  measured on Btrfs. Our counterexample is ext4 on a GB10 Spark where the
+  single-lane reader is the difference between a dead lane and a clean one.
+- The transport code (`helios/native/roce_proxy.c`) has not changed since
+  `e350914a` (2026-10-21) apart from chain support and licensing.
+- **Nothing upstream tracks a stalled rank turning into a poisoned
+  transport**, and only rank 0 exposes `/metrics` — the ranks that wait are
+  blind.
+
+### Issue draft (ready to paste)
+
+> **Title:** Disk-Engram read stall on one rank poisons the TP transport
+> (`RoCE collective … timed out … transport is poisoned`) — reproduced 3/3,
+> fixed by PR #184's io_uring reader
+>
+> **Summary.** On a 4× DGX Spark (GB10) TP4 lane serving
+> `local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE` (`gb10`,
+> `--embedding-table-storage disk --embedding-table-cache-size 1GiB
+> --embedding-table-resident-scales`, `--max-concurrency 32
+> --mtp-draft-tokens 5 --mtp-adaptive-verification`), a 157 K-token
+> first-touch prompt kills the run on the same hardware it serves well
+> otherwise:
+>
+> ```
+> RoCE collective on rank 2 timed out waiting for rank 0, HCA 0, at
+> sequence 7122648; the transport is poisoned
+> ```
+>
+> **Diagnosis.** Rank 0's `/metrics` shows stop-and-go prefill:
+> `flare_prefill_tokens_total` frozen for 3–6 s at a time while
+> `flare_engram_disk_wait_nanoseconds_total` jumps (+10.2 s within one 2 s
+> window) and `flare_engram_row_cache_misses_total` climbs at ≈24 K
+> random-reads/s — 80–90 % of that node's measured 4 K O_DIRECT ceiling
+> (≈29 K IOPS). The waiting rank hits the 20 M-poll kernel spin limit
+> (`DEFAULT_SPIN_LIMIT`, compiled into the package) and poisons the
+> transport. RDMA counters are clean throughout: `local_ack_timeout_err`,
+> `req_transport_retries_exceeded`, `req_cqe_error` all 0;
+> `rx_write_requests` freezes on every rank because the producer stopped.
+>
+> **Rank-0 role is not the trigger.** Reordering the orbit so another node
+> is rank 0 still fails — the same node's data is what the peer waits for.
+> Removing that node from the cluster (dgx1+dgx2+dgx3+dgx8) serves the same
+> prompt cleanly with the packaged binary.
+>
+> **Fix.** Current `main` + PR #184 cherry-picked (`pr184-on-main`) serves
+> the identical package and prompt: prefill 0→157 K in ~60 s, total Engram
+> disk wait 48.7 s (healthy-lane parity), no poison — 1/1 so far after 3/3
+> failures without it.
+>
+> **Requests.**
+> 1. Reconsider #184: its close reasoning ("the filesystem was the
+>    bottleneck") holds for Btrfs, but on ext4/GB10 the single-lane reader
+>    is what makes the lane survive a disk that is merely slow.
+> 2. Make the collective spin limit configurable at serve time (or raise it
+>    when disk-Engram delivery is in play) — it is currently baked into the
+>    prepared package, so operators cannot trade latency for stability.
+> 3. Expose `/metrics` on **every** rank (or at least the readiness/disk
+>    counters): the rank that waits is invisible today, which turns a disk
+>    stall into an unattributable fabric error.
+> 4. Consider a preflight/warning when a host's Engram disk falls below the
+>    package's implied read rate; a 6 W consumer 2242 at 29 K IOPS is not
+>    the same class as the reference platforms.
+
+## Operational state after this investigation
+
+- Production lane: orbit **`prod4b`** (`dgx1 → dgx2 → dgx3 → dgx4`,
+  rank 0 = dgx1) + `executable = /home/kilork/flare-patched/flare`
+  (staged on all four prod nodes; only the controller's copy is uploaded).
+- `prod4` (rank 0 = dgx4) is kept in `orbit.toml` for rollback; `prod5`
+  (dgx1+dgx2+dgx3+dgx8) is the no-dgx4 control.
+- Hardware follow-up still open: replace dgx4's KingSpec
+  `XG7000-2TB 2242` with an enterprise-class 2242 (Samsung PM9A3/PM9E1 M.2
+  2242, or Corsair MP700 Micro) — the software patch buys stability, not a
+  fast disk. Back up `~/.config/flare/orbit/controller.key` first: restoring
+  it after a DGX OS reinstall avoids re-pairing all eight agents.

src/config/flare_cluster.rs

@@ -25,6 +25,14 @@ pub struct FlareCluster {
     /// the unpaired legacy ports and is no longer used.
     #[serde(default)]
     pub extra_flags: Vec<String>,
+    /// `orbit run --executable <path>` on the controller: serving ranks run
+    /// THIS build instead of each node's installed `flare`. Used to run a
+    /// locally patched runtime (e.g. the Engram disk-reader fix) without
+    /// touching the packaged agent install; the launcher uploads the file to
+    /// every launch node. Relative paths are refused — the controller's cwd
+    /// is not the operator's.
+    #[serde(default)]
+    pub executable: Option<String>,
 }
 
 /// `[flare]` in the private config: how the launcher talks to FLARE.
@@ -67,4 +75,8 @@ pub struct FlareSettings {
     /// Flat fallback for the per-cluster extra `orbit run` flags.
     #[serde(default)]
     pub extra_flags: Vec<String>,
+    /// Flat fallback for the serving executable override (see
+    /// [`FlareCluster::executable`]).
+    #[serde(default)]
+    pub executable: Option<String>,
 }

src/native/flare.rs

@@ -27,7 +27,7 @@ use tracing::info;
 
 use crate::config::{Config, Recipe};
 use crate::docker::{ServeOpts, shq};
-use crate::native::flare_orbit::{controller_for, extra_flags_for};
+use crate::native::flare_orbit::{controller_for, executable_for, extra_flags_for};
 // Re-exported: the driver is the FLARE entry point the rest of the launcher uses.
 pub use crate::native::flare_orbit::{
     local_dir_for, orbit_cluster_for, orbit_cluster_paired, orbit_node_name,
@@ -118,6 +118,11 @@ pub fn run_cmd(
     for flag in extra_flags_for(config, our_cluster) {
         cmd.push_str(&format!(" {}", shq(&flag)));
     }
+    // A patched runtime (Engram disk-reader fix, ABI-compatible builds) serves
+    // the lane without replacing the packaged agent install on every node.
+    if let Some(exe) = executable_for(config, our_cluster) {
+        cmd.push_str(&format!(" --executable {}", shq(&exe)));
+    }
     Ok(cmd)
 }
 

src/native/flare_orbit.rs

@@ -82,6 +82,23 @@ pub fn extra_flags_for(config: &Config, cluster: &str) -> Vec<String> {
     }
 }
 
+/// Serving-executable override for a cluster (per-cluster map first, then
+/// flat). `orbit run --executable` uploads this file to every launch node, so
+/// a patched runtime can serve a lane without replacing the packaged agent
+/// install. Only absolute paths are accepted: the CLI's cwd on the controller
+/// is not the operator's, so a relative path would resolve elsewhere.
+pub fn executable_for(config: &Config, cluster: &str) -> Option<String> {
+    let f = config.flare.as_ref()?;
+    let per = f.clusters.get(cluster).and_then(|c| c.executable.clone());
+    let path = per.or_else(|| f.executable.clone())?;
+    let path = path.trim().to_owned();
+    if path.starts_with('/') {
+        Some(path)
+    } else {
+        None
+    }
+}
+
 /// Local package mirror dir for a cluster (per-cluster map first).
 pub fn local_dir_for(config: &Config, cluster: &str) -> Option<String> {
     let f = config.flare.as_ref()?;

src/native/flare_tests.rs

@@ -129,6 +129,7 @@ fn per_cluster_mapping_wins_over_the_flat_block() {
             controller_node: None,
             local_dir: Some("/data/flare".into()),
             extra_flags: Vec::new(),
+            executable: None,
         },
     );
     assert_eq!(orbit_cluster_for(&c, "prod4"), "prod4");
@@ -429,3 +430,46 @@ fn an_orbit_list_that_does_not_name_our_nodes_stays_positional() {
     let node = cl.nodes.iter().find(|x| x.rank == 2).unwrap();
     assert_eq!(orbit_node_name(&c, "4nodes-switch", node), "spark3");
 }
+
+#[test]
+fn cluster_executable_override_reaches_orbit_run() {
+    // a locally built, ABI-compatible runtime (the Engram disk-reader fix)
+    // serves the lane instead of each node's packaged agent install
+    let mut c = cfg();
+    if let Some(f) = c.flare.as_mut() {
+        f.clusters.insert(
+            "4nodes-switch".into(),
+            crate::config::flare_cluster::FlareCluster {
+                executable: Some("/home/u/flare-patched/flare".into()),
+                ..Default::default()
+            },
+        );
+    }
+    let cmd = run_cmd(
+        &c,
+        &recipe(),
+        "4nodes-switch",
+        "prod4b",
+        "gx10-dgx1",
+        "/pkgs/x.safetensors",
+    )
+    .unwrap();
+    assert!(
+        cmd.contains("--executable '/home/u/flare-patched/flare'"),
+        "{cmd}"
+    );
+    // another cluster keeps the packaged install
+    let other = run_cmd(&c, &recipe(), "dgx58-switch", "sparks", "dgx5", "/p").unwrap();
+    assert!(!other.contains("--executable"), "{other}");
+}
+
+#[test]
+fn a_relative_executable_is_refused() {
+    // the CLI runs on the controller, so its cwd is not the operator's
+    let mut c = cfg();
+    c.flare.as_mut().unwrap().executable = Some("./target/release/flare".into());
+    assert_eq!(executable_for(&c, "dgx58-switch"), None);
+    let flat = "/opt/flare/flare".to_string();
+    c.flare.as_mut().unwrap().executable = Some(flat.clone());
+    assert_eq!(executable_for(&c, "dgx58-switch"), Some(flat));
+}