launcher/specs/tech-architecture/flare-roce-poison.md

FLARE transport is poisoned: an Engram disk stall, not a RoCE fault (2026-11-03)

The prod FLARE lane on dgx1–4 (dsv41-flash-1017-4x-flare-gb10-prod4) died repeatedly with two messages that look like fabric failures:

tensor-parallel peer failed: RoCE collective on rank 2 timed out waiting for
rank 0, HCA 0, at sequence 7122648; the transport is poisoned

RDMA write to rank 3 failed: transport retry counter exceeded
(vendor_err 0x81, seq 250342)

They are not fabric faults on this stand. The first is the collective spin-wait expiring because a peer rank is starved on disk-backed Engram reads; the second belongs to a different, rarer class (real QP retry exhaustion) and did not reproduce in any 2026-11-03 run. Everything below is measured, on the production package and flags.

Verdict

• RoCE collective … timed out … transport is poisoned — the b12x/RoCEnante GPU kernel polls a peer's stripe flag for DEFAULT_SPIN_LIMIT = 20_000_000 polls (≈20 s on GB10; compiled into the prepared package, not a serve-time flag) and then poisons the transport. FLARE is fail-stop: no retry, no reroute, restart every rank.
• The stall is flare_engram_disk_wait_nanoseconds_total on the starved rank: prefill advances in bursts, GPU idle 10–20 s at a time, and the slowest reader still gets its data — the peer's spin limit expires first.
• Root cause on this stand: dgx4's SSD. All four prod nodes run the generic ESL01TBTLCZ-27J2-TYN 1 TB drive except dgx4, which carries a KingSpec XG7000-2TB 2242 (fw GT0d9d63, vid 0x1e4b) — a budget consumer drive, no first-party firmware tool, our firmware not even listed in the public database (SN13683 is what smarthdd knows for this model). Sustained random-read ceiling measured on the wedge node: ≈29 K IOPS (4 threads, O_DIRECT 4 K, 10 s) vs ≈33 K on a healthy dgx5 — while a first-touch prefill demanded ≈24 K reads/s, i.e. 80–90 % of the drive.
• Mitigation that works today: the unmerged upstream io_uring reader (PR #184) on top of current main — partially. Same hardware, same package, same prompt: 3/3 wedges before, clean run after on a warm stand (numbers below). On a freshly rebooted node the cold-boot wedge persists: 2026-11-04, 07:12 UTC boot, first long request ~9 min later died at sequence 93 with prefill frozen at token 1 — dgx4's drive could not deliver the first chunk's ≈350 MB of rows within the ≈8 s spin window even with the parallel reader (its reads ran hot at ≈87 K reads/s for ~4 s and then stalled mid-burst). Removing dgx4 remains the only complete fix on this hardware; the reader patch is strictly beneficial but not sufficient on a cold stand.

Evidence

Run ledger (same package, same flags, 157 K-token first-touch prompt)

#	arrangement	runtime	result
1	orbit prod4, rank 0 = dgx4	packaged flare	poison @ seq 7122648 (93 K tokens prefilled)
2	orbit prod4, rank 0 = dgx4	packaged flare	poison @ seq 2109 (frozen ~93 K)
3	orbit prod4, rank 0 = dgx4	packaged flare	poison @ seq 1284 (frozen ~61 K)
4	orbit prod4b, rank 0 = dgx1, dgx4 = rank 3	packaged flare	poison @ seq 1557 (rank 1 waiting for rank 3)
5	orbit prod5 = dgx1+dgx2+dgx3+dgx8, no dgx4	packaged flare	clean, 0→157,086 in ~61 s (×2)
6	orbit prod4b, dgx4 = rank 3	main + PR #184	clean, 0→157,086 in ~60 s (~2,600 t/s) — warm stand
8	orbit prod4b (2026-11-04, freshly rebooted node)	main + PR #184	poison @ seq 93, prefill frozen at token 1 — the cold-boot case survives the reader patch
7	orbit sparks = dgx5–8 (2 runs, incl. after 45 min idle)	packaged flare	clean

Runs 4 and 6 matter most: with dgx4 demoted from rank 0 the failure followed the node, not the rank. Run 5 isolates it further: remove dgx4 entirely and the packaged binary is clean. The wedging pair is always a peer waiting on dgx4's data. Run 8 adds the cold-boot dimension: the reader patch rescues a warm stand but not the first long request after a reboot — the drive stalls mid-burst inside the spin window at chunk 1.

Rank 0 metrics through one wedge (1 Hz, prod4, layout ESP)

t	prefill	engram disk reads	disk wait (cum)
20:52:55	52,993	2.32 M	17.01 s
20:52:56–59	frozen	+108 K reads	17.01 s (frozen)
20:53:00–01	frozen	slow	27.17 s (+10.2 s in 2 s)
20:53:02	burst → 61,185	+2.6 K	27.17 s
20:53:03–08	frozen	frozen	27.17 s
20:53:09	server dead — poisoned, seq 1284

The disk queue backs up while the GPU starves; engram_row_cache_misses keeps climbing (69 % miss with the 1 GiB row cache the package recommends for unified-memory GB10).

Wire counters (all four nodes, sampled every 10 s across the wedges)

local_ack_timeout_err, req_transport_retries_exceeded, req_cqe_error: 0 everywhere, all day. out_of_sequence / packet_seq_err / roce_adp_retrans are non-zero but flat-rate and retried successfully (98.7 M RDMA write requests on dgx4). rx_write_requests freezes on all four ranks simultaneously at the wedge — writes stop because the producer stopped, not because the fabric dropped them. No mlx5/Xid events in the journal.

What did not help

• --embedding-table-cache-size 4GiB (from 1 GiB): wedged faster (froze ~35 K). More cached rows did not change the read burst pattern.
• --embedding-table-workers 4: rejected — parallel embedding disk workers require Windows. The Linux multi-lane reader exists only as unmerged PR #184.
• MTU / GID / link checks: MTU 9000 on both CX7 functions, RoCEv2 IPv4 GID at index 3, static addresses (one /24 per host pair), ping -M do -s 8972 clean, SMART clean (media_errors 0, num_err_log_entries 0, 42 °C).
• Rank-0 role / controller co-residency: disproved by runs 4 and 5.

The mitigation (our build)

pr184-on-main = NVIDIA/spark-local-inference main (44ec239) + cherry-pick of PR #184 4ed10e1b ("Run parallel Engram disk workers on Linux with io_uring lanes"). Cherry-pick merges cleanly; 5 files, +508/−309, all in the runtime (helios/src/engram/reader*, runtime_package/selection.rs, docs/cli-design.md) — no package ABI change, the same prepared packages serve unchanged. Release build aarch64 on a Spark: ~50 s.

Deployed with orbit run --executable /home/kilork/flare-patched/flare (launcher: executable in [flare.clusters.<cluster>]), packaged agent install untouched.

Result (run 6): prefill 0 → 157,086 tokens in ~60 s, no burst gaps, total engram_disk_wait 48.7 s — statistically identical to the healthy dgx5 lane (48.4 s). dgx4's KingSpec drive is still the slowest disk in the fleet; the pipelined reader simply keeps it under the collective's patience.

But not on a cold stand (run 8, 2026-11-04): after a node reboot the first long request wedges at sequence 93, prefill frozen at token 1. Rank 0's 1 Hz capture shows the first chunk's reads running hot (≈87 K reads/s, disk wait +2.7 s) and then the whole runtime dead within ~8 s — dgx4's drive stalls mid-burst inside the spin window, which the parallel reader raises in throughput but cannot bound in time. Operationally this means: after every reboot, a cold stand can lose its first long prompt even with the patch. The production lane therefore excludes dgx4 (prod-no4) until the drive is replaced; the prod4b+patch configuration is reserved for after the swap.

Production decision (2026-11-04)

• Serving lane: prod-no4 = dgx1+dgx2+dgx3+dgx8 (orbit prod5, rank 0 = dgx1), recipe dsv41-flash-1017-4x-flare-gb10-nodgx4. Clean on the same prompts that killed the dgx1–4 lane, with or without the reader patch.
• prod4b + executable stays configured for after the NVMe swap (Samsung PM9A3/PM9E1 M.2 2242 or Corsair MP700 Micro class; back up ~/.config/flare/orbit/controller.key before the DGX OS reinstall to skip re-pairing).
• The prod4 orbit (rank 0 = dgx4) remains in orbit.toml for reference.

Upstream material

Status in NVIDIA/spark-local-inference (was flare-local-inference)

• Poison = fail-stop is documented behaviour (docs/multi-host.md, docs/roce-ring.md): "A collective failure invalidates the serving run … stop and restart all ranks together".
• Open issue #151 carries the same message text, but deterministic at sequence 1 with two rails on the f1 functions; a commenter serves fine on f0. Not this failure (ours is mid-run and needs no second rail).
• Open issue #78 documents disk-Engram first-touch prefill cost (28.6 s @250 K, 61.3 s @500 K on a 4× RTX PRO box); cache budget 8→32 GiB does not fix first touch.
• PR #184 (parallel Windows-style lanes via io_uring on Linux) was closed unmerged on 2026-11-03 with "the filesystem was the real bottleneck" — measured on Btrfs. Our counterexample is ext4 on a GB10 Spark where the single-lane reader is the difference between a dead lane and a clean one.
• The transport code (helios/native/roce_proxy.c) has not changed since e350914a (2026-10-21) apart from chain support and licensing.
• Nothing upstream tracks a stalled rank turning into a poisoned transport, and only rank 0 exposes /metrics — the ranks that wait are blind.

Issue draft (ready to paste)

Title: Disk-Engram read stall on one rank poisons the TP transport (RoCE collective … timed out … transport is poisoned) — reproduced 3/3, fixed by PR #184's io_uring reader

Summary. On a 4× DGX Spark (GB10) TP4 lane serving local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE (gb10, --embedding-table-storage disk --embedding-table-cache-size 1GiB --embedding-table-resident-scales, --max-concurrency 32 --mtp-draft-tokens 5 --mtp-adaptive-verification), a 157 K-token first-touch prompt kills the run on the same hardware it serves well otherwise:

RoCE collective on rank 2 timed out waiting for rank 0, HCA 0, at
sequence 7122648; the transport is poisoned

Diagnosis. Rank 0's /metrics shows stop-and-go prefill: flare_prefill_tokens_total frozen for 3–6 s at a time while flare_engram_disk_wait_nanoseconds_total jumps (+10.2 s within one 2 s window) and flare_engram_row_cache_misses_total climbs at ≈24 K random-reads/s — 80–90 % of that node's measured 4 K O_DIRECT ceiling (≈29 K IOPS). The waiting rank hits the 20 M-poll kernel spin limit (DEFAULT_SPIN_LIMIT, compiled into the package) and poisons the transport. RDMA counters are clean throughout: local_ack_timeout_err, req_transport_retries_exceeded, req_cqe_error all 0; rx_write_requests freezes on every rank because the producer stopped.

Rank-0 role is not the trigger. Reordering the orbit so another node is rank 0 still fails — the same node's data is what the peer waits for. Removing that node from the cluster (dgx1+dgx2+dgx3+dgx8) serves the same prompt cleanly with the packaged binary.

Fix (partial). Current main + PR #184 cherry-picked (pr184-on-main) serves the identical package and prompt on a warm stand: prefill 0→157 K in ~60 s, total Engram disk wait 48.7 s (healthy-lane parity), no poison — 1/1 after 3/3 failures without it. On a freshly rebooted node the wedge persists (2026-11-04): the first long request dies at sequence 93 with prefill frozen at token 1 — the first chunk's ≈350 MB of rows must arrive within the ≈8 s spin window and the slow drive stalls mid-burst (reads hot at ≈87 K/s, then the runtime is dead). Excluding that node from the cluster (dgx1+dgx2+dgx3+dgx8) is the only configuration that survived every case, with or without the patch.

Requests.

1. Reconsider #184: its close reasoning ("the filesystem was the bottleneck") holds for Btrfs, but on ext4/GB10 the single-lane reader is what makes a warm stand survive a disk that is merely slow.
2. Make the collective spin limit configurable at serve time (or make it Engram-aware / auto-extend on the first chunks of a cold stand) — it is currently baked into the prepared package, so operators cannot trade latency for stability. The cold-boot case (run 8) dies at token 1: no reader improvement fixes a deadline that is shorter than the first chunk's cold read time.
3. Expose /metrics on every rank (or at least the readiness/disk counters): the rank that waits is invisible today, which turns a disk stall into an unattributable fabric error.
4. Consider a preflight/warning when a host's Engram disk falls below the package's implied read rate; a 6 W consumer 2242 at 29 K IOPS is not the same class as the reference platforms. A serve-time flag to shrink the prefill chunk (package-baked at 4096 tokens here) would also bound the per-chunk read volume on slow disks.

Operational state after this investigation

• Production lane: orbit prod4b (dgx1 → dgx2 → dgx3 → dgx4, rank 0 = dgx1) + executable = /home/kilork/flare-patched/flare (staged on all four prod nodes; only the controller's copy is uploaded).
• prod4 (rank 0 = dgx4) is kept in orbit.toml for rollback; prod5 (dgx1+dgx2+dgx3+dgx8) is the no-dgx4 control.
• Hardware follow-up still open: replace dgx4's KingSpec XG7000-2TB 2242 with an enterprise-class 2242 (Samsung PM9A3/PM9E1 M.2 2242, or Corsair MP700 Micro) — the software patch buys stability, not a fast disk. Back up ~/.config/flare/orbit/controller.key first: restoring it after a DGX OS reinstall avoids re-pairing all eight agents.