# FLARE `transport is poisoned`: an Engram disk stall, not a RoCE fault (2026-11-03)

The prod FLARE lane on dgx1–4 (`dsv41-flash-1017-4x-flare-gb10-prod4`) died
repeatedly with two messages that look like fabric failures:

```
tensor-parallel peer failed: RoCE collective on rank 2 timed out waiting for
rank 0, HCA 0, at sequence 7122648; the transport is poisoned

RDMA write to rank 3 failed: transport retry counter exceeded
(vendor_err 0x81, seq 250342)
```

They are **not** fabric faults on this stand. The first is the collective
spin-wait expiring because a peer rank is starved on **disk-backed Engram
reads**; the second belongs to a different, rarer class (real QP retry
exhaustion) and did not reproduce in any 2026-11-03 run. Everything below is
measured, on the production package and flags.

## Verdict

- `RoCE collective … timed out … transport is poisoned` — the b12x/RoCEnante
  GPU kernel polls a peer's stripe flag for `DEFAULT_SPIN_LIMIT = 20_000_000`
  polls (≈20 s on GB10; compiled **into the prepared package**, not a
  serve-time flag) and then poisons the transport. FLARE is fail-stop: no
  retry, no reroute, restart every rank.
- The stall is `flare_engram_disk_wait_nanoseconds_total` on the starved rank:
  prefill advances in bursts, GPU idle 10–20 s at a time, and the slowest
  reader still gets its data — the *peer's* spin limit expires first.
- Root cause on this stand: **dgx4's SSD**. All four prod nodes run the
  generic `ESL01TBTLCZ-27J2-TYN` 1 TB drive except dgx4, which carries a
  **KingSpec XG7000-2TB 2242** (fw `GT0d9d63`, vid `0x1e4b`) — a budget
  consumer drive, no first-party firmware tool, our firmware not even listed
  in the public database (`SN13683` is what smarthdd knows for this model).
  Sustained random-read ceiling measured on the wedge node: **≈29 K IOPS**
  (4 threads, O_DIRECT 4 K, 10 s) vs ≈33 K on a healthy dgx5 — while a
  first-touch prefill demanded **≈24 K reads/s**, i.e. 80–90 % of the drive.
- **Mitigation that works today**: the unmerged upstream io_uring reader
  (PR #184) on top of current `main` — **partially**. Same hardware, same
  package, same prompt: 3/3 wedges before, clean run after **on a warm stand**
  (numbers below). On a **freshly rebooted** node the cold-boot wedge persists:
  2026-11-04, 07:12 UTC boot, first long request ~9 min later died at
  `sequence 93` with prefill frozen at **token 1** — dgx4's drive could not
  deliver the first chunk's ≈350 MB of rows within the ≈8 s spin window even
  with the parallel reader (its reads ran hot at ≈87 K reads/s for ~4 s and
  then stalled mid-burst). Removing dgx4 remains the only complete fix on
  this hardware; the reader patch is strictly beneficial but not sufficient
  on a cold stand.

## Evidence

### Run ledger (same package, same flags, 157 K-token first-touch prompt)

| # | arrangement | runtime | result |
|---|---|---|---|
| 1 | orbit `prod4`, rank 0 = dgx4 | packaged `flare` | **poison** @ seq 7122648 (93 K tokens prefilled) |
| 2 | orbit `prod4`, rank 0 = dgx4 | packaged `flare` | **poison** @ seq 2109 (frozen ~93 K) |
| 3 | orbit `prod4`, rank 0 = dgx4 | packaged `flare` | **poison** @ seq 1284 (frozen ~61 K) |
| 4 | orbit `prod4b`, rank 0 = **dgx1**, dgx4 = rank 3 | packaged `flare` | **poison** @ seq 1557 (`rank 1 waiting for rank 3`) |
| 5 | orbit `prod5` = dgx1+dgx2+dgx3+**dgx8**, no dgx4 | packaged `flare` | **clean**, 0→157,086 in ~61 s (×2) |
| 6 | orbit `prod4b`, dgx4 = rank 3 | **main + PR #184** | **clean**, 0→157,086 in ~60 s (~2,600 t/s) — warm stand |
| 8 | orbit `prod4b` (2026-11-04, **freshly rebooted** node) | **main + PR #184** | **poison** @ seq 93, prefill frozen at **token 1** — the cold-boot case survives the reader patch |
| 7 | orbit `sparks` = dgx5–8 (2 runs, incl. after 45 min idle) | packaged `flare` | clean |

Runs 4 and 6 matter most: with dgx4 demoted from rank 0 the failure
**followed the node, not the rank**. Run 5 isolates it further: remove dgx4
entirely and the packaged binary is clean. The wedging pair is always
a peer waiting on **dgx4's** data. Run 8 adds the cold-boot dimension: the
reader patch rescues a warm stand but not the first long request after a
reboot — the drive stalls mid-burst inside the spin window at chunk 1.

### Rank 0 metrics through one wedge (1 Hz, `prod4`, layout `ESP`)

| t | prefill | engram disk reads | disk wait (cum) |
|---|---|---|---|
| 20:52:55 | 52,993 | 2.32 M | 17.01 s |
| 20:52:56–59 | **frozen** | +108 K reads | 17.01 s (frozen) |
| 20:53:00–01 | **frozen** | slow | **27.17 s (+10.2 s in 2 s)** |
| 20:53:02 | burst → 61,185 | +2.6 K | 27.17 s |
| 20:53:03–08 | **frozen** | frozen | 27.17 s |
| 20:53:09 | server dead — `poisoned`, seq 1284 | | |

The disk queue backs up while the GPU starves; `engram_row_cache_misses`
keeps climbing (69 % miss with the 1 GiB row cache the package recommends for
unified-memory GB10).

### Wire counters (all four nodes, sampled every 10 s across the wedges)

`local_ack_timeout_err`, `req_transport_retries_exceeded`, `req_cqe_error`:
**0 everywhere, all day**. `out_of_sequence` / `packet_seq_err` /
`roce_adp_retrans` are non-zero but flat-rate and retried successfully
(98.7 M RDMA write requests on dgx4). `rx_write_requests` freezes on all four
ranks simultaneously at the wedge — writes stop because the *producer* stopped,
not because the fabric dropped them. No mlx5/Xid events in the journal.

### What did **not** help

- `--embedding-table-cache-size 4GiB` (from 1 GiB): wedged **faster**
  (froze ~35 K). More cached rows did not change the read burst pattern.
- `--embedding-table-workers 4`: rejected —
  `parallel embedding disk workers require Windows`. The Linux multi-lane
  reader exists only as unmerged PR #184.
- MTU / GID / link checks: MTU 9000 on both CX7 functions, RoCEv2 IPv4 GID at
  index 3, static addresses (one /24 per host pair), `ping -M do -s 8972`
  clean, SMART clean (`media_errors 0`, `num_err_log_entries 0`, 42 °C).
- Rank-0 role / controller co-residency: disproved by runs 4 and 5.

### The mitigation (our build)

`pr184-on-main` = `NVIDIA/spark-local-inference` `main` (`44ec239`) +
cherry-pick of PR #184 `4ed10e1b` ("Run parallel Engram disk workers on Linux
with io_uring lanes"). Cherry-pick merges cleanly; 5 files, +508/−309, all in
the runtime (`helios/src/engram/reader*`, `runtime_package/selection.rs`,
`docs/cli-design.md`) — no package ABI change, the same prepared packages
serve unchanged. Release build aarch64 on a Spark: ~50 s.

Deployed with `orbit run --executable /home/kilork/flare-patched/flare`
(launcher: `executable` in `[flare.clusters.<cluster>]`), packaged agent
install untouched.

Result (run 6): prefill 0 → 157,086 tokens in ~60 s, no burst gaps, total
`engram_disk_wait` **48.7 s** — statistically identical to the healthy dgx5
lane (48.4 s). dgx4's KingSpec drive is still the slowest disk in the fleet;
the pipelined reader simply keeps it under the collective's patience.

**But not on a cold stand** (run 8, 2026-11-04): after a node reboot the
first long request wedges at `sequence 93`, prefill frozen at token 1. Rank
0's 1 Hz capture shows the first chunk's reads running hot (≈87 K reads/s,
disk wait +2.7 s) and then the whole runtime dead within ~8 s — dgx4's drive
stalls mid-burst inside the spin window, which the parallel reader raises in
throughput but cannot bound in time. Operationally this means: after every
reboot, a cold stand can lose its first long prompt even with the patch.
The production lane therefore excludes dgx4 (`prod-no4`) until the drive is
replaced; the `prod4b`+patch configuration is reserved for after the swap.

## Production decision (2026-11-04)

- Serving lane: **`prod-no4`** = dgx1+dgx2+dgx3+dgx8 (orbit `prod5`, rank 0
  = dgx1), recipe `dsv41-flash-1017-4x-flare-gb10-nodgx4`. Clean on the same
  prompts that killed the dgx1–4 lane, with or without the reader patch.
- `prod4b` + `executable` stays configured for **after the NVMe swap**
  (Samsung PM9A3/PM9E1 M.2 2242 or Corsair MP700 Micro class; back up
  `~/.config/flare/orbit/controller.key` before the DGX OS reinstall to skip
  re-pairing).
- The `prod4` orbit (rank 0 = dgx4) remains in `orbit.toml` for reference.

## Upstream material

### Status in `NVIDIA/spark-local-inference` (was `flare-local-inference`)

- Poison = fail-stop is **documented behaviour** (`docs/multi-host.md`,
  `docs/roce-ring.md`): "A collective failure invalidates the serving run …
  stop and restart all ranks together".
- Open issue **#151** carries the same message text, but deterministic at
  `sequence 1` with two rails on the `f1` functions; a commenter serves fine
  on `f0`. Not this failure (ours is mid-run and needs no second rail).
- Open issue **#78** documents disk-Engram first-touch prefill cost (28.6 s
  @250 K, 61.3 s @500 K on a 4× RTX PRO box); cache budget 8→32 GiB does not
  fix first touch.
- PR **#184** (parallel Windows-style lanes via io_uring on Linux) was closed
  unmerged on 2026-11-03 with "the filesystem was the real bottleneck" —
  measured on Btrfs. Our counterexample is ext4 on a GB10 Spark where the
  single-lane reader is the difference between a dead lane and a clean one.
- The transport code (`helios/native/roce_proxy.c`) has not changed since
  `e350914a` (2026-10-21) apart from chain support and licensing.
- **Nothing upstream tracks a stalled rank turning into a poisoned
  transport**, and only rank 0 exposes `/metrics` — the ranks that wait are
  blind.

### Issue draft (ready to paste)

> **Title:** Disk-Engram read stall on one rank poisons the TP transport
> (`RoCE collective … timed out … transport is poisoned`) — reproduced 3/3,
> fixed by PR #184's io_uring reader
>
> **Summary.** On a 4× DGX Spark (GB10) TP4 lane serving
> `local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE` (`gb10`,
> `--embedding-table-storage disk --embedding-table-cache-size 1GiB
> --embedding-table-resident-scales`, `--max-concurrency 32
> --mtp-draft-tokens 5 --mtp-adaptive-verification`), a 157 K-token
> first-touch prompt kills the run on the same hardware it serves well
> otherwise:
>
> ```
> RoCE collective on rank 2 timed out waiting for rank 0, HCA 0, at
> sequence 7122648; the transport is poisoned
> ```
>
> **Diagnosis.** Rank 0's `/metrics` shows stop-and-go prefill:
> `flare_prefill_tokens_total` frozen for 3–6 s at a time while
> `flare_engram_disk_wait_nanoseconds_total` jumps (+10.2 s within one 2 s
> window) and `flare_engram_row_cache_misses_total` climbs at ≈24 K
> random-reads/s — 80–90 % of that node's measured 4 K O_DIRECT ceiling
> (≈29 K IOPS). The waiting rank hits the 20 M-poll kernel spin limit
> (`DEFAULT_SPIN_LIMIT`, compiled into the package) and poisons the
> transport. RDMA counters are clean throughout: `local_ack_timeout_err`,
> `req_transport_retries_exceeded`, `req_cqe_error` all 0;
> `rx_write_requests` freezes on every rank because the producer stopped.
>
> **Rank-0 role is not the trigger.** Reordering the orbit so another node
> is rank 0 still fails — the same node's data is what the peer waits for.
> Removing that node from the cluster (dgx1+dgx2+dgx3+dgx8) serves the same
> prompt cleanly with the packaged binary.
>
> **Fix (partial).** Current `main` + PR #184 cherry-picked (`pr184-on-main`)
> serves the identical package and prompt on a warm stand: prefill 0→157 K in
> ~60 s, total Engram disk wait 48.7 s (healthy-lane parity), no poison — 1/1
> after 3/3 failures without it. **On a freshly rebooted node the wedge
> persists** (2026-11-04): the first long request dies at `sequence 93` with
> prefill frozen at token 1 — the first chunk's ≈350 MB of rows must arrive
> within the ≈8 s spin window and the slow drive stalls mid-burst (reads hot
> at ≈87 K/s, then the runtime is dead). Excluding that node from the cluster
> (`dgx1+dgx2+dgx3+dgx8`) is the only configuration that survived every case,
> with or without the patch.
>
> **Requests.**
> 1. Reconsider #184: its close reasoning ("the filesystem was the
>    bottleneck") holds for Btrfs, but on ext4/GB10 the single-lane reader
>    is what makes a warm stand survive a disk that is merely slow.
> 2. Make the collective spin limit configurable at serve time (or make it
>    Engram-aware / auto-extend on the first chunks of a cold stand) — it is
>    currently baked into the prepared package, so operators cannot trade
>    latency for stability. The cold-boot case (run 8) dies at token 1:
>    no reader improvement fixes a deadline that is shorter than the first
>    chunk's cold read time.
> 3. Expose `/metrics` on **every** rank (or at least the readiness/disk
>    counters): the rank that waits is invisible today, which turns a disk
>    stall into an unattributable fabric error.
> 4. Consider a preflight/warning when a host's Engram disk falls below the
>    package's implied read rate; a 6 W consumer 2242 at 29 K IOPS is not
>    the same class as the reference platforms. A serve-time flag to shrink
>    the prefill chunk (package-baked at 4096 tokens here) would also bound
>    the per-chunk read volume on slow disks.

## Operational state after this investigation

- Production lane: orbit **`prod4b`** (`dgx1 → dgx2 → dgx3 → dgx4`,
  rank 0 = dgx1) + `executable = /home/kilork/flare-patched/flare`
  (staged on all four prod nodes; only the controller's copy is uploaded).
- `prod4` (rank 0 = dgx4) is kept in `orbit.toml` for rollback; `prod5`
  (dgx1+dgx2+dgx3+dgx8) is the no-dgx4 control.
- Hardware follow-up still open: replace dgx4's KingSpec
  `XG7000-2TB 2242` with an enterprise-class 2242 (Samsung PM9A3/PM9E1 M.2
  2242, or Corsair MP700 Micro) — the software patch buys stability, not a
  fast disk. Back up `~/.config/flare/orbit/controller.key` first: restoring
  it after a DGX OS reinstall avoids re-pairing all eight agents.