|
@@ -431,9 +431,28 @@ by node name (`gx10-dgx4` ↔ `dgx4.lan`), never by our rank index.
|
|
|
`executable` (per cluster, flat fallback) adds `orbit run --executable <path>`:
|
|
`executable` (per cluster, flat fallback) adds `orbit run --executable <path>`:
|
|
|
serving ranks run **that** build instead of each node's packaged agent install,
|
|
serving ranks run **that** build instead of each node's packaged agent install,
|
|
|
so a locally patched runtime serves a lane without touching `/usr/bin/flare`
|
|
so a locally patched runtime serves a lane without touching `/usr/bin/flare`
|
|
|
-(absolute paths only — the CLI's cwd is on the controller). That is how the
|
|
|
|
|
-Engram disk-reader fix (#184 + main) runs on `4nodes-switch`; the failure it
|
|
|
|
|
-fixes is documented in `specs/tech-architecture/flare-roce-poison.md`.
|
|
|
|
|
|
|
+(absolute paths only — the CLI's cwd is on the controller). It is the escape
|
|
|
|
|
+hatch for a one-off build; for a lane that must run patched permanently the
|
|
|
|
|
+build is **installed as the agent binary** on every rank node instead (that is
|
|
|
|
|
+the current state, see below).
|
|
|
|
|
+
|
|
|
|
|
+**Lane state (2026-11-04).** Production is `prod-no4`:
|
|
|
|
|
+`dsv41-flash-1017-4x-flare-gb10-nodgx4 --cluster prod-no4` = dgx1 (rank 0,
|
|
|
|
|
+API :30000) + dgx2 + dgx3 + **dgx8**, orbit `prod5`, controller dgx4.
|
|
|
|
|
+**dgx4 does not serve**: its NVMe is a budget `KingSpec XG7000-2TB 2242`
|
|
|
|
|
+(the only non-`ESL01TBTLCZ` disk in the fleet) and it cannot sustain the
|
|
|
|
|
+Engram random-read duty — the TP transport poisons on a cold-boot first
|
|
|
|
|
+chunk (`seq 93`, prefill=1) *and* after sustained load (`seq 2833389` after a
|
|
|
|
|
+full bench), with or without the upstream reader patch. `4nodes-switch`
|
|
|
|
|
+(`prod4b`, dgx1–4) stays configured for after that drive is replaced. Full
|
|
|
|
|
+evidence: `specs/tech-architecture/flare-roce-poison.md`.
|
|
|
|
|
+
|
|
|
|
|
+**Uniform binary rule.** The patched runtime (upstream `main` + PR #184) is
|
|
|
|
|
+installed as `~/.local/lib/flare-agent/flare` **and** `/usr/bin/flare` on
|
|
|
|
|
+dgx1–4 and dgx8 (old apt build saved as `*.orig-20261104`). Every rank node
|
|
|
|
|
+must run the same build: a 2026-11-04 boot with dgx8 still on the packaged
|
|
|
|
|
+`0.1.0` failed at rendezvous (`incompatible protocol` on rank 0, `failed to
|
|
|
|
|
+fill whole buffer` on the peer). After any install, all ranks before relaunch.
|
|
|
|
|
|
|
|
Two stores, both budgeted: `flare download` fills the **CLI store**
|
|
Two stores, both budgeted: `flare download` fills the **CLI store**
|
|
|
(`paths.package_cache`), and `orbit run` then stages into each node's **agent
|
|
(`paths.package_cache`), and `orbit run` then stages into each node's **agent
|