Răsfoiți Sursa

docs: current FLARE lane state, uniform-binary rule, refreshed upstream pin

- AGENTS.md: prod-no4 is production (dgx8 instead of dgx4), why dgx4 does
  not serve (KingSpec XG7000-2TB fails the Engram read duty cold and under
  load, with or without the reader patch), and the uniform-binary rule that
  a mixed runtime set fails at rendezvous.
- scripts/flare-upstream-watch.sh: follow the repo rename to
  NVIDIA/spark-local-inference and document that OUR runtime pin is
  upstream main + PR #184 (branch pr184-on-main) - rebase, rebuild, and
  reinstall on every rank node when bumping.
- specs/tech-architecture/flare-upstream.pin: 585159c9 (2026-11-04), the
  base the installed binary was built from (was 3b9e86a, 2026-10-22).
Alexander Korolev 10 ore în urmă
părinte
comite
2977785ed3
3 a modificat fișierele cu 35 adăugiri și 6 ștergeri
  1. 22 3
      AGENTS.md
  2. 12 2
      scripts/flare-upstream-watch.sh
  3. 1 1
      specs/tech-architecture/flare-upstream.pin

+ 22 - 3
AGENTS.md

@@ -431,9 +431,28 @@ by node name (`gx10-dgx4` ↔ `dgx4.lan`), never by our rank index.
 `executable` (per cluster, flat fallback) adds `orbit run --executable <path>`:
 serving ranks run **that** build instead of each node's packaged agent install,
 so a locally patched runtime serves a lane without touching `/usr/bin/flare`
-(absolute paths only — the CLI's cwd is on the controller). That is how the
-Engram disk-reader fix (#184 + main) runs on `4nodes-switch`; the failure it
-fixes is documented in `specs/tech-architecture/flare-roce-poison.md`.
+(absolute paths only — the CLI's cwd is on the controller). It is the escape
+hatch for a one-off build; for a lane that must run patched permanently the
+build is **installed as the agent binary** on every rank node instead (that is
+the current state, see below).
+
+**Lane state (2026-11-04).** Production is `prod-no4`:
+`dsv41-flash-1017-4x-flare-gb10-nodgx4 --cluster prod-no4` = dgx1 (rank 0,
+API :30000) + dgx2 + dgx3 + **dgx8**, orbit `prod5`, controller dgx4.
+**dgx4 does not serve**: its NVMe is a budget `KingSpec XG7000-2TB 2242`
+(the only non-`ESL01TBTLCZ` disk in the fleet) and it cannot sustain the
+Engram random-read duty — the TP transport poisons on a cold-boot first
+chunk (`seq 93`, prefill=1) *and* after sustained load (`seq 2833389` after a
+full bench), with or without the upstream reader patch. `4nodes-switch`
+(`prod4b`, dgx1–4) stays configured for after that drive is replaced. Full
+evidence: `specs/tech-architecture/flare-roce-poison.md`.
+
+**Uniform binary rule.** The patched runtime (upstream `main` + PR #184) is
+installed as `~/.local/lib/flare-agent/flare` **and** `/usr/bin/flare` on
+dgx1–4 and dgx8 (old apt build saved as `*.orig-20261104`). Every rank node
+must run the same build: a 2026-11-04 boot with dgx8 still on the packaged
+`0.1.0` failed at rendezvous (`incompatible protocol` on rank 0, `failed to
+fill whole buffer` on the peer). After any install, all ranks before relaunch.
 
 Two stores, both budgeted: `flare download` fills the **CLI store**
 (`paths.package_cache`), and `orbit run` then stages into each node's **agent

+ 12 - 2
scripts/flare-upstream-watch.sh

@@ -1,5 +1,15 @@
 #!/usr/bin/env bash
-# Watch NVIDIA/flare-local-inference upstream.
+# Watch NVIDIA/spark-local-inference upstream (renamed from
+# `flare-local-inference` on 2026-11-03; the old name redirects, so an older
+# clone keeps working).
+#
+# OUR RUNTIME PIN is not just the upstream HEAD: the prod lanes build
+# `main` + PR #184 ("Run parallel Engram disk workers on Linux with io_uring
+# lanes") — branch `pr184-on-main`, which is what the agent binary on the
+# nodes runs. When bumping the pin, rebase that branch onto the new HEAD,
+# rebuild, and reinstall on EVERY rank node (a mixed set fails at rendezvous
+# with "incompatible protocol"). Why that patch exists:
+# specs/tech-architecture/flare-roce-poison.md.
 #
 # FLARE moves several times a day (12+ commits in one night), and our clones,
 # the per-host agent binaries and the prepared packages all have to agree with
@@ -13,7 +23,7 @@
 # flare-upstream.pin) as a single line: <sha> <committer-date>.
 set -euo pipefail
 
-REPO="NVIDIA/flare-local-inference"
+REPO="NVIDIA/spark-local-inference"
 ROOT="$(cd "$(dirname "$0")/.." && pwd)"
 PIN_FILE="$ROOT/specs/tech-architecture/flare-upstream.pin"
 API="https://api.github.com/repos/$REPO"

+ 1 - 1
specs/tech-architecture/flare-upstream.pin

@@ -1 +1 @@
-3b9e86a000000000000000000000000000000000 2026-10-22T15:48
+585159c95ef6f5d126dd070a79f5f48e16d2b1c1 2026-11-04T02:14