Parcourir la source

docs(flare-storage): R5 pre-flight done — swap rehearsed on idle dgx4; the agent copy is what runs and dpkg does not own it

dgx4 now has /usr/bin/flare AND ~/.local/lib/flare-agent/flare at e3fe697e with the
agent restarted (new pid, readlink /proc/PID/exe has no "(deleted)"), Nov-5 backups
kept at both paths as flare.orig-20261112. Procedure proven: install temp + mv,
systemctl --user restart flare-agent.service, then verify sha + exe.

Root cause of the fleet's deleted inodes, proven not inferred: dpkg -S
~/.local/lib/flare-agent/flare finds no owning package, so apt refreshes
/usr/bin/flare and never touches the file the unit ExecStarts. "apt says updated"
has never meant "runtime updated" on this fleet.

The ## Checks probe now prints agent= beside /usrbin= and exe=, since agent= is the
only column describing what actually runs. dgx4 is intentionally ahead of dgx1/2/3/8
(it holds no ranks) and must not join a group until the window swaps the others.
Alexander Korolev il y a 1 semaine
Parent
commit
f392e4764c
1 fichiers modifiés avec 32 ajouts et 11 suppressions
  1. 32 11
      specs/tech-architecture/flare-storage-nas.md

+ 32 - 11
specs/tech-architecture/flare-storage-nas.md

@@ -20,9 +20,12 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       stays 30000** (operator directive), `--mtp-adaptive-verification-alpha 0.75` added
 - [ ] **Every node on the v0.1.0 runtime** — **staged identically on dgx1/2/3/4/8 at
       `/usr/local/share/hyperflare/v0.1.0/flare`** (archive `7ba5ba43…`, binary
-      `e3fe697e…` 26,054,848 B, same sha256 on all five), nothing in `/usr/bin` or the
-      agent path touched yet. Remaining: dgx5–7 (off) + the temp+`mv` swap and one
-      restart window; **PR #184 rebuild dropped** (§6 item 5)
+      `e3fe697e…` 26,054,848 B, same sha256 on all five). **Swapped and running on dgx4**
+      (2026-11-12: `/usr/bin` + agent copy both `e3fe697e`, agent `exe` not `(deleted)`,
+      `flare.orig-20261112` backups) — the swap is rehearsed and known-good. Remaining:
+      dgx1/2/3/8 (needs the window, they hold the ranks) and dgx5–7 (off).
+      **PR #184 rebuild dropped** (§6 item 5). Note `~/.local/lib/flare-agent/flare` is
+      **not dpkg-owned** — apt never updates what the unit actually runs
 - [x] **Current `gb10` core acquired onto the NAS store and verified** —
       `checkpoint-expanded-scales/gb10-tp4-202611112207`, 474.66 GiB, **TP 4 · ready**,
       signatures verified, visible from dgx1/dgx4 over NFS (Phase 2R)
@@ -52,8 +55,15 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       the cache flag, and the flag renamed `--package-cache` → `--core-cache` — see the
       new “v0.1.0 vocabulary and paths” section. Still open: confirm the *launched*
       cmdline after the first restart (needs the window)
-- [ ] **R5 (Q1)** — Phase 0.1 as rewritten: pre-flight that the packaged release is
-      the binary every node starts into at cutover (no PR-184 rebuild — dropped)
+- [x] **R5 (Q1)** — Phase 0.1 pre-flight done, and the swap **rehearsed end-to-end on
+      idle dgx4**: `/usr/bin/flare` and `~/.local/lib/flare-agent/flare` both `e3fe697e`,
+      agent restarted (`systemctl --user restart flare-agent.service`, new pid,
+      `readlink /proc/<pid>/exe` **without `(deleted)`**), Nov-5 backups kept at both
+      paths as `flare.orig-20261112`. The mechanism behind the fleet's deleted inodes is
+      proven: **`dpkg -S ~/.local/lib/flare-agent/flare` → owned by no package**, so apt
+      refreshes `/usr/bin/flare` and *never* the copy the unit ExecStarts. dgx4 is
+      therefore ahead of dgx1/2/3/8 — do **not** add it to a group until the window swaps
+      the others (one identical binary per running group)
 - [x] **R6a (Q1)** — Phase 3 bind-mount gate, part 1 on idle dgx4: v0.1.0 accepts the
       layout, `--stage-only` passes **Verify core** through a NAS bind, mount survives
       five stage attempts. Also found: the plan's Phase 3 procedure was **invalid**
@@ -78,15 +88,26 @@ The runtime check a round should run (staged-vs-running, never inferred from
 
 ```bash
 for h in dgx1 dgx2 dgx3 dgx4 dgx8; do printf "%-6s " $h
-  ssh -o BatchMode=yes $h.lan 'printf "staged=%s installed=%s running=%s\n" \
-    "$(sha256sum /usr/local/share/hyperflare/v0.1.0/flare 2>/dev/null | cut -c1-8)" \
-    "$(sha256sum /usr/bin/flare | cut -c1-8)" \
-    "$(readlink /proc/$(pgrep -f "[f]lare run" | head -1)/exe 2>/dev/null)"' 2>/dev/null || echo off
+  ssh -o BatchMode=yes $h.lan 'A=$(pgrep -f "[f]lare run" | head -1); B=$(pgrep -f "[f]lare agent" | head -1)
+    printf "staged=%s /usrbin=%s agent=%s exe=%s\n" \
+      "$(sha256sum /usr/local/share/hyperflare/v0.1.0/flare 2>/dev/null | cut -c1-8)" \
+      "$(sha256sum /usr/bin/flare | cut -c1-8)" \
+      "$(sha256sum ~/.local/lib/flare-agent/flare | cut -c1-8)" \
+      "$(readlink /proc/${A:-$B}/exe 2>/dev/null)"' 2>/dev/null || echo off
 done
 ```
 
-Expected today: `staged=e3fe697e installed=03c8e463 running=…flare-agent/flare (deleted)`.
-All three equal to `e3fe697e` (and no `(deleted)`) is the gate for this milestone.
+**`agent=` is the column that matters.** The unit starts
+`%h/.local/lib/flare-agent/flare`, and `dpkg` owns no such file, so `/usrbin=` can move
+while the runtime does not. Expected now (dgx4 swapped, ranks still on the Nov-5 build):
+
+```
+dgx1  staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
+dgx4  staged=e3fe697e /usrbin=e3fe697e agent=e3fe697e exe=/home/kilork/.local/lib/flare-agent/flare
+```
+
+Gate for the milestone: `agent=e3fe697e` on every node, and no `exe` ending in
+`(deleted)`. A node reporting `agent=03c8e463` after an apt upgrade is **not updated**.
 
 For the NAS mount, run from the controller; every node must report `fstab=1 live=1`: