|
|
@@ -20,9 +20,12 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
stays 30000** (operator directive), `--mtp-adaptive-verification-alpha 0.75` added
|
|
|
- [ ] **Every node on the v0.1.0 runtime** — **staged identically on dgx1/2/3/4/8 at
|
|
|
`/usr/local/share/hyperflare/v0.1.0/flare`** (archive `7ba5ba43…`, binary
|
|
|
- `e3fe697e…` 26,054,848 B, same sha256 on all five), nothing in `/usr/bin` or the
|
|
|
- agent path touched yet. Remaining: dgx5–7 (off) + the temp+`mv` swap and one
|
|
|
- restart window; **PR #184 rebuild dropped** (§6 item 5)
|
|
|
+ `e3fe697e…` 26,054,848 B, same sha256 on all five). **Swapped and running on dgx4**
|
|
|
+ (2026-11-12: `/usr/bin` + agent copy both `e3fe697e`, agent `exe` not `(deleted)`,
|
|
|
+ `flare.orig-20261112` backups) — the swap is rehearsed and known-good. Remaining:
|
|
|
+ dgx1/2/3/8 (needs the window, they hold the ranks) and dgx5–7 (off).
|
|
|
+ **PR #184 rebuild dropped** (§6 item 5). Note `~/.local/lib/flare-agent/flare` is
|
|
|
+ **not dpkg-owned** — apt never updates what the unit actually runs
|
|
|
- [x] **Current `gb10` core acquired onto the NAS store and verified** —
|
|
|
`checkpoint-expanded-scales/gb10-tp4-202611112207`, 474.66 GiB, **TP 4 · ready**,
|
|
|
signatures verified, visible from dgx1/dgx4 over NFS (Phase 2R)
|
|
|
@@ -52,8 +55,15 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
the cache flag, and the flag renamed `--package-cache` → `--core-cache` — see the
|
|
|
new “v0.1.0 vocabulary and paths” section. Still open: confirm the *launched*
|
|
|
cmdline after the first restart (needs the window)
|
|
|
-- [ ] **R5 (Q1)** — Phase 0.1 as rewritten: pre-flight that the packaged release is
|
|
|
- the binary every node starts into at cutover (no PR-184 rebuild — dropped)
|
|
|
+- [x] **R5 (Q1)** — Phase 0.1 pre-flight done, and the swap **rehearsed end-to-end on
|
|
|
+ idle dgx4**: `/usr/bin/flare` and `~/.local/lib/flare-agent/flare` both `e3fe697e`,
|
|
|
+ agent restarted (`systemctl --user restart flare-agent.service`, new pid,
|
|
|
+ `readlink /proc/<pid>/exe` **without `(deleted)`**), Nov-5 backups kept at both
|
|
|
+ paths as `flare.orig-20261112`. The mechanism behind the fleet's deleted inodes is
|
|
|
+ proven: **`dpkg -S ~/.local/lib/flare-agent/flare` → owned by no package**, so apt
|
|
|
+ refreshes `/usr/bin/flare` and *never* the copy the unit ExecStarts. dgx4 is
|
|
|
+ therefore ahead of dgx1/2/3/8 — do **not** add it to a group until the window swaps
|
|
|
+ the others (one identical binary per running group)
|
|
|
- [x] **R6a (Q1)** — Phase 3 bind-mount gate, part 1 on idle dgx4: v0.1.0 accepts the
|
|
|
layout, `--stage-only` passes **Verify core** through a NAS bind, mount survives
|
|
|
five stage attempts. Also found: the plan's Phase 3 procedure was **invalid**
|
|
|
@@ -78,15 +88,26 @@ The runtime check a round should run (staged-vs-running, never inferred from
|
|
|
|
|
|
```bash
|
|
|
for h in dgx1 dgx2 dgx3 dgx4 dgx8; do printf "%-6s " $h
|
|
|
- ssh -o BatchMode=yes $h.lan 'printf "staged=%s installed=%s running=%s\n" \
|
|
|
- "$(sha256sum /usr/local/share/hyperflare/v0.1.0/flare 2>/dev/null | cut -c1-8)" \
|
|
|
- "$(sha256sum /usr/bin/flare | cut -c1-8)" \
|
|
|
- "$(readlink /proc/$(pgrep -f "[f]lare run" | head -1)/exe 2>/dev/null)"' 2>/dev/null || echo off
|
|
|
+ ssh -o BatchMode=yes $h.lan 'A=$(pgrep -f "[f]lare run" | head -1); B=$(pgrep -f "[f]lare agent" | head -1)
|
|
|
+ printf "staged=%s /usrbin=%s agent=%s exe=%s\n" \
|
|
|
+ "$(sha256sum /usr/local/share/hyperflare/v0.1.0/flare 2>/dev/null | cut -c1-8)" \
|
|
|
+ "$(sha256sum /usr/bin/flare | cut -c1-8)" \
|
|
|
+ "$(sha256sum ~/.local/lib/flare-agent/flare | cut -c1-8)" \
|
|
|
+ "$(readlink /proc/${A:-$B}/exe 2>/dev/null)"' 2>/dev/null || echo off
|
|
|
done
|
|
|
```
|
|
|
|
|
|
-Expected today: `staged=e3fe697e installed=03c8e463 running=…flare-agent/flare (deleted)`.
|
|
|
-All three equal to `e3fe697e` (and no `(deleted)`) is the gate for this milestone.
|
|
|
+**`agent=` is the column that matters.** The unit starts
|
|
|
+`%h/.local/lib/flare-agent/flare`, and `dpkg` owns no such file, so `/usrbin=` can move
|
|
|
+while the runtime does not. Expected now (dgx4 swapped, ranks still on the Nov-5 build):
|
|
|
+
|
|
|
+```
|
|
|
+dgx1 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
|
|
|
+dgx4 staged=e3fe697e /usrbin=e3fe697e agent=e3fe697e exe=/home/kilork/.local/lib/flare-agent/flare
|
|
|
+```
|
|
|
+
|
|
|
+Gate for the milestone: `agent=e3fe697e` on every node, and no `exe` ending in
|
|
|
+`(deleted)`. A node reporting `agent=03c8e463` after an apt upgrade is **not updated**.
|
|
|
|
|
|
For the NAS mount, run from the controller; every node must report `fstab=1 live=1`:
|
|
|
|