Parcourir la source

docs(flare-storage): Phase 3 bind gate half-passes; a standalone model-dir copy is invalid in v0.1.0; rm -rf deletes through live binds

Validated the Phase 4 shape on idle dgx4 with the staged v0.1.0 binary: per-node
artefacts on the NAS (nodes/<node>/<model>/<version>/deployments/<id>) bind-mounted
over the model dir's deployments inside a store root. flare run --stage-only passed
Verify core and reached Wait for tensor-parallel hosts; the bind was unchanged
across five attempts (same inode, findmnt still showing the NAS source). models list
is bind-safe too: full inventory (474.66 GiB, 181 hardlinked objects) with blobs/
and weights/ bind-mounted from a different device (NFS st_dev 80 vs local 66306).
The symlink ban is explained exactly: escape check = openat(7,"tp4",O_NOFOLLOW|DIR).

The plan's Phase 3 steps 1-2 were wrong: a copied-out model dir fails v0.1.0 with
"open core directory: No such file or directory" because the envelope resolves its
core within the store that contains it and a store has no metadata files at all -
all context is structural. Replaced with the verified command shape.

Data-loss trap found the hard way: cleanup ran rm -rf over a tree with two live
binds into the NAS store. rm unmounts nothing - the mount points' rmdir fails EBUSY
(looks like a refusal) while every file inside is deleted remotely. models list went
to 32.4 MiB. Nothing lost: blobs/ still holds the full 475 G (link count 1) and
nodes/gx10-dgx4/ holds all four deployment slices; recovery is re-running download
into the same store, which re-links from local blobs (running, pid 2037594). There
is no verify/repair subcommand, so download IS the repair path. Documented the
three false alarms that look like corruption: du dedups hardlinks across arguments
(why blobs read as 6.2M), NFS actimeo=600 dir caching, and cp -al EXDEV on NAS<->NAS.

R6 split: R6a done, R6b needs a completed TP4 stage (the window) to prove reads
traverse the bind and that FLARE never renames over or hardlinks into deployments/.
Serving untouched: dgx1/2/3/8 ranks=1, dgx1 api=200.
Alexander Korolev il y a 1 semaine
Parent
commit
3a9d1219b8

+ 3 - 3
specs/tech-architecture/.rounds-state.flare-storage-nas.json

@@ -1,12 +1,12 @@
 {
   "objective": "/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md",
-  "round": 4,
+  "round": 5,
   "maxRounds": 12,
   "turnsBudget": 60,
   "progressEntries": 12,
   "verifyTimeout": 120,
   "minutesBudget": null,
-  "lastObjectiveHash": "c3766b67-37083",
+  "lastObjectiveHash": "538560e0-44582",
   "attemptsBudget": 3,
   "attemptsUsed": 0,
   "lastCheck": null,
@@ -20,5 +20,5 @@
   "active": true,
   "finished": false,
   "lastError": null,
-  "updated": "2026-11-11T22:12:57.406Z"
+  "updated": "2026-11-12T00:18:07.696Z"
 }

+ 56 - 5
specs/tech-architecture/RALPH_PROGRESS.md

@@ -158,11 +158,6 @@ staging recipe; `references/troubleshooting.md` gained the acquisition section (
 hang, `--tensor-parallel 4`, publisher trust, token file, pkill self-match, rank-0-only
 API).
 
-## Loop log
-- round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…
-- round 2 · -- · 74 turns · continue → continue · (no ROUND line)
-- round 3 · -- · 36 turns · continue → continue · outcome
-
 ## 2026-11-11 — R4 (flare-storage-nas): v0.1.0 renamed the artefact; the CLI store is on the NAS
 
 **Goal quadrant: impact × effort — close R3's verification and do R4 (the config drift)
@@ -254,3 +249,59 @@ ranks' 261 G dirs are what is actually serving).
 Next: **R6 (Phase 3 gate) on dgx4** — now unblocked, with real paths in the doc; then
 the window (R7/R8) which is also what lands v0.1.0 as the *running* binary and frees
 the superseded 477 G.
+
+## Loop log
+- round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…
+- round 2 · -- · 74 turns · continue → continue · (no ROUND line)
+- round 3 · -- · 36 turns · continue → continue · outcome
+- round 4 · -- · 28 turns · continue → continue · outcome
+
+## 2026-11-12 — R5 round (flare-storage-nas): the bind-mount design works, and I broke a core while testing it
+
+**Round goal was R6 (Phase 3 gate) on idle dgx4. Half of it is now proven, and the plan
+gained its most important safety rule.**
+
+Proven on dgx4 with the staged v0.1.0 binary, using the real Phase 4 shape (per-node
+artefacts under `nodes/gx10-dgx4/<model>/<version>/deployments/<id>/` on the NAS,
+bind-mounted over the model directory's `deployments/` inside a store root):
+`flare run --stage-only` **passed `Verify core`** and reached `Wait for tensor-parallel
+hosts`, and the bind was byte-identical before and after five stage attempts (same inode,
+`findmnt` still showing the NAS source). The inventory layer is bind-safe too: `models
+list` over a locally-assembled root reported `474.66 GiB · 181 hardlinked objects` while
+`blobs/` and `weights/` were bind mounts onto a **different device** (NFS st_dev 80 vs
+local NVMe 66306). strace explains the symlink ban precisely: the escape check is
+`openat(7, "tp4", O_RDONLY|O_NOFOLLOW|O_DIRECTORY)`.
+
+**The plan's own Phase 3 procedure was invalid and is rewritten.** Copying
+`model.safetensors` + `deployments` out of a store and pointing `run`/`inspect` at the
+copy fails in v0.1.0 with `open core directory: No such file or directory (os error 2)` —
+the envelope resolves its core *within the store that contains it*, and a store has no
+metadata files at all (purely structural), so the model path must live inside a coherent
+store root with `blobs/ refs/ sources/ origins/ staging/ locks/` present. The verified
+command shape is in the doc.
+
+**Then I destroyed the core's materialisation.** Cleanup ran `rm -rf` over a tree that
+still had two live bind mounts into the NAS store. `rm` unmounts nothing: the `rmdir` of
+the mount points failed `EBUSY` — which reads like a refusal — while every file *inside*
+them was deleted on the NAS. `models list` went to `Disk used 32.4 MiB`. **No data was
+lost**: the store is content-addressed, so `blobs/` still held the whole 475 G (link
+count 1) and `nodes/gx10-dgx4/` still held all four deployment slices. Recovery is
+`flare download` into the same store, which re-links from local blobs; it is running now
+(pid 2037594, `~/flare-rematerialise.log`). There is **no** verify/repair subcommand —
+`download` *is* the repair path.
+
+Three false alarms that look like corruption, all documented so the next round does not
+chase them: `du -sh a b` **dedups hardlinks across arguments** (that is how `blobs` read
+as `6.2M` before I had actually broken anything); NFS `actimeo=600` dir caching made a
+live `rank-0` look missing while binds were stacked; and `cp -al` between two NAS paths
+fails `Invalid cross-device link` — per-node artefacts need `cp -a`.
+
+Rule added to the skill as hard rule 6 (unmount deepest-first before recursive deletes,
+`rm -rf --one-file-system`, then blob-first triage), plus a store-triage section in
+`references/troubleshooting.md` and a validated bind-layout section in
+`references/storage.md`. R6 is split: **R6a done**, **R6b** needs a *completed* TP4
+stage (peers ⇒ the window) to prove deployment reads traverse the bind and that FLARE
+never renames over / hardlinks into `deployments/` — either would take option D.
+
+Serving untouched all round: dgx1/2/3/8 `ranks=1`, dgx1 `api=200`. dgx4 has 1.5 T free
+and no ranks, and now holds the first real `nodes/<node>/…` artefacts (5.2 G).

+ 85 - 3
specs/tech-architecture/flare-storage-nas.md

@@ -54,7 +54,15 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       cmdline after the first restart (needs the window)
 - [ ] **R5 (Q1)** — Phase 0.1 as rewritten: pre-flight that the packaged release is
       the binary every node starts into at cutover (no PR-184 rebuild — dropped)
-- [ ] **R6 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4
+- [x] **R6a (Q1)** — Phase 3 bind-mount gate, part 1 on idle dgx4: v0.1.0 accepts the
+      layout, `--stage-only` passes **Verify core** through a NAS bind, mount survives
+      five stage attempts. Also found: the plan's Phase 3 procedure was **invalid**
+      (a standalone model-dir copy is rejected in v0.1.0) and there is a **data-loss
+      trap** in Phase 4 cleanup — see “Phase 3 gate results”. No data was lost.
+- [ ] **R6b (Q2)** — part 2 needs a *completed* TP4 stage (peers, so it belongs to the
+      window): do deployment reads really traverse the bind, and does FLARE ever
+      replace `deployments/` by rename or hardlink into it (both would break a
+      cross-device bind)? If either happens → **option D**.
 - [ ] **R7 (Q2)** — Phase 4 sparks rollout (window; ranks stop)
 - [ ] **R8 (Q2)** — Phase 5 prod4 rollout (window; ranks stop)
 - [ ] **R9 (Q3)** — Phase 6 verification matrix and rollback drill
@@ -655,6 +663,78 @@ Put the local copy at
 Gate: `flare inspect` shows the deployment, and `findmnt` still reports the bind
 mount after any stage operation.
 
+### Phase 3 gate results (2026-11-12, dgx4) — procedure corrected, one data-loss trap
+
+**What passed.** With the exact Phase 4 shape — per-node artefacts on the NAS under
+`nodes/gx10-dgx4/<model>/<version>/deployments/<id>/`, bind-mounted over the model
+directory's `deployments/` inside a store root — `flare run --stage-only` **passed
+`Verify core`** and reached `Wait for tensor-parallel hosts` (the furthest a lone node
+can get). The bind was unchanged before and after: same inode, same mount,
+`findmnt` still reporting the NAS source. v0.1.0's inventory layer is also bind-safe:
+`flare models list` over a locally-assembled root reported the full inventory
+(`474.66 GiB`, `181 hardlinked objects`) while `blobs/` and `weights/` were bind mounts
+onto a **different device** (st_dev 80 = NFS vs 66306 = local NVMe).
+
+**The plan's Phase 3 step 1–2 procedure was wrong.** Copying `model.safetensors` +
+`deployments` out of a store and pointing `run`/`inspect` at the copy **fails in v0.1.0**
+at `Verify core` with `open core directory: No such file or directory (os error 2)`: the
+envelope resolves its core **within the store that contains it**, so the model path must
+sit inside a coherent store root (the `blobs/ refs/ sources/ origins/ staging/ locks/`
+scaffolding alongside `models/`). The store has **no metadata files at all** — it is
+purely structural, which is why copying only the model dir loses the context. Working
+test shape (verified):
+
+```bash
+S=/mnt/nas/ai/flare/store; R2=$HOME/flare-bindtest          # any local store root
+mkdir -p $R2 && sudo mount --bind $S $R2                    # root scaffolding
+M=checkpoint-expanded-scales; V=gb10-tp4-202611112207
+D=$R2/models/$M/$V; DEP=db20209d361f473b8452f24ad517fa0d
+N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V              # per-node artefacts on NAS
+sudo mount --bind $N/deployments $D/deployments             # per-node slice on top
+flare run $D/model.safetensors --package-cache $R2 --rank 0 \
+      --rendezvous 127.0.0.1:29814 --gpus 0 --stage-only
+findmnt -no TARGET,SOURCE $D/deployments                    # must still print the bind
+```
+
+**Data-loss trap — this is the most important line in the document.** A `rm -rf` over a
+tree that still has **live bind mounts deletes through them**, into the NAS. That is how
+the freshly acquired core's `models/<m>/<v>/weights/` + `deployments/` materialisation
+was destroyed during test cleanup on dgx4: the `rmdir` of the two mount points failed
+with `EBUSY` (which looks like “it refused”), while every file *inside* them was deleted
+on the NAS. **Unmount first, always**, or make the refusal real:
+
+```bash
+findmnt -rno TARGET | grep -F "$D" | sort -r | xargs -r -n1 sudo umount   # deepest first
+sudo rm -rf --one-file-system "$D"        # belt and braces
+```
+
+Nothing was lost, because the store is content-addressed: `blobs/` still held the whole
+**475 G** (link count dropped to 1), and `nodes/gx10-dgx4/` still held all four
+deployment slices (5.2 G). Recovery is just re-running `flare download` into the same
+store — all blobs are local, so it re-materialises the hardlink farm instead of
+re-downloading. **Do not panic at `Disk used 32.4 MiB` in `models list`; check
+`du -sh <store>/blobs` first.**
+
+Three measurement gotchas that look like corruption and are not:
+
+- `du -sh a b c` **deduplicates hardlinks across its arguments** — measured
+  `du -sh $S/models $S/blobs` as `475G` + `6.2M`, i.e. “the blobs vanished”, when in
+  fact each directory alone is 475 G. Measure one directory per call.
+- NFS `actimeo=600` dir caching makes a live artefact look **missing** while test binds
+  are stacked (`cp: cannot stat …/rank-0: No such file or directory`); the file was
+  there the whole time. Unmount the binds before debugging.
+- `cp -al` between two NAS paths failed with **`Invalid cross-device link`** even though
+  both were under `/mnt/nas`. Use `cp -a` for per-node artefacts (1.3 GiB, cheap).
+
+**What is still unproven (R6b).** strace shows FLARE opens with dirfd-relative paths —
+`openat(7, "tp4", O_RDONLY|O_NOFOLLOW|O_DIRECTORY)` — so grepping strace for full paths
+**undercounts reads**; the positive test that a deployment file was read *through* the
+bind needs a completed stage, which needs the other three ranks. The `O_NOFOLLOW` in
+that same line is the escape check, and it is why **symlinks remain banned** here. If a
+completed stage turns out to rename over `deployments/` or to hardlink into it
+(cross-device hardlinks fail with EXDEV, as just demonstrated on NFS), take **option D**
+without arguing: leave `deployments/` local (1.3 GiB) and move only the package store.
+
 ### Phase 4 — roll out to sparks (window; ranks stop)
 
 Repeat per node in rank order, one node at a time, verifying before moving on.
@@ -671,8 +751,10 @@ Repeat per node in rank order, one node at a time, verifying before moving on.
    N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V
    sudo mkdir -p "$N"
    rsync -aH --remove-source-files "$D/model.safetensors" "$N/"
-   rsync -aH "$D/deployments/" "$N/deployments/"
-   rm -rf "$D/deployments"
+   rsync -aH "$D/deployments/" "$N/deployments/"     # cp -a/rsync, never cp -al: NAS↔NAS hardlinks can EXDEV
+   # NOTHING here may run while a bind is mounted at the target — rm deletes through it:
+   findmnt -rno TARGET | grep -F "$D" | sort -r | xargs -r -n1 sudo umount
+   rm -rf --one-file-system "$D/deployments"
    sudo mount --bind "$N/deployments" "$D/deployments"
    ```
    Persist it in `/etc/fstab` (bind mounts are ordinary fstab entries and are

+ 1 - 0
specs/tech-architecture/rounds-history.flare-storage-nas.jsonl

@@ -1,3 +1,4 @@
 {"timestamp":"2026-11-11T21:49:43.957Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":1,"goal":"Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every powered node (quadrant Q1)","quadrant":null,"turns":27,"marker":"continue","decision":"continue","reason":null,"check":null}
 {"timestamp":"2026-11-11T22:03:33.500Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":2,"goal":null,"quadrant":null,"turns":74,"marker":"continue","decision":"continue","reason":null,"check":null}
 {"timestamp":"2026-11-11T22:12:57.404Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":3,"goal":"outcome","quadrant":null,"turns":36,"marker":"continue","decision":"continue","reason":null,"check":null}
+{"timestamp":"2026-11-12T00:18:07.693Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":4,"goal":"outcome","quadrant":null,"turns":28,"marker":"continue","decision":"continue","reason":null,"check":null}