|
|
@@ -54,7 +54,15 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
cmdline after the first restart (needs the window)
|
|
|
- [ ] **R5 (Q1)** — Phase 0.1 as rewritten: pre-flight that the packaged release is
|
|
|
the binary every node starts into at cutover (no PR-184 rebuild — dropped)
|
|
|
-- [ ] **R6 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4
|
|
|
+- [x] **R6a (Q1)** — Phase 3 bind-mount gate, part 1 on idle dgx4: v0.1.0 accepts the
|
|
|
+ layout, `--stage-only` passes **Verify core** through a NAS bind, mount survives
|
|
|
+ five stage attempts. Also found: the plan's Phase 3 procedure was **invalid**
|
|
|
+ (a standalone model-dir copy is rejected in v0.1.0) and there is a **data-loss
|
|
|
+ trap** in Phase 4 cleanup — see “Phase 3 gate results”. No data was lost.
|
|
|
+- [ ] **R6b (Q2)** — part 2 needs a *completed* TP4 stage (peers, so it belongs to the
|
|
|
+ window): do deployment reads really traverse the bind, and does FLARE ever
|
|
|
+ replace `deployments/` by rename or hardlink into it (both would break a
|
|
|
+ cross-device bind)? If either happens → **option D**.
|
|
|
- [ ] **R7 (Q2)** — Phase 4 sparks rollout (window; ranks stop)
|
|
|
- [ ] **R8 (Q2)** — Phase 5 prod4 rollout (window; ranks stop)
|
|
|
- [ ] **R9 (Q3)** — Phase 6 verification matrix and rollback drill
|
|
|
@@ -655,6 +663,78 @@ Put the local copy at
|
|
|
Gate: `flare inspect` shows the deployment, and `findmnt` still reports the bind
|
|
|
mount after any stage operation.
|
|
|
|
|
|
+### Phase 3 gate results (2026-11-12, dgx4) — procedure corrected, one data-loss trap
|
|
|
+
|
|
|
+**What passed.** With the exact Phase 4 shape — per-node artefacts on the NAS under
|
|
|
+`nodes/gx10-dgx4/<model>/<version>/deployments/<id>/`, bind-mounted over the model
|
|
|
+directory's `deployments/` inside a store root — `flare run --stage-only` **passed
|
|
|
+`Verify core`** and reached `Wait for tensor-parallel hosts` (the furthest a lone node
|
|
|
+can get). The bind was unchanged before and after: same inode, same mount,
|
|
|
+`findmnt` still reporting the NAS source. v0.1.0's inventory layer is also bind-safe:
|
|
|
+`flare models list` over a locally-assembled root reported the full inventory
|
|
|
+(`474.66 GiB`, `181 hardlinked objects`) while `blobs/` and `weights/` were bind mounts
|
|
|
+onto a **different device** (st_dev 80 = NFS vs 66306 = local NVMe).
|
|
|
+
|
|
|
+**The plan's Phase 3 step 1–2 procedure was wrong.** Copying `model.safetensors` +
|
|
|
+`deployments` out of a store and pointing `run`/`inspect` at the copy **fails in v0.1.0**
|
|
|
+at `Verify core` with `open core directory: No such file or directory (os error 2)`: the
|
|
|
+envelope resolves its core **within the store that contains it**, so the model path must
|
|
|
+sit inside a coherent store root (the `blobs/ refs/ sources/ origins/ staging/ locks/`
|
|
|
+scaffolding alongside `models/`). The store has **no metadata files at all** — it is
|
|
|
+purely structural, which is why copying only the model dir loses the context. Working
|
|
|
+test shape (verified):
|
|
|
+
|
|
|
+```bash
|
|
|
+S=/mnt/nas/ai/flare/store; R2=$HOME/flare-bindtest # any local store root
|
|
|
+mkdir -p $R2 && sudo mount --bind $S $R2 # root scaffolding
|
|
|
+M=checkpoint-expanded-scales; V=gb10-tp4-202611112207
|
|
|
+D=$R2/models/$M/$V; DEP=db20209d361f473b8452f24ad517fa0d
|
|
|
+N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V # per-node artefacts on NAS
|
|
|
+sudo mount --bind $N/deployments $D/deployments # per-node slice on top
|
|
|
+flare run $D/model.safetensors --package-cache $R2 --rank 0 \
|
|
|
+ --rendezvous 127.0.0.1:29814 --gpus 0 --stage-only
|
|
|
+findmnt -no TARGET,SOURCE $D/deployments # must still print the bind
|
|
|
+```
|
|
|
+
|
|
|
+**Data-loss trap — this is the most important line in the document.** A `rm -rf` over a
|
|
|
+tree that still has **live bind mounts deletes through them**, into the NAS. That is how
|
|
|
+the freshly acquired core's `models/<m>/<v>/weights/` + `deployments/` materialisation
|
|
|
+was destroyed during test cleanup on dgx4: the `rmdir` of the two mount points failed
|
|
|
+with `EBUSY` (which looks like “it refused”), while every file *inside* them was deleted
|
|
|
+on the NAS. **Unmount first, always**, or make the refusal real:
|
|
|
+
|
|
|
+```bash
|
|
|
+findmnt -rno TARGET | grep -F "$D" | sort -r | xargs -r -n1 sudo umount # deepest first
|
|
|
+sudo rm -rf --one-file-system "$D" # belt and braces
|
|
|
+```
|
|
|
+
|
|
|
+Nothing was lost, because the store is content-addressed: `blobs/` still held the whole
|
|
|
+**475 G** (link count dropped to 1), and `nodes/gx10-dgx4/` still held all four
|
|
|
+deployment slices (5.2 G). Recovery is just re-running `flare download` into the same
|
|
|
+store — all blobs are local, so it re-materialises the hardlink farm instead of
|
|
|
+re-downloading. **Do not panic at `Disk used 32.4 MiB` in `models list`; check
|
|
|
+`du -sh <store>/blobs` first.**
|
|
|
+
|
|
|
+Three measurement gotchas that look like corruption and are not:
|
|
|
+
|
|
|
+- `du -sh a b c` **deduplicates hardlinks across its arguments** — measured
|
|
|
+ `du -sh $S/models $S/blobs` as `475G` + `6.2M`, i.e. “the blobs vanished”, when in
|
|
|
+ fact each directory alone is 475 G. Measure one directory per call.
|
|
|
+- NFS `actimeo=600` dir caching makes a live artefact look **missing** while test binds
|
|
|
+ are stacked (`cp: cannot stat …/rank-0: No such file or directory`); the file was
|
|
|
+ there the whole time. Unmount the binds before debugging.
|
|
|
+- `cp -al` between two NAS paths failed with **`Invalid cross-device link`** even though
|
|
|
+ both were under `/mnt/nas`. Use `cp -a` for per-node artefacts (1.3 GiB, cheap).
|
|
|
+
|
|
|
+**What is still unproven (R6b).** strace shows FLARE opens with dirfd-relative paths —
|
|
|
+`openat(7, "tp4", O_RDONLY|O_NOFOLLOW|O_DIRECTORY)` — so grepping strace for full paths
|
|
|
+**undercounts reads**; the positive test that a deployment file was read *through* the
|
|
|
+bind needs a completed stage, which needs the other three ranks. The `O_NOFOLLOW` in
|
|
|
+that same line is the escape check, and it is why **symlinks remain banned** here. If a
|
|
|
+completed stage turns out to rename over `deployments/` or to hardlink into it
|
|
|
+(cross-device hardlinks fail with EXDEV, as just demonstrated on NFS), take **option D**
|
|
|
+without arguing: leave `deployments/` local (1.3 GiB) and move only the package store.
|
|
|
+
|
|
|
### Phase 4 — roll out to sparks (window; ranks stop)
|
|
|
|
|
|
Repeat per node in rank order, one node at a time, verifying before moving on.
|
|
|
@@ -671,8 +751,10 @@ Repeat per node in rank order, one node at a time, verifying before moving on.
|
|
|
N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V
|
|
|
sudo mkdir -p "$N"
|
|
|
rsync -aH --remove-source-files "$D/model.safetensors" "$N/"
|
|
|
- rsync -aH "$D/deployments/" "$N/deployments/"
|
|
|
- rm -rf "$D/deployments"
|
|
|
+ rsync -aH "$D/deployments/" "$N/deployments/" # cp -a/rsync, never cp -al: NAS↔NAS hardlinks can EXDEV
|
|
|
+ # NOTHING here may run while a bind is mounted at the target — rm deletes through it:
|
|
|
+ findmnt -rno TARGET | grep -F "$D" | sort -r | xargs -r -n1 sudo umount
|
|
|
+ rm -rf --one-file-system "$D/deployments"
|
|
|
sudo mount --bind "$N/deployments" "$D/deployments"
|
|
|
```
|
|
|
Persist it in `/etc/fstab` (bind mounts are ordinary fstab entries and are
|