|
|
@@ -27,7 +27,7 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
**PR #184 rebuild dropped** (§6 item 5). Note `~/.local/lib/flare-agent/flare` is
|
|
|
**not dpkg-owned** — apt never updates what the unit actually runs
|
|
|
- [x] **Current `gb10` core acquired onto the NAS store and verified** —
|
|
|
- `checkpoint-expanded-scales/gb10-tp4-202611112207`, 474.66 GiB, **TP 4 · ready**,
|
|
|
+ `checkpoint-expanded-scales/gb10-tp4-202611120031`, 474.66 GiB, **TP 4 · ready**,
|
|
|
signatures verified, visible from dgx1/dgx4 over NFS (Phase 2R)
|
|
|
- [ ] That core **deployed to the ranks, and the superseded build deleted**
|
|
|
(needs the window; dgx1's 477 G and the ranks' 261 G dirs are still the live data)
|
|
|
@@ -43,7 +43,7 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
**783 GiB freed** on dgx1/2/3, all ranks stayed up; also identified that every
|
|
|
local package is a superseded build from a deleted repo (Phase 2R)
|
|
|
- [x] **R3 (Q1)** — current `gb10` core **acquired and verified on the NAS store**:
|
|
|
- `models/checkpoint-expanded-scales/gb10-tp4-202611112207`, 474.66 GiB,
|
|
|
+ `models/checkpoint-expanded-scales/gb10-tp4-202611120031`, 474.66 GiB,
|
|
|
`payload v2`, deployment `db20209d361f473b8452f24ad517fa0d` **TP 4 · ready**,
|
|
|
474.63 GiB in **181 hardlinked objects**, signatures verified by the publisher,
|
|
|
`flare models list` correct from **dgx1 (prod) and dgx4**. 2 h 8 min, avg
|
|
|
@@ -570,10 +570,20 @@ the superseded 477 G on dgx1 and the 261 G dirs everywhere, start on the new pac
|
|
|
bind-mount work on the old version dirs would be wasted: the paths contain the version
|
|
|
name, and every fstab entry would have to be rewritten.
|
|
|
|
|
|
+> **Version ids are mutable; deployment ids are not.** The core was first acquired as
|
|
|
+> `gb10-tp4-202611112207`. That version's *materialisation* was destroyed by a recursive
|
|
|
+> delete through a live bind (see “Phase 3 gate results”), and re-running `download`
|
|
|
+> re-linked it from local blobs as **`gb10-tp4-202611120031`** — which is the id used by
|
|
|
+> every command in this document now. The deployment id
|
|
|
+> **`db20209d361f473b8452f24ad517fa0d` (TP 4 · ready)** is content-derived and **did not
|
|
|
+> change**, as did the inventory (`474.66 GiB`, `181 hardlinked objects`). Recovery cost:
|
|
|
+> **34 MB of network**, the remaining 509.6 GB re-read from local blobs, 1251 s.
|
|
|
+> Current state: `1 model · 1 version`, `[DEFAULT]` = `gb10-tp4-202611120031`.
|
|
|
+
|
|
|
#### Acquisition result (2026-11-12 00:16 UTC) — measured, use these numbers
|
|
|
|
|
|
```
|
|
|
-store/…/models/checkpoint-expanded-scales/gb10-tp4-202611112207/
|
|
|
+store/…/models/checkpoint-expanded-scales/gb10-tp4-202611120031/
|
|
|
model.safetensors 33 MiB mode 0444 (read-only)
|
|
|
weights/ 470 G
|
|
|
deployments/db20209d361f473b8452f24ad517fa0d/
|
|
|
@@ -679,11 +689,11 @@ Do this **before** touching dgx1–dgx3.
|
|
|
Use **dgx4**: it is idle (no ranks), and now boots from the new MP600 Mini.
|
|
|
|
|
|
The paths for this gate now exist — do not invent them. Source (NAS):
|
|
|
-`/mnt/nas/ai/flare/store/models/checkpoint-expanded-scales/gb10-tp4-202611112207/`
|
|
|
+`/mnt/nas/ai/flare/store/models/checkpoint-expanded-scales/gb10-tp4-202611120031/`
|
|
|
with `model.safetensors` (0444), `weights/` (470 G) and
|
|
|
`deployments/db20209d361f473b8452f24ad517fa0d/rank-<N>/` (3 files, 1.3 G each).
|
|
|
Put the local copy at
|
|
|
-`~/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-202611112207/`
|
|
|
+`~/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-202611120031/`
|
|
|
(v0.1.0's default root is `~/.cache/flare/cores`, but this fleet pins
|
|
|
`paths.package_cache`, so use whatever that key says on the node under test).
|
|
|
|
|
|
@@ -741,7 +751,7 @@ test shape (verified):
|
|
|
```bash
|
|
|
S=/mnt/nas/ai/flare/store; R2=$HOME/flare-bindtest # any local store root
|
|
|
mkdir -p $R2 && sudo mount --bind $S $R2 # root scaffolding
|
|
|
-M=checkpoint-expanded-scales; V=gb10-tp4-202611112207
|
|
|
+M=checkpoint-expanded-scales; V=gb10-tp4-202611120031
|
|
|
D=$R2/models/$M/$V; DEP=db20209d361f473b8452f24ad517fa0d
|
|
|
N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V # per-node artefacts on NAS
|
|
|
sudo mount --bind $N/deployments $D/deployments # per-node slice on top
|