Browse Source

docs(flare-storage): store recovered - re-linked from local blobs for 34 MB of network; version id is mutable, deployment id is not

Repair finished at 1251 s: network 33,966,105 B, local source 509,653,222,270 B,
verification 509,646,855,013 B. Store is back to 1 model · 1 version, [DEFAULT]
gb10-tp4-202611120031, 474.66 GiB, 181 hardlinked objects, deployment
db20209d361f473b8452f24ad517fa0d TP 4 ready. Every command in the plan now carries
the new version id; the deployment id is content-derived and never changed, which is
the durable lesson: scripts must query the version id and may pin the deployment id.
Per-node artefacts rebuilt for the new version (5.1 G).flare models remove takes
MODEL@VERSION, not MODEL/VERSION. Serving never disturbed: ranks=1 on all four,
dgx1 api=200.
Alexander Korolev 1 week ago
parent
commit
b4a80f084d

+ 12 - 0
specs/tech-architecture/RALPH_PROGRESS.md

@@ -348,3 +348,15 @@ Repair of my own damage is still running healthily on dgx4 (pid 2037594, `read_b
 +370 MB/s against a 475 G blob re-verification, NIC idle — the progress bar counts
 network bytes only, which is why it reads `32.4 MiB` the whole time). When it lands,
 `models list` returns to `474.66 GiB`.
+
+**Recovery confirmed inside the same round.** The repair finished at 1251 s:
+`network 33,966,105 bytes · local source 509,653,222,270 · verification 509,646,855,013`
+— i.e. **34 MB re-downloaded, 509.6 GB re-read from local blobs**. The store now shows
+`1 model · 1 version`, `[DEFAULT] gb10-tp4-202611120031`, `474.66 GiB`,
+`181 hardlinked objects`, `deployment db20209d361f473b8452f24ad517fa0d · TP 4 · ready`.
+Two durable facts: **version ids are mutable** (a re-materialisation mints a new one, so
+scripts must query rather than hardcode) and **deployment ids are content-derived and
+stable** (every reference to `db20209d…` in the plan stayed valid). `flare models remove`
+wants `MODEL@VERSION`, not `MODEL/VERSION`. Per-node artefacts were rebuilt for the new
+version (5.1 G). NAS at 1.6 T/33 T. Serving untouched throughout: dgx1/2/3/8 ranks=1,
+dgx1 api=200.

+ 16 - 6
specs/tech-architecture/flare-storage-nas.md

@@ -27,7 +27,7 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       **PR #184 rebuild dropped** (§6 item 5). Note `~/.local/lib/flare-agent/flare` is
       **not dpkg-owned** — apt never updates what the unit actually runs
 - [x] **Current `gb10` core acquired onto the NAS store and verified** —
-      `checkpoint-expanded-scales/gb10-tp4-202611112207`, 474.66 GiB, **TP 4 · ready**,
+      `checkpoint-expanded-scales/gb10-tp4-202611120031`, 474.66 GiB, **TP 4 · ready**,
       signatures verified, visible from dgx1/dgx4 over NFS (Phase 2R)
 - [ ] That core **deployed to the ranks, and the superseded build deleted**
       (needs the window; dgx1's 477 G and the ranks' 261 G dirs are still the live data)
@@ -43,7 +43,7 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       **783 GiB freed** on dgx1/2/3, all ranks stayed up; also identified that every
       local package is a superseded build from a deleted repo (Phase 2R)
 - [x] **R3 (Q1)** — current `gb10` core **acquired and verified on the NAS store**:
-      `models/checkpoint-expanded-scales/gb10-tp4-202611112207`, 474.66 GiB,
+      `models/checkpoint-expanded-scales/gb10-tp4-202611120031`, 474.66 GiB,
       `payload v2`, deployment `db20209d361f473b8452f24ad517fa0d` **TP 4 · ready**,
       474.63 GiB in **181 hardlinked objects**, signatures verified by the publisher,
       `flare models list` correct from **dgx1 (prod) and dgx4**. 2 h 8 min, avg
@@ -570,10 +570,20 @@ the superseded 477 G on dgx1 and the 261 G dirs everywhere, start on the new pac
 bind-mount work on the old version dirs would be wasted: the paths contain the version
 name, and every fstab entry would have to be rewritten.
 
+> **Version ids are mutable; deployment ids are not.** The core was first acquired as
+> `gb10-tp4-202611112207`. That version's *materialisation* was destroyed by a recursive
+> delete through a live bind (see “Phase 3 gate results”), and re-running `download`
+> re-linked it from local blobs as **`gb10-tp4-202611120031`** — which is the id used by
+> every command in this document now. The deployment id
+> **`db20209d361f473b8452f24ad517fa0d` (TP 4 · ready)** is content-derived and **did not
+> change**, as did the inventory (`474.66 GiB`, `181 hardlinked objects`). Recovery cost:
+> **34 MB of network**, the remaining 509.6 GB re-read from local blobs, 1251 s.
+> Current state: `1 model · 1 version`, `[DEFAULT]` = `gb10-tp4-202611120031`.
+
 #### Acquisition result (2026-11-12 00:16 UTC) — measured, use these numbers
 
 ```
-store/…/models/checkpoint-expanded-scales/gb10-tp4-202611112207/
+store/…/models/checkpoint-expanded-scales/gb10-tp4-202611120031/
   model.safetensors   33 MiB   mode 0444 (read-only)
   weights/            470 G
   deployments/db20209d361f473b8452f24ad517fa0d/
@@ -679,11 +689,11 @@ Do this **before** touching dgx1–dgx3.
 Use **dgx4**: it is idle (no ranks), and now boots from the new MP600 Mini.
 
 The paths for this gate now exist — do not invent them. Source (NAS):
-`/mnt/nas/ai/flare/store/models/checkpoint-expanded-scales/gb10-tp4-202611112207/`
+`/mnt/nas/ai/flare/store/models/checkpoint-expanded-scales/gb10-tp4-202611120031/`
 with `model.safetensors` (0444), `weights/` (470 G) and
 `deployments/db20209d361f473b8452f24ad517fa0d/rank-<N>/` (3 files, 1.3 G each).
 Put the local copy at
-`~/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-202611112207/`
+`~/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-202611120031/`
 (v0.1.0's default root is `~/.cache/flare/cores`, but this fleet pins
 `paths.package_cache`, so use whatever that key says on the node under test).
 
@@ -741,7 +751,7 @@ test shape (verified):
 ```bash
 S=/mnt/nas/ai/flare/store; R2=$HOME/flare-bindtest          # any local store root
 mkdir -p $R2 && sudo mount --bind $S $R2                    # root scaffolding
-M=checkpoint-expanded-scales; V=gb10-tp4-202611112207
+M=checkpoint-expanded-scales; V=gb10-tp4-202611120031
 D=$R2/models/$M/$V; DEP=db20209d361f473b8452f24ad517fa0d
 N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V              # per-node artefacts on NAS
 sudo mount --bind $N/deployments $D/deployments             # per-node slice on top