Prechádzať zdrojové kódy

docs(flare-storage): purge stale data (783 GiB freed); replace, don't migrate

R2. The user was right that the local data is stale, and it is worse than stale:
the acquired package's source repo (...DeepSeek-V4.1-Flash-1017-FLARE2) answers
404 with a valid token, and the current source is the same publisher without the
"2", revision = branch gb10, last commit 2026-11-05 "Publish the GB10 TP4 package
with vocabulary-parallel sampling and an FP8 Markov head" (187 files, 474.7 GiB).
It requires flare-inference >= 0.1.0~git20261105002735.0009b89d-1 - which every
node has installed, but the *running* PR-184 binaries predate it, so the Nov-5
based rebuild is now a functional prerequisite, not a tuning nicety.

Phase 2 (rsync the old store to the NAS) is marked SUPERSEDED, do not run: the
operator rejected archiving a dead build, and the started rsync (105 GiB in) was
killed and its partial copy deleted.

Purge instead, with fd/maps/lsof+D verified empty on each node first: the
orphaned Nov-2 build checkpoint-expanded-scales/gb10-tp4-202611021936 was 261 GiB
on each of dgx1/2/3 -> 783 GiB freed. dgx1 went from 98% full (22 G free of a
915 GiB root - it is not a 1.8 TiB disk like dgx4) to 283 G free. All rank PIDs
unchanged (7-day uptime), API still answering.

Also documented: dgx1's package_cache had drifted to an empty ~/flare-pkgs/gb10
(CLI sees "no local model versions" while the rank uses ~/.cache/flare) - the
config redirection works but already diverged CLI from agent on one node; the
package format here is flat (0 hardlinks / 184 files) so rsync -H was never
load-bearing; HF token is export HF_TOKEN in ~/.zshenv on the nodes, not on the
mac; and the upstream gb10 launch recipe for when we cut over.
Alexander Korolev 1 týždeň pred
rodič
commit
cc2e7d5dbf

+ 34 - 0
specs/tech-architecture/.rounds-state.flare-storage-nas.json

@@ -0,0 +1,34 @@
+{
+  "objective": "/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md",
+  "round": 2,
+  "maxRounds": 12,
+  "turnsBudget": 60,
+  "progressEntries": 12,
+  "verifyTimeout": 120,
+  "minutesBudget": null,
+  "lastObjectiveHash": "35158cc0-17743",
+  "attemptsBudget": 3,
+  "attemptsUsed": 1,
+  "lastCheck": null,
+  "todoWatch": [
+    {
+      "id": 9,
+      "subject": "Compile table + conclusion",
+      "firstSeen": 1
+    },
+    {
+      "id": 11,
+      "subject": "Rebuild flare with PR184 on the Nov-5 base before restart",
+      "firstSeen": 1
+    },
+    {
+      "id": 12,
+      "subject": "Migrate flare model root to NAS in a maintenance window",
+      "firstSeen": 1
+    }
+  ],
+  "active": true,
+  "finished": false,
+  "lastError": null,
+  "updated": "2026-11-11T21:53:43.180Z"
+}

+ 3 - 0
specs/tech-architecture/RALPH_PROGRESS.md

@@ -46,3 +46,6 @@ Watch out for:
 - dgx5/6/7 are off; they still need Phase 1 when they come up (the `## Checks`
   command reports them as `off`).
 - R4 (PR #184 rebuild) needs dgx5 powered on and must precede any agent restart.
+
+## Loop log
+- round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…

+ 121 - 13
specs/tech-architecture/flare-storage-nas.md

@@ -14,8 +14,12 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       that works there (4.0) — see the correction in Phase 1
 - [x] Pre-migration state of dgx1/2/3/4/8 captured locally and on the NAS
 - [x] NAS throughput measured (was open item 6.3)
-- [ ] PR #184 binary rebuilt and staged (Phase 0.1) — needs dgx5 powered on
-- [ ] Package store (477 GiB) on the NAS, local copy removed, config pointed at it (Phase 2)
+- [x] **Stale model data purged from the nodes — 783 GiB freed**, dgx1 from 98 % full
+      (22 G free) to 283 G free; serving never interrupted
+- [ ] PR #184 binary rebuilt on the Nov-5 base and staged (Phase 0.1) — needs dgx5
+      powered on, and is now **required** to load the current package at all
+- [ ] **Fresh `gb10` package acquired onto the NAS store and deployed; superseded
+      build deleted** (Phase 2R — replaces the "migrate the old store" plan)
 - [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
 - [ ] Sparks rolled out (Phase 4, needs a window)
 - [ ] prod4 rolled out (Phase 5, needs a window)
@@ -24,15 +28,20 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
 ## Round plan
 
 - [x] **R1 (Q1)** — Phase 0.2–0.4 + Phase 1 on dgx1/2/3/4/8
-- [ ] **R2 (Q1)** — Phase 2 first half: `rsync -aH` the 477 GiB store to the NAS in
-      the background, then verify with `flare models list --model-cache`
-- [ ] **R3 (Q1)** — Phase 2 second half: point `[paths] package_cache` at the NAS,
-      remove the local store, confirm the launched rank cmdline (open item 6.1)
-- [ ] **R4 (Q1)** — PR #184 rebuild and staging (needs dgx5 on; prerequisite for any restart)
-- [ ] **R5 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4
-- [ ] **R6 (Q2)** — Phase 4 sparks rollout (window; ranks stop)
-- [ ] **R7 (Q2)** — Phase 5 prod4 rollout (window; ranks stop)
-- [ ] **R8 (Q3)** — Phase 6 verification matrix and rollback drill
+- [x] **R2 (Q1)** — stale-data purge instead of migration (operator decision):
+      **783 GiB freed** on dgx1/2/3, all ranks stayed up; also identified that every
+      local package is a superseded build from a deleted repo (Phase 2R)
+- [ ] **R3 (Q1)** — Phase 2R first half: download the current `gb10` package
+      (474.7 GiB) into the NAS store with `flare download … --revision gb10`,
+      verify with `flare models list --model-cache /mnt/nas/ai/flare/store`
+- [ ] **R4 (Q1)** — Phase 2R second half: fix dgx1's `package_cache` drift (it points
+      at an empty `~/flare-pkgs/gb10`), confirm the launched rank cmdline (open item 6.1)
+- [ ] **R5 (Q1)** — PR #184 rebuild **on the Nov-5 base** and staging (needs dgx5 on;
+      prerequisite for both any restart and loading the new package)
+- [ ] **R6 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4
+- [ ] **R7 (Q2)** — Phase 4 sparks rollout (window; ranks stop)
+- [ ] **R8 (Q2)** — Phase 5 prod4 rollout (window; ranks stop)
+- [ ] **R9 (Q3)** — Phase 6 verification matrix and rollback drill
 
 ## Checks
 
@@ -218,9 +227,43 @@ dgx1/2/3/4/8 (2026-11-11):** live options show `timeo=100`, `mount -a` returns 0
 without remounting, all rank PIDs kept their 7-day uptime and the API on
 `dgx1:30000` still answered. dgx5–7 are off and still need this.
 
-### Phase 2 — package store (the 477 GiB) to NAS
+### Phase 2 — package store (the 477 GiB) to NAS — **SUPERSEDED, do not run**
 
-Low risk, no bind mounts, no FLARE code path surprises.
+> **Superseded on 2026-11-11 (R2).** Two independent reasons:
+> 1. The operator rejected it: *“Syncing those old flare cache to store does not make
+>    much sense honestly.”* Copying a superseded build onto the NAS is not migration,
+>    it is archiving garbage.
+> 2. It is superseded by fact: the local package is an **out-of-date build whose source
+>    repo no longer exists** (see Phase 2R). The rsync was started, reached 105 GiB,
+>    was killed, and the partial copy was deleted.
+
+What replaced it: **purge the stale data** (done, below) and **acquire the current
+revision onto the NAS store** (Phase 2R).
+
+#### Purge performed on 2026-11-11
+
+Every node held the **Nov-2 build** `checkpoint-expanded-scales/gb10-tp4-202611021936`
+(261 GiB each) with nothing holding it open — verified three ways per node before
+deleting (`/proc/<rank>/fd`, `/proc/<rank>/maps`, and `lsof +D`):
+
+| node | before | after | freed |
+|---|---|---|---|
+| dgx1 | 848 G used, **22 G free (98 %)** | 587 G used, **283 G free** | 261 G |
+| dgx2 | 664 G used, 206 G free | 403 G used, 467 G free | 261 G |
+| dgx3 | 661 G used, 209 G free | 400 G used, 470 G free | 261 G |
+
+**Total 783 GiB freed.** Rank PIDs were unchanged (7-day uptime) and `dgx1:30000/v1/models`
+kept answering. Note dgx1's root is **915 GiB**, not 1.8 TB like dgx4 — the 261 GiB
+was 28 % of that disk.
+
+Still present and **not deletable while serving**:
+- dgx1 `~/.cache/flare/packages/7cf40bec…-35842073` — 477 G, and rank 0 has its
+  `weights/engram-*.safetensors` (92 G each) open. Space returns only when the rank
+  exits, so this one dies in the Phase 5 window, not before.
+- dgx2/dgx3 `models/…202611032142` and dgx8 `models/…202611022100` (261 G each) — the
+  live ranks read these.
+
+For reference, the original (not executed) procedure was:
 
 ```bash
 sudo mkdir -p /mnt/nas/ai/flare/store
@@ -260,8 +303,73 @@ If that path needs to change, it is a **launcher** change
 config, `src/native/flare_orbit.rs::extra_flags_for`) — never a hand edit on a
 node.
 
+### Phase 2R — replace, don't migrate (2026-11-11)
+
+**What we are actually serving is superseded, and its source is gone.** The acquired
+package's manifest (read out of the `flare.manifest` tensor inside `model.safetensors`
+— it is protobuf, not JSON, but the strings are legible) names:
+
+```
+source   https://huggingface.co/local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE2
+model    checkpoint-expanded-scales   identity gb10   publisher local-inference-lab
+```
+
+That repo answers **404 with a valid token** (401 unauthenticated is HF masking a
+missing repo). The current source is the same publisher **without the `2`**, and
+`gb10` is a *branch* of it:
+
+| | |
+|---|---|
+| repo | `local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE` (public, no token needed) |
+| revision | branch `gb10` — last commit **2026-11-05T01:42:45** *“Publish the GB10 TP4 package with vocabulary-parallel sampling and an FP8 Markov head”* |
+| size | **187 files, 474.7 GiB** (we hold 184 files / 477 GiB from a Nov-3 pull) |
+| requires | `flare-inference` **`0.1.0~git20261105002735.0009b89d-1` or newer** — “earlier versions cannot load it” |
+
+That version is exactly what the fleet has *installed* on every node. The binaries
+*running* are the older PR-184 inodes, so **no node can load the new package until
+the Nov-5-based rebuild (R5) lands** — the same conclusion #11 reached, now with a
+functional reason instead of a performance one.
+
+Download into the NAS store (this is what the store is *for* — fresh acquisitions,
+not old caches). The `hf` token, when one is needed, is `export HF_TOKEN=…` in
+`~/.zshenv` **on the nodes** — there is no token on the mac:
+
+```bash
+package_path=$(flare download \
+  local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE --revision gb10)
+# to land it on the NAS store rather than the local cache, point the CLI at it:
+flare download local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE --revision gb10 \
+  --model-cache /mnt/nas/ai/flare/store
+flare models list --model-cache /mnt/nas/ai/flare/store     # must list the new version
+```
+
+The upstream launch recipe (`SPARK-DS4-GUIDE.md` / the `gb10` README) is worth
+aligning our launcher flags with — note it differs from what prod4 runs today:
+
+```bash
+flare orbit run "spark1:${package_path}" \
+  --cluster sparks --name deepseek1017 \
+  --max-concurrency 32 --mtp-draft-tokens 5 --mtp-adaptive-verification \
+  --mtp-adaptive-verification-alpha 0.75 \
+  --embedding-table-storage disk \
+  --embedding-table-resident-scales --embedding-table-cache-size 1GiB \
+  --host 0.0.0.0 --port 8000 --detach
+```
+
+(Differences from our running rank 0: it adds `--mtp-adaptive-verification-alpha 0.75`,
+uses port 8000 not 30000, and names the group. Orbit ships each peer only the data it
+needs — “You do not need four full downloads.”)
+
+Order of work: **R5 rebuild → download fresh to NAS (R3) → window: stop ranks, delete
+the superseded 477 G on dgx1 and the 261 G dirs everywhere, start on the new package
+→ then the bind-mount layout (Phase 3+) on top of the new version dirs.** Doing the
+bind-mount work on the old version dirs would be wasted: the paths contain the version
+name, and every fstab entry would have to be rewritten.
+
 ### Phase 3 — validate the bind-mount model on an idle node (gate)
 
+Do this **before** touching dgx1–dgx3.
+
 Use **dgx4**: it is idle (no ranks), and now boots from the new MP600 Mini.
 
 1. Copy one model's small artefacts from a running node:

+ 1 - 0
specs/tech-architecture/rounds-history.flare-storage-nas.jsonl

@@ -0,0 +1 @@
+{"timestamp":"2026-11-11T21:49:43.957Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":1,"goal":"Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every powered node (quadrant Q1)","quadrant":null,"turns":27,"marker":"continue","decision":"continue","reason":null,"check":null}