Bladeren bron

docs(flare-storage): gb10 core acquired and verified on the NAS (R3 gate passed)

Acquisition finished at 00:16 UTC after 7682 s (avg 63 MiB/s, peak 104):
checkpoint-expanded-scales/gb10-tp4-202611112207, payload v2, 474.66 GiB,
181 hardlinked objects, deployment db20209d361f473b8452f24ad517fa0d TP 4 ready,
publisher signatures verified. `flare models list` reads it from dgx4 and from dgx1
(prod rank 0) over NFS, so the store-on-NAS premise holds from a serving node.
models inspect: DeepSeek-V4.1-Flash, sm_121, ctx 1,048,576, prefill 8192,
concurrency 32 / 4 ranks, vocab 129,280, mtp widths 1-6.

Two plan numbers were wrong and are now measured. The store does NOT hold a 2x
copy: transport/ peaked at 475 G while blobs/ grew beside it, and at the end
transport/ is 88 K and blobs/ is the whole 475 G - no models gc needed, the 2x
warning is mid-run only. And a rank slice is THREE files (initial-state
1,321,056,820 B + kernels 39,429,126 B + graphs 4,207,295 B = 1.3 GiB each,
deployments/ 5.1 G total, weights/ 470 G, model.safetensors 33 M at mode 0444),
i.e. ~2 s per rank at our measured NAS read rates - the small-file worry in 6.3
does not apply, so the bind-mount phase is viable on measured numbers.

Milestones split honestly: acquisition done; deploying the core to the ranks and
deleting the superseded build still needs the window (dgx1's 477 G and the ranks'
261 G dirs are the live data). Serving survived 2 h of NAS writes: all ranks up,
dgx1 api=200, mounts still timeo=100.
Alexander Korolev 1 week geleden
bovenliggende
commit
11d1a02211
2 gewijzigde bestanden met toevoegingen van 91 en 9 verwijderingen
  1. 38 0
      specs/tech-architecture/RALPH_PROGRESS.md
  2. 53 9
      specs/tech-architecture/flare-storage-nas.md

+ 38 - 0
specs/tech-architecture/RALPH_PROGRESS.md

@@ -216,3 +216,41 @@ backup paths, verification, why the old value existed), R4 ticked. Skill
 `references/storage.md`: same vocabulary table, the no-migration warning, the agent
 synthesises-the-flag note, the `models` subcommands, and the transport-first layout with
 its 2× peak.
+
+## 2026-11-12 — R4 close (flare-storage-nas): the current `gb10` core is on the NAS and verified
+
+**R3's gate passed.** The acquisition that started 22:07 UTC finished at **00:16 UTC**
+(7682 s, avg 63 MiB/s, peak 104) and `flare models list` now reports, from both dgx4
+and **dgx1 (prod)** over NFS:
+
+```
+checkpoint-expanded-scales  gb10-tp4-202611112207  [DEFAULT]
+Format payload v2 · Disk used 474.66 GiB · Shared files 474.63 GiB · 181 hardlinked objects
+Deployment db20209d361f473b8452f24ad517fa0d · TP 4 · ready
+Context / prefill 1,048,576 / 8,192 tokens · Pages / state rows 128 / 448
+```
+
+`models inspect` confirms identity: **DeepSeek-V4.1-Flash**, *local-inference-lab
+(signatures verified)*, **sm_121**, vocab 129,280, **concurrency 32 · 4 ranks**,
+speculation `mtp · widths 1–6`.
+
+**Two numbers in the plan were wrong and are now measured.** (1) The store does **not**
+keep a 2× copy: `transport/` peaked at 475 G *while* `blobs/` grew beside it, and at the
+end `transport/` is 88 K and `blobs/` is the whole 475 G — so no `models gc` was needed,
+and the 2× warning only applies mid-run. (2) A **rank slice is three files** —
+`initial-state.safetensors` 1,321,056,820 B + `kernels.safetensors` 39,429,126 B +
+`graphs.safetensors` 4,207,295 B = 1.3 GiB each, `deployments/` 5.1 G total,
+`weights/` 470 G, `model.safetensors` 33 M at **mode 0444**. At our measured NAS read
+rates that is ~2 s per rank, so the small-file worry in §6 item 3 simply does not
+apply to this package — the bind-mount phase is viable on the numbers, not by hope.
+
+Serving survived the 2 h of NAS writes untouched: dgx1/2/3/8 all `ranks=1`, dgx1
+`api=200`, uptime continuing, every mount still `timeo=100`.
+
+Milestones restructured honestly: acquisition **done**, deployment of the core +
+deletion of the superseded build still open (it needs the window — dgx1's 477 G and the
+ranks' 261 G dirs are what is actually serving).
+
+Next: **R6 (Phase 3 gate) on dgx4** — now unblocked, with real paths in the doc; then
+the window (R7/R8) which is also what lands v0.1.0 as the *running* binary and frees
+the superseded 477 G.

+ 53 - 9
specs/tech-architecture/flare-storage-nas.md

@@ -23,8 +23,11 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       `e3fe697e…` 26,054,848 B, same sha256 on all five), nothing in `/usr/bin` or the
       agent path touched yet. Remaining: dgx5–7 (off) + the temp+`mv` swap and one
       restart window; **PR #184 rebuild dropped** (§6 item 5)
-- [ ] **Fresh `gb10` package acquired onto the NAS store and deployed; superseded
-      build deleted** (Phase 2R — replaces the "migrate the old store" plan)
+- [x] **Current `gb10` core acquired onto the NAS store and verified** —
+      `checkpoint-expanded-scales/gb10-tp4-202611112207`, 474.66 GiB, **TP 4 · ready**,
+      signatures verified, visible from dgx1/dgx4 over NFS (Phase 2R)
+- [ ] That core **deployed to the ranks, and the superseded build deleted**
+      (needs the window; dgx1's 477 G and the ranks' 261 G dirs are still the live data)
 - [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
 - [ ] Sparks rolled out (Phase 4, needs a window)
 - [ ] prod4 rolled out (Phase 5, needs a window)
@@ -36,13 +39,13 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
 - [x] **R2 (Q1)** — stale-data purge instead of migration (operator decision):
       **783 GiB freed** on dgx1/2/3, all ranks stayed up; also identified that every
       local package is a superseded build from a deleted repo (Phase 2R)
-- [ ] **R3 (Q1)** — Phase 2R first half: download the current `gb10` package
-      (474.7 GiB) into the NAS store with `flare download … --revision gb10`,
-      verify with `flare models list --model-cache /mnt/nas/ai/flare/store`.
-      **In flight 2026-11-11 22:07 on dgx4**; the Nov-5 apt binary could not do it at
-      all (§ Phase 0.1), the tagged v0.1.0 binary does it at 55–78 MiB/s ⇒ ETA ≈ 2 h.
-      Also done in this round: v0.1.0 **staged** on dgx1/2/3/4/8, and SGLang (Ripple
-      eval server) **stopped on dgx4** — 103 GiB → 4 GiB RAM used
+- [x] **R3 (Q1)** — current `gb10` core **acquired and verified on the NAS store**:
+      `models/checkpoint-expanded-scales/gb10-tp4-202611112207`, 474.66 GiB,
+      `payload v2`, deployment `db20209d361f473b8452f24ad517fa0d` **TP 4 · ready**,
+      474.63 GiB in **181 hardlinked objects**, signatures verified by the publisher,
+      `flare models list` correct from **dgx1 (prod) and dgx4**. 2 h 8 min, avg
+      63 MiB/s (peak 104). Note the gate command is `flare models list` /
+      `--core-cache`, not `--model-cache` (see the vocabulary section)
 - [x] **R4 (Q1)** — `package_cache` now `/mnt/nas/ai/flare/store` on dgx4 + dgx1
       (backups kept; dgx1 rank untouched, `api=200`, same PID). Answered the config
       question: v0.1.0 still honours `paths.package_cache`, the launcher never passes
@@ -505,6 +508,38 @@ the superseded 477 G on dgx1 and the 261 G dirs everywhere, start on the new pac
 bind-mount work on the old version dirs would be wasted: the paths contain the version
 name, and every fstab entry would have to be rewritten.
 
+#### Acquisition result (2026-11-12 00:16 UTC) — measured, use these numbers
+
+```
+store/…/models/checkpoint-expanded-scales/gb10-tp4-202611112207/
+  model.safetensors   33 MiB   mode 0444 (read-only)
+  weights/            470 G
+  deployments/db20209d361f473b8452f24ad517fa0d/
+      rank-0 rank-1 rank-2 rank-3      1.3 G each, 5.1 G total
+```
+
+`flare models list`: `1 model · 1 version`, `Format payload v2`,
+`Disk used 474.66 GiB`, `Shared files 474.63 GiB · 181 hardlinked objects`,
+`Deployment db20209d361f473b8452f24ad517fa0d · TP 4 · ready`,
+`Context / prefill 1,048,576 / 8,192 tokens`, `Pages / state rows 128 / 448`.
+`flare models inspect checkpoint-expanded-scales`: name **DeepSeek-V4.1-Flash**,
+publisher *local-inference-lab (signatures verified)*, **sm_121**, vocab 129,280,
+**concurrency 32 · 4 ranks**, speculation `mtp · widths 1,2,3,4,5,6`.
+
+Acquisition accounting: `network 509,628,494,618 B`, `local source 509,653,222,270 B`,
+`verification 509,646,855,013 B`, `materialized 0 B`, **elapsed 7682 s (2 h 08 m)**,
+average 63 MiB/s (peak 104 MiB/s measured on `du`). **No 2× leftover**: the transport
+cache was folded into `blobs/` — it peaked at 475 G next to a growing `blobs/`, and at
+the end `transport/` is 88 K and `blobs/` is the whole 475 G. So the earlier 2×-peak
+warning applies to the *middle* of the run only, and `models gc` was not needed.
+
+**This changes the Phase 3 arithmetic, in our favour.** A rank slice is
+**three files**: `initial-state.safetensors` 1,321,056,820 B,
+`kernels.safetensors` 39,429,126 B, `graphs.safetensors` 4,207,295 B. At the measured
+NAS rates (660–820 MiB/s at 1 MiB reads) that is **~2 s per rank** — the small-file
+fear in §6 item 3 does not apply to this package. `model.safetensors` is mode 0444, so
+bind-mount it read-write and do not expect to replace it.
+
 ### v0.1.0 vocabulary and paths — read before running any command (2026-11-11, R4)
 
 The runtime we are moving to renamed the artefact. Measured against both binaries on
@@ -576,6 +611,15 @@ Do this **before** touching dgx1–dgx3.
 
 Use **dgx4**: it is idle (no ranks), and now boots from the new MP600 Mini.
 
+The paths for this gate now exist — do not invent them. Source (NAS):
+`/mnt/nas/ai/flare/store/models/checkpoint-expanded-scales/gb10-tp4-202611112207/`
+with `model.safetensors` (0444), `weights/` (470 G) and
+`deployments/db20209d361f473b8452f24ad517fa0d/rank-<N>/` (3 files, 1.3 G each).
+Put the local copy at
+`~/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-202611112207/`
+(v0.1.0's default root is `~/.cache/flare/cores`, but this fleet pins
+`paths.package_cache`, so use whatever that key says on the node under test).
+
 1. Copy one model's small artefacts from a running node:
    ```bash
    ssh dgx8.lan 'tar -C ~/.cache/flare/models/<model>/<version> -cf - model.safetensors deployments' \