|
|
@@ -23,8 +23,11 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
`e3fe697e…` 26,054,848 B, same sha256 on all five), nothing in `/usr/bin` or the
|
|
|
agent path touched yet. Remaining: dgx5–7 (off) + the temp+`mv` swap and one
|
|
|
restart window; **PR #184 rebuild dropped** (§6 item 5)
|
|
|
-- [ ] **Fresh `gb10` package acquired onto the NAS store and deployed; superseded
|
|
|
- build deleted** (Phase 2R — replaces the "migrate the old store" plan)
|
|
|
+- [x] **Current `gb10` core acquired onto the NAS store and verified** —
|
|
|
+ `checkpoint-expanded-scales/gb10-tp4-202611112207`, 474.66 GiB, **TP 4 · ready**,
|
|
|
+ signatures verified, visible from dgx1/dgx4 over NFS (Phase 2R)
|
|
|
+- [ ] That core **deployed to the ranks, and the superseded build deleted**
|
|
|
+ (needs the window; dgx1's 477 G and the ranks' 261 G dirs are still the live data)
|
|
|
- [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
|
|
|
- [ ] Sparks rolled out (Phase 4, needs a window)
|
|
|
- [ ] prod4 rolled out (Phase 5, needs a window)
|
|
|
@@ -36,13 +39,13 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
- [x] **R2 (Q1)** — stale-data purge instead of migration (operator decision):
|
|
|
**783 GiB freed** on dgx1/2/3, all ranks stayed up; also identified that every
|
|
|
local package is a superseded build from a deleted repo (Phase 2R)
|
|
|
-- [ ] **R3 (Q1)** — Phase 2R first half: download the current `gb10` package
|
|
|
- (474.7 GiB) into the NAS store with `flare download … --revision gb10`,
|
|
|
- verify with `flare models list --model-cache /mnt/nas/ai/flare/store`.
|
|
|
- **In flight 2026-11-11 22:07 on dgx4**; the Nov-5 apt binary could not do it at
|
|
|
- all (§ Phase 0.1), the tagged v0.1.0 binary does it at 55–78 MiB/s ⇒ ETA ≈ 2 h.
|
|
|
- Also done in this round: v0.1.0 **staged** on dgx1/2/3/4/8, and SGLang (Ripple
|
|
|
- eval server) **stopped on dgx4** — 103 GiB → 4 GiB RAM used
|
|
|
+- [x] **R3 (Q1)** — current `gb10` core **acquired and verified on the NAS store**:
|
|
|
+ `models/checkpoint-expanded-scales/gb10-tp4-202611112207`, 474.66 GiB,
|
|
|
+ `payload v2`, deployment `db20209d361f473b8452f24ad517fa0d` **TP 4 · ready**,
|
|
|
+ 474.63 GiB in **181 hardlinked objects**, signatures verified by the publisher,
|
|
|
+ `flare models list` correct from **dgx1 (prod) and dgx4**. 2 h 8 min, avg
|
|
|
+ 63 MiB/s (peak 104). Note the gate command is `flare models list` /
|
|
|
+ `--core-cache`, not `--model-cache` (see the vocabulary section)
|
|
|
- [x] **R4 (Q1)** — `package_cache` now `/mnt/nas/ai/flare/store` on dgx4 + dgx1
|
|
|
(backups kept; dgx1 rank untouched, `api=200`, same PID). Answered the config
|
|
|
question: v0.1.0 still honours `paths.package_cache`, the launcher never passes
|
|
|
@@ -505,6 +508,38 @@ the superseded 477 G on dgx1 and the 261 G dirs everywhere, start on the new pac
|
|
|
bind-mount work on the old version dirs would be wasted: the paths contain the version
|
|
|
name, and every fstab entry would have to be rewritten.
|
|
|
|
|
|
+#### Acquisition result (2026-11-12 00:16 UTC) — measured, use these numbers
|
|
|
+
|
|
|
+```
|
|
|
+store/…/models/checkpoint-expanded-scales/gb10-tp4-202611112207/
|
|
|
+ model.safetensors 33 MiB mode 0444 (read-only)
|
|
|
+ weights/ 470 G
|
|
|
+ deployments/db20209d361f473b8452f24ad517fa0d/
|
|
|
+ rank-0 rank-1 rank-2 rank-3 1.3 G each, 5.1 G total
|
|
|
+```
|
|
|
+
|
|
|
+`flare models list`: `1 model · 1 version`, `Format payload v2`,
|
|
|
+`Disk used 474.66 GiB`, `Shared files 474.63 GiB · 181 hardlinked objects`,
|
|
|
+`Deployment db20209d361f473b8452f24ad517fa0d · TP 4 · ready`,
|
|
|
+`Context / prefill 1,048,576 / 8,192 tokens`, `Pages / state rows 128 / 448`.
|
|
|
+`flare models inspect checkpoint-expanded-scales`: name **DeepSeek-V4.1-Flash**,
|
|
|
+publisher *local-inference-lab (signatures verified)*, **sm_121**, vocab 129,280,
|
|
|
+**concurrency 32 · 4 ranks**, speculation `mtp · widths 1,2,3,4,5,6`.
|
|
|
+
|
|
|
+Acquisition accounting: `network 509,628,494,618 B`, `local source 509,653,222,270 B`,
|
|
|
+`verification 509,646,855,013 B`, `materialized 0 B`, **elapsed 7682 s (2 h 08 m)**,
|
|
|
+average 63 MiB/s (peak 104 MiB/s measured on `du`). **No 2× leftover**: the transport
|
|
|
+cache was folded into `blobs/` — it peaked at 475 G next to a growing `blobs/`, and at
|
|
|
+the end `transport/` is 88 K and `blobs/` is the whole 475 G. So the earlier 2×-peak
|
|
|
+warning applies to the *middle* of the run only, and `models gc` was not needed.
|
|
|
+
|
|
|
+**This changes the Phase 3 arithmetic, in our favour.** A rank slice is
|
|
|
+**three files**: `initial-state.safetensors` 1,321,056,820 B,
|
|
|
+`kernels.safetensors` 39,429,126 B, `graphs.safetensors` 4,207,295 B. At the measured
|
|
|
+NAS rates (660–820 MiB/s at 1 MiB reads) that is **~2 s per rank** — the small-file
|
|
|
+fear in §6 item 3 does not apply to this package. `model.safetensors` is mode 0444, so
|
|
|
+bind-mount it read-write and do not expect to replace it.
|
|
|
+
|
|
|
### v0.1.0 vocabulary and paths — read before running any command (2026-11-11, R4)
|
|
|
|
|
|
The runtime we are moving to renamed the artefact. Measured against both binaries on
|
|
|
@@ -576,6 +611,15 @@ Do this **before** touching dgx1–dgx3.
|
|
|
|
|
|
Use **dgx4**: it is idle (no ranks), and now boots from the new MP600 Mini.
|
|
|
|
|
|
+The paths for this gate now exist — do not invent them. Source (NAS):
|
|
|
+`/mnt/nas/ai/flare/store/models/checkpoint-expanded-scales/gb10-tp4-202611112207/`
|
|
|
+with `model.safetensors` (0444), `weights/` (470 G) and
|
|
|
+`deployments/db20209d361f473b8452f24ad517fa0d/rank-<N>/` (3 files, 1.3 G each).
|
|
|
+Put the local copy at
|
|
|
+`~/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-202611112207/`
|
|
|
+(v0.1.0's default root is `~/.cache/flare/cores`, but this fleet pins
|
|
|
+`paths.package_cache`, so use whatever that key says on the node under test).
|
|
|
+
|
|
|
1. Copy one model's small artefacts from a running node:
|
|
|
```bash
|
|
|
ssh dgx8.lan 'tar -C ~/.cache/flare/models/<model>/<version> -cf - model.safetensors deployments' \
|