|
|
@@ -14,8 +14,12 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
that works there (4.0) — see the correction in Phase 1
|
|
|
- [x] Pre-migration state of dgx1/2/3/4/8 captured locally and on the NAS
|
|
|
- [x] NAS throughput measured (was open item 6.3)
|
|
|
-- [ ] PR #184 binary rebuilt and staged (Phase 0.1) — needs dgx5 powered on
|
|
|
-- [ ] Package store (477 GiB) on the NAS, local copy removed, config pointed at it (Phase 2)
|
|
|
+- [x] **Stale model data purged from the nodes — 783 GiB freed**, dgx1 from 98 % full
|
|
|
+ (22 G free) to 283 G free; serving never interrupted
|
|
|
+- [ ] PR #184 binary rebuilt on the Nov-5 base and staged (Phase 0.1) — needs dgx5
|
|
|
+ powered on, and is now **required** to load the current package at all
|
|
|
+- [ ] **Fresh `gb10` package acquired onto the NAS store and deployed; superseded
|
|
|
+ build deleted** (Phase 2R — replaces the "migrate the old store" plan)
|
|
|
- [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
|
|
|
- [ ] Sparks rolled out (Phase 4, needs a window)
|
|
|
- [ ] prod4 rolled out (Phase 5, needs a window)
|
|
|
@@ -24,15 +28,20 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
## Round plan
|
|
|
|
|
|
- [x] **R1 (Q1)** — Phase 0.2–0.4 + Phase 1 on dgx1/2/3/4/8
|
|
|
-- [ ] **R2 (Q1)** — Phase 2 first half: `rsync -aH` the 477 GiB store to the NAS in
|
|
|
- the background, then verify with `flare models list --model-cache`
|
|
|
-- [ ] **R3 (Q1)** — Phase 2 second half: point `[paths] package_cache` at the NAS,
|
|
|
- remove the local store, confirm the launched rank cmdline (open item 6.1)
|
|
|
-- [ ] **R4 (Q1)** — PR #184 rebuild and staging (needs dgx5 on; prerequisite for any restart)
|
|
|
-- [ ] **R5 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4
|
|
|
-- [ ] **R6 (Q2)** — Phase 4 sparks rollout (window; ranks stop)
|
|
|
-- [ ] **R7 (Q2)** — Phase 5 prod4 rollout (window; ranks stop)
|
|
|
-- [ ] **R8 (Q3)** — Phase 6 verification matrix and rollback drill
|
|
|
+- [x] **R2 (Q1)** — stale-data purge instead of migration (operator decision):
|
|
|
+ **783 GiB freed** on dgx1/2/3, all ranks stayed up; also identified that every
|
|
|
+ local package is a superseded build from a deleted repo (Phase 2R)
|
|
|
+- [ ] **R3 (Q1)** — Phase 2R first half: download the current `gb10` package
|
|
|
+ (474.7 GiB) into the NAS store with `flare download … --revision gb10`,
|
|
|
+ verify with `flare models list --model-cache /mnt/nas/ai/flare/store`
|
|
|
+- [ ] **R4 (Q1)** — Phase 2R second half: fix dgx1's `package_cache` drift (it points
|
|
|
+ at an empty `~/flare-pkgs/gb10`), confirm the launched rank cmdline (open item 6.1)
|
|
|
+- [ ] **R5 (Q1)** — PR #184 rebuild **on the Nov-5 base** and staging (needs dgx5 on;
|
|
|
+ prerequisite for both any restart and loading the new package)
|
|
|
+- [ ] **R6 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4
|
|
|
+- [ ] **R7 (Q2)** — Phase 4 sparks rollout (window; ranks stop)
|
|
|
+- [ ] **R8 (Q2)** — Phase 5 prod4 rollout (window; ranks stop)
|
|
|
+- [ ] **R9 (Q3)** — Phase 6 verification matrix and rollback drill
|
|
|
|
|
|
## Checks
|
|
|
|
|
|
@@ -218,9 +227,43 @@ dgx1/2/3/4/8 (2026-11-11):** live options show `timeo=100`, `mount -a` returns 0
|
|
|
without remounting, all rank PIDs kept their 7-day uptime and the API on
|
|
|
`dgx1:30000` still answered. dgx5–7 are off and still need this.
|
|
|
|
|
|
-### Phase 2 — package store (the 477 GiB) to NAS
|
|
|
+### Phase 2 — package store (the 477 GiB) to NAS — **SUPERSEDED, do not run**
|
|
|
|
|
|
-Low risk, no bind mounts, no FLARE code path surprises.
|
|
|
+> **Superseded on 2026-11-11 (R2).** Two independent reasons:
|
|
|
+> 1. The operator rejected it: *“Syncing those old flare cache to store does not make
|
|
|
+> much sense honestly.”* Copying a superseded build onto the NAS is not migration,
|
|
|
+> it is archiving garbage.
|
|
|
+> 2. It is superseded by fact: the local package is an **out-of-date build whose source
|
|
|
+> repo no longer exists** (see Phase 2R). The rsync was started, reached 105 GiB,
|
|
|
+> was killed, and the partial copy was deleted.
|
|
|
+
|
|
|
+What replaced it: **purge the stale data** (done, below) and **acquire the current
|
|
|
+revision onto the NAS store** (Phase 2R).
|
|
|
+
|
|
|
+#### Purge performed on 2026-11-11
|
|
|
+
|
|
|
+Every node held the **Nov-2 build** `checkpoint-expanded-scales/gb10-tp4-202611021936`
|
|
|
+(261 GiB each) with nothing holding it open — verified three ways per node before
|
|
|
+deleting (`/proc/<rank>/fd`, `/proc/<rank>/maps`, and `lsof +D`):
|
|
|
+
|
|
|
+| node | before | after | freed |
|
|
|
+|---|---|---|---|
|
|
|
+| dgx1 | 848 G used, **22 G free (98 %)** | 587 G used, **283 G free** | 261 G |
|
|
|
+| dgx2 | 664 G used, 206 G free | 403 G used, 467 G free | 261 G |
|
|
|
+| dgx3 | 661 G used, 209 G free | 400 G used, 470 G free | 261 G |
|
|
|
+
|
|
|
+**Total 783 GiB freed.** Rank PIDs were unchanged (7-day uptime) and `dgx1:30000/v1/models`
|
|
|
+kept answering. Note dgx1's root is **915 GiB**, not 1.8 TB like dgx4 — the 261 GiB
|
|
|
+was 28 % of that disk.
|
|
|
+
|
|
|
+Still present and **not deletable while serving**:
|
|
|
+- dgx1 `~/.cache/flare/packages/7cf40bec…-35842073` — 477 G, and rank 0 has its
|
|
|
+ `weights/engram-*.safetensors` (92 G each) open. Space returns only when the rank
|
|
|
+ exits, so this one dies in the Phase 5 window, not before.
|
|
|
+- dgx2/dgx3 `models/…202611032142` and dgx8 `models/…202611022100` (261 G each) — the
|
|
|
+ live ranks read these.
|
|
|
+
|
|
|
+For reference, the original (not executed) procedure was:
|
|
|
|
|
|
```bash
|
|
|
sudo mkdir -p /mnt/nas/ai/flare/store
|
|
|
@@ -260,8 +303,73 @@ If that path needs to change, it is a **launcher** change
|
|
|
config, `src/native/flare_orbit.rs::extra_flags_for`) — never a hand edit on a
|
|
|
node.
|
|
|
|
|
|
+### Phase 2R — replace, don't migrate (2026-11-11)
|
|
|
+
|
|
|
+**What we are actually serving is superseded, and its source is gone.** The acquired
|
|
|
+package's manifest (read out of the `flare.manifest` tensor inside `model.safetensors`
|
|
|
+— it is protobuf, not JSON, but the strings are legible) names:
|
|
|
+
|
|
|
+```
|
|
|
+source https://huggingface.co/local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE2
|
|
|
+model checkpoint-expanded-scales identity gb10 publisher local-inference-lab
|
|
|
+```
|
|
|
+
|
|
|
+That repo answers **404 with a valid token** (401 unauthenticated is HF masking a
|
|
|
+missing repo). The current source is the same publisher **without the `2`**, and
|
|
|
+`gb10` is a *branch* of it:
|
|
|
+
|
|
|
+| | |
|
|
|
+|---|---|
|
|
|
+| repo | `local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE` (public, no token needed) |
|
|
|
+| revision | branch `gb10` — last commit **2026-11-05T01:42:45** *“Publish the GB10 TP4 package with vocabulary-parallel sampling and an FP8 Markov head”* |
|
|
|
+| size | **187 files, 474.7 GiB** (we hold 184 files / 477 GiB from a Nov-3 pull) |
|
|
|
+| requires | `flare-inference` **`0.1.0~git20261105002735.0009b89d-1` or newer** — “earlier versions cannot load it” |
|
|
|
+
|
|
|
+That version is exactly what the fleet has *installed* on every node. The binaries
|
|
|
+*running* are the older PR-184 inodes, so **no node can load the new package until
|
|
|
+the Nov-5-based rebuild (R5) lands** — the same conclusion #11 reached, now with a
|
|
|
+functional reason instead of a performance one.
|
|
|
+
|
|
|
+Download into the NAS store (this is what the store is *for* — fresh acquisitions,
|
|
|
+not old caches). The `hf` token, when one is needed, is `export HF_TOKEN=…` in
|
|
|
+`~/.zshenv` **on the nodes** — there is no token on the mac:
|
|
|
+
|
|
|
+```bash
|
|
|
+package_path=$(flare download \
|
|
|
+ local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE --revision gb10)
|
|
|
+# to land it on the NAS store rather than the local cache, point the CLI at it:
|
|
|
+flare download local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE --revision gb10 \
|
|
|
+ --model-cache /mnt/nas/ai/flare/store
|
|
|
+flare models list --model-cache /mnt/nas/ai/flare/store # must list the new version
|
|
|
+```
|
|
|
+
|
|
|
+The upstream launch recipe (`SPARK-DS4-GUIDE.md` / the `gb10` README) is worth
|
|
|
+aligning our launcher flags with — note it differs from what prod4 runs today:
|
|
|
+
|
|
|
+```bash
|
|
|
+flare orbit run "spark1:${package_path}" \
|
|
|
+ --cluster sparks --name deepseek1017 \
|
|
|
+ --max-concurrency 32 --mtp-draft-tokens 5 --mtp-adaptive-verification \
|
|
|
+ --mtp-adaptive-verification-alpha 0.75 \
|
|
|
+ --embedding-table-storage disk \
|
|
|
+ --embedding-table-resident-scales --embedding-table-cache-size 1GiB \
|
|
|
+ --host 0.0.0.0 --port 8000 --detach
|
|
|
+```
|
|
|
+
|
|
|
+(Differences from our running rank 0: it adds `--mtp-adaptive-verification-alpha 0.75`,
|
|
|
+uses port 8000 not 30000, and names the group. Orbit ships each peer only the data it
|
|
|
+needs — “You do not need four full downloads.”)
|
|
|
+
|
|
|
+Order of work: **R5 rebuild → download fresh to NAS (R3) → window: stop ranks, delete
|
|
|
+the superseded 477 G on dgx1 and the 261 G dirs everywhere, start on the new package
|
|
|
+→ then the bind-mount layout (Phase 3+) on top of the new version dirs.** Doing the
|
|
|
+bind-mount work on the old version dirs would be wasted: the paths contain the version
|
|
|
+name, and every fstab entry would have to be rewritten.
|
|
|
+
|
|
|
### Phase 3 — validate the bind-mount model on an idle node (gate)
|
|
|
|
|
|
+Do this **before** touching dgx1–dgx3.
|
|
|
+
|
|
|
Use **dgx4**: it is idle (no ranks), and now boots from the new MP600 Mini.
|
|
|
|
|
|
1. Copy one model's small artefacts from a running node:
|