|
@@ -0,0 +1,290 @@
|
|
|
|
|
+# FLARE storage: expanded checkpoint on node NVMe, everything else on NAS (2026-11-10)
|
|
|
|
|
+
|
|
|
|
|
+Goal, in the operator's words: **only the expanded checkpoint stays on the sparks
|
|
|
|
|
+NVMe** (260 GiB for `DeepSeek-V4.1-Flash` tp4) and **all other model artefacts live
|
|
|
|
|
+on the NAS**. This document is written to be executed by someone with no context
|
|
|
|
|
+beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
|
|
+
|
|
|
|
|
+## 1. Measured baseline (2026-11-10, read-only)
|
|
|
|
|
+
|
|
|
|
|
+| node | role | `models/` | `packages/` | rank cmdline cache |
|
|
|
|
|
+|---|---|---|---|---|
|
|
|
|
|
+| dgx8 | sparks rank 3 | 261 GiB | — | `--package-cache /home/kilork/.cache/flare` |
|
|
|
|
|
+| dgx1 | prod4 rank 0 | 261 GiB | 477 GiB | `--package-cache /home/kilork/.cache/flare` |
|
|
|
|
|
+
|
|
|
|
|
+Anatomy of one model directory — **this split is the whole plan**:
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+~/.cache/flare/models/checkpoint-expanded-scales/<version>/
|
|
|
|
|
+ model.safetensors 35 MiB signed envelope (manifest + locations)
|
|
|
|
|
+ deployments/<pkg>/rank-<N>/ 788 KiB (rank 0) … 1.8 GiB (rank 1-3)
|
|
|
|
|
+ weights/ 260 GiB ← the expanded checkpoint
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Package store (CLI side, rank 0 only):
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+~/.cache/flare/packages/<hash>-<rev>/
|
|
|
|
|
+ model.safetensors 35 MiB
|
|
|
|
|
+ deployments/ 7.2 GiB
|
|
|
|
|
+ weights/ 470 GiB ← second copy of the same content
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+NAS: `/mnt/nas` = 33 TiB, 1.1 TiB used, `/mnt/nas/ai/flare` effectively empty.
|
|
|
|
|
+
|
|
|
|
|
+### What the move actually saves — read before committing
|
|
|
|
|
+
|
|
|
|
|
+- sparks ranks 1–3: **1.8 GiB each** (261 → 260 GiB). Small.
|
|
|
|
|
+- rank 0 (prod4): **477 GiB** (the duplicate package store) plus 7.2 GiB of deployments.
|
|
|
|
|
+- Future model versions stop multiplying: only their `weights/` ever lands on NVMe.
|
|
|
|
|
+
|
|
|
|
|
+If the intent is "NVMe holds one checkpoint and nothing else", this plan delivers
|
|
|
|
|
+it. If the intent was primarily disk relief on sparks, the win is small and the
|
|
|
|
|
+cheap part (phase 2, the package store) may be enough.
|
|
|
|
|
+
|
|
|
|
|
+## 2. Target layout
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+# every rank node, local NVMe
|
|
|
|
|
+/home/kilork/.cache/flare/models/<model>/<version>/
|
|
|
|
|
+ weights/ real directory 260 GiB
|
|
|
|
|
+ model.safetensors bind mount → NAS/nodes/<node>/<version>/model.safetensors
|
|
|
|
|
+ deployments/ bind mount → NAS/nodes/<node>/<version>/deployments
|
|
|
|
|
+
|
|
|
|
|
+# NAS
|
|
|
|
|
+/mnt/nas/ai/flare/
|
|
|
|
|
+ store/ paths.package_cache — CLI store, packages/, blobs/, staging/,
|
|
|
|
|
+ refs/ (rank 0's 477 GiB lives here, once)
|
|
|
|
|
+ nodes/<node>/<model>/<version>/{model.safetensors,deployments/}
|
|
|
|
|
+ per-node small artefacts (bind targets)
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Only `weights/` is allowed on node NVMe. Everything else resolves through the
|
|
|
|
|
+bind mounts, so FLARE's view of a model directory is unchanged.
|
|
|
|
|
+
|
|
|
|
|
+## 3. Why bind mounts and not symlinks
|
|
|
|
|
+
|
|
|
|
|
+FLARE's own error strings include *"a package primary escapes its directory"* and
|
|
|
|
|
+*"staged entry is not a regular file"* — a symlink pointing outside the model
|
|
|
|
|
+directory is exactly what those checks reject. Bind mounts are invisible to the
|
|
|
|
|
+process: the path stays inside the directory and the entry is a real file. Do not
|
|
|
|
|
+substitute symlinks.
|
|
|
|
|
+
|
|
|
|
|
+**Known risk:** if FLARE ever replaces `deployments/` by rename (`rm -rf` +
|
|
|
|
|
+`mv`), the mount point disappears and writes fall back to local disk. Phase 3 is a
|
|
|
|
|
+gate that measures this; if it happens, fall back to **option D**: leave
|
|
|
|
|
+`deployments/` local (1.8 GiB) and move only the package store. Do not fight it.
|
|
|
|
|
+
|
|
|
|
|
+## 4. Phases
|
|
|
|
|
+
|
|
|
|
|
+Each phase is independently revertible. Do not start a phase whose gate the
|
|
|
|
|
+previous phase failed.
|
|
|
|
|
+
|
|
|
|
|
+### Phase 0 — prerequisites (no FLARE restart yet)
|
|
|
|
|
+
|
|
|
|
|
+1. **Rebuild the runtime before any agent restart.** The installed binary is the
|
|
|
|
|
+ PPA build (`0.1.0~git20261105002735.0009b89d-1`), which lacks PR #184. The
|
|
|
|
|
+ running agents still hold the PR-184 inode; restarting without the rebuild
|
|
|
|
|
+ drops the io_uring Engram reader on dgx1/2/3/8. See `references/build.md` in
|
|
|
|
|
+ the `flare` skill, or `flare-upstream.pin` + `pr184-on-main` here. Verify:
|
|
|
|
|
+ `strings <binary> | grep -c "embedding read worker panicked"` → `1`.
|
|
|
|
|
+2. Confirm NAS health and space, from every node that will use it:
|
|
|
|
|
+ ```bash
|
|
|
|
|
+ ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
|
|
|
|
|
+ ```
|
|
|
|
|
+3. Record the current state of each node for rollback:
|
|
|
|
|
+ ```bash
|
|
|
|
|
+ du -sh ~/.cache/flare/* | sort -h > /tmp/flare-before-$(hostname).txt
|
|
|
|
|
+ findmnt -no TARGET,SOURCE,OPTIONS /mnt/nas
|
|
|
|
|
+ pgrep -af "[f]lare run" | head -1
|
|
|
|
|
+ ```
|
|
|
|
|
+4. **dgx8 currently has the NAS in fstab but not mounted.** Fix that first
|
|
|
|
|
+ (`sudo mount /mnt/nas`), and make it non-blocking for boot by adding `nofail`
|
|
|
|
|
+ and a timeout to the fstab line (see phase 1).
|
|
|
|
|
+
|
|
|
|
|
+Gate: binary rebuilt and staged, NAS mounted on every participating node.
|
|
|
|
|
+
|
|
|
|
|
+### Phase 1 — make the NAS mount boot-proof
|
|
|
|
|
+
|
|
|
|
|
+The existing line (all nodes) is:
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+nas.lan:/sna /mnt/nas nfs4 rw,hard,rsize=1048576,wsize=1048576,timeo=600,retrans=5,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Two problems for this plan: a NAS hiccup blocks boot, and `hard` + long `timeo`
|
|
|
|
|
+means a stale mount hangs reads (this NAS has already produced a
|
|
|
|
|
+`server not responding` outage that needed reboots).
|
|
|
|
|
+
|
|
|
|
|
+Change to:
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+nas.lan:/sna /mnt/nas nfs4 rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+sudo cp /etc/fstab /etc/fstab.bak-$(date +%F)
|
|
|
|
|
+sudo sed -i 's|^\(nas.lan:/sna.*\)$|# replaced by the storage plan\nnas.lan:/sna /mnt/nas nfs4 rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0|' /etc/fstab
|
|
|
|
|
+sudo systemctl daemon-reload
|
|
|
|
|
+sudo umount /mnt/nas 2>/dev/null; sudo mount /mnt/nas
|
|
|
|
|
+findmnt -no TARGET,OPTIONS /mnt/nas | grep -o 'nofail'
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Gate: mount present, `nofail` in effect, a `sudo mount -a` is a no-op.
|
|
|
|
|
+
|
|
|
|
|
+### Phase 2 — package store (the 477 GiB) to NAS
|
|
|
|
|
+
|
|
|
|
|
+Low risk, no bind mounts, no FLARE code path surprises.
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+sudo mkdir -p /mnt/nas/ai/flare/store
|
|
|
|
|
+sudo chown kilork:kilork /mnt/nas/ai/flare/store
|
|
|
|
|
+# rsync -H is mandatory: format-2 packages are hardlink stores
|
|
|
|
|
+rsync -aH --info=progress2 ~/.cache/flare/packages/ /mnt/nas/ai/flare/store/
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Point the CLI at it, in `~/.config/flare/config.toml` on the node that downloads
|
|
|
|
|
+(rank 0 / controller):
|
|
|
|
|
+
|
|
|
|
|
+```toml
|
|
|
|
|
+[paths]
|
|
|
|
|
+package_cache = "/mnt/nas/ai/flare/store"
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Then remove the local copy only after a verified download check:
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+flare models list --model-cache /mnt/nas/ai/flare/store # must list the version
|
|
|
|
|
+rm -rf ~/.cache/flare/packages
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Gate: `flare models list` sees the version on NAS, `df -h /` shows the space back,
|
|
|
|
|
+and the next `orbit run` still starts (see phase 4 gate for the full test).
|
|
|
|
|
+
|
|
|
|
|
+**Open item to verify, not assume:** ranks are launched with an explicit
|
|
|
|
|
+`--package-cache /home/kilork/.cache/flare` that does **not** come from
|
|
|
|
|
+`[paths]` (the agent synthesises it). Check the launched cmdline after a start:
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+pgrep -af "[f]lare run" | head -1 | tr ' ' '\n' | grep -A1 -e package-cache -e model-cache
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+If that path needs to change, it is a **launcher** change
|
|
|
|
|
+(`[flare].extra_flags` / `[flare.clusters.<name>].extra_flags` in the sm121
|
|
|
|
|
+config, `src/native/flare_orbit.rs::extra_flags_for`) — never a hand edit on a
|
|
|
|
|
+node.
|
|
|
|
|
+
|
|
|
|
|
+### Phase 3 — validate the bind-mount model on an idle node (gate)
|
|
|
|
|
+
|
|
|
|
|
+Use **dgx4**: it is idle (no ranks), and now boots from the new MP600 Mini.
|
|
|
|
|
+
|
|
|
|
|
+1. Copy one model's small artefacts from a running node:
|
|
|
|
|
+ ```bash
|
|
|
|
|
+ ssh dgx8.lan 'tar -C ~/.cache/flare/models/<model>/<version> -cf - model.safetensors deployments' \
|
|
|
|
|
+ | tar -C /tmp -xf -
|
|
|
|
|
+ ```
|
|
|
|
|
+2. Build a fake local model dir whose `weights/` is a symlink to nothing and whose
|
|
|
|
|
+ `deployments/` is a **bind mount from the NAS copy**:
|
|
|
|
|
+ ```bash
|
|
|
|
|
+ mkdir -p ~/.cache/flare/models/<model>/<version>/weights
|
|
|
|
|
+ mv /tmp/model.safetensors /tmp/deployments ~/.cache/flare/models/<model>/<version>/
|
|
|
|
|
+ sudo mkdir -p /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>
|
|
|
|
|
+ sudo mv ~/.cache/flare/models/<model>/<version>/deployments /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>/
|
|
|
|
|
+ sudo mount --bind /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>/deployments \
|
|
|
|
|
+ ~/.cache/flare/models/<model>/<version>/deployments
|
|
|
|
|
+ ```
|
|
|
|
|
+3. Ask FLARE what it sees:
|
|
|
|
|
+ ```bash
|
|
|
|
|
+ flare inspect ~/.cache/flare/models/<model>/<version>/model.safetensors | head -20
|
|
|
|
|
+ flare models list
|
|
|
|
|
+ ```
|
|
|
|
|
+4. If a stage/launch is possible on dgx4 (`flare run --stage-only` with a dummy
|
|
|
|
|
+ agent), run it and then check the mount survived:
|
|
|
|
|
+ ```bash
|
|
|
|
|
+ findmnt -no TARGET ~/.cache/flare/models/<model>/<version>/deployments
|
|
|
|
|
+ ```
|
|
|
|
|
+ **Empty output = FLARE replaced the directory → take option D.**
|
|
|
|
|
+
|
|
|
|
|
+Gate: `flare inspect` shows the deployment, and `findmnt` still reports the bind
|
|
|
|
|
+mount after any stage operation.
|
|
|
|
|
+
|
|
|
|
|
+### Phase 4 — roll out to sparks (window; ranks stop)
|
|
|
|
|
+
|
|
|
|
|
+Repeat per node in rank order, one node at a time, verifying before moving on.
|
|
|
|
|
+
|
|
|
|
|
+1. Stop the group (needs a working controller; see the re-pair note in
|
|
|
|
|
+ `flare-bringup.md`):
|
|
|
|
|
+ ```bash
|
|
|
|
|
+ flare orbit stop <group> --cluster sparks
|
|
|
|
|
+ ```
|
|
|
|
|
+2. Move the node's small artefacts to NAS and bind them back:
|
|
|
|
|
+ ```bash
|
|
|
|
|
+ V=gb10-tp4-<build>; M=checkpoint-expanded-scales
|
|
|
|
|
+ D=/home/kilork/.cache/flare/models/$M/$V
|
|
|
|
|
+ N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V
|
|
|
|
|
+ sudo mkdir -p "$N"
|
|
|
|
|
+ rsync -aH --remove-source-files "$D/model.safetensors" "$N/"
|
|
|
|
|
+ rsync -aH "$D/deployments/" "$N/deployments/"
|
|
|
|
|
+ rm -rf "$D/deployments"
|
|
|
|
|
+ sudo mount --bind "$N/deployments" "$D/deployments"
|
|
|
|
|
+ ```
|
|
|
|
|
+ Persist it in `/etc/fstab` (bind mounts are ordinary fstab entries and are
|
|
|
|
|
+ mounted before the agent starts):
|
|
|
|
|
+ ```
|
|
|
|
|
+ /mnt/nas/ai/flare/nodes/dgx5/checkpoint-expanded-scales/gb10-tp4-<build>/deployments /home/kilork/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-<build>/deployments none bind,nofail,x-systemd.requires=/mnt/nas,x-systemd.requires-mounts-for=/mnt/nas 0 0
|
|
|
|
|
+ ```
|
|
|
|
|
+3. Start the group and verify on the node:
|
|
|
|
|
+ ```bash
|
|
|
|
|
+ findmnt -no TARGET,SOURCE ~/.cache/flare/models/$M/$V/deployments
|
|
|
|
|
+ du -sh ~/.cache/flare/models/$M/$V/* | sort -h # weights ~260G, rest tiny
|
|
|
|
|
+ pgrep -af "[f]lare run" | head -1
|
|
|
|
|
+ ```
|
|
|
|
|
+4. Only then move to the next node.
|
|
|
|
|
+
|
|
|
|
|
+Gate: ranks serve, `findmnt` shows the bind mounts, `du` shows ~260 GiB of
|
|
|
|
|
+`weights/` and nothing else of size.
|
|
|
|
|
+
|
|
|
|
|
+### Phase 5 — prod4 (dgx1–dgx4), same steps
|
|
|
|
|
+
|
|
|
|
|
+Same procedure with `--cluster prod4`. dgx1 is rank 0 and is the one that gains the
|
|
|
|
|
+477 GiB; it is also the node whose `packages/` phase 2 removed.
|
|
|
|
|
+
|
|
|
|
|
+### Phase 6 — verification and rollback
|
|
|
|
|
+
|
|
|
|
|
+Verification, per node:
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+du -sh ~/.cache/flare/models/*/*/* # weights dominates
|
|
|
|
|
+findmnt -no TARGET,SOURCE | grep deployments
|
|
|
|
|
+df -h / /mnt/nas
|
|
|
|
|
+flare orbit status --cluster sparks # agents reachable, groups listed
|
|
|
|
|
+curl -s localhost:<port>/v1/models | head # API answers (port from the rank cmdline)
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Rollback (per node, no data loss — the NAS copy is authoritative):
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+sudo umount ~/.cache/flare/models/$M/$V/deployments
|
|
|
|
|
+sudo sed -i "\|$N/deployments|d" /etc/fstab
|
|
|
|
|
+rsync -aH /mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V/deployments/ ~/.cache/flare/models/$M/$V/deployments/
|
|
|
|
|
+flare orbit run <group> --cluster sparks
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+## 5. Risks and their fallbacks
|
|
|
|
|
+
|
|
|
|
|
+| risk | detection | fallback |
|
|
|
|
|
+|---|---|---|
|
|
|
|
|
+| FLARE replaces `deployments/` by rename; bind mount lost | `findmnt` empty after a stage | **Option D**: deployments local, only the package store moves |
|
|
|
|
|
+| NAS stall at boot | node boots with `/mnt/nas` missing | `nofail` (phase 1) keeps boot going; agent then writes deployments locally — re-sync after recovery |
|
|
|
|
|
+| NAS outage during serving | reads hang (`hard` mount) | bind mounts are only touched at load; keep `timeo=100,retrans=3`; do not put `staging/` on NAS |
|
|
|
|
|
+| hardlink-blind copy duplicates the package | `du` doubles after a copy | always `rsync -aH` |
|
|
|
|
|
+| mixed flare binaries after a restart | rendezvous `incompatible protocol` | phase 0 rebuild first, restart all ranks together |
|
|
|
|
|
+
|
|
|
|
|
+## 6. What must be measured, not assumed
|
|
|
|
|
+
|
|
|
|
|
+1. Whether the agent's `--package-cache` path can be redirected by config or needs
|
|
|
|
|
+ a launcher `extra_flags` change (phase 2 note).
|
|
|
|
|
+2. Whether FLARE tolerates the bind mount across a real stage (phase 3 gate).
|
|
|
|
|
+3. Real NAS read throughput for `deployments/` at load — 1.8 GiB over NFS at
|
|
|
|
|
+ 1 GbE is ~15 s per rank; measure before rolling to prod, and keep `weights/`
|
|
|
|
|
+ local regardless.
|