Pārlūkot izejas kodu

docs(flare): plan for weights-local / everything-else-on-NAS storage

Measured baseline (dgx8 261 GiB model dir: 260 weights + 1.8 deployments + 35 MiB
envelope; dgx1 additionally 477 GiB package store) and a phased, revertible
procedure for moving everything except weights/ to the NAS.

Records what the move is actually worth: 1.8 GiB per sparks rank, 477 GiB on
rank 0 — so the package store (phase 2) may be the only part worth doing on
sparks. Documents why symlinks are not an option (FLARE rejects a package primary
that escapes its directory), the bind-mount design, the gate that decides between
it and 'deployments stay local', and the boot-safety changes the NAS mount needs
(nofail, shorter timeo) given this NAS's outage history.
Alexander Korolev 1 nedēļu atpakaļ
vecāks
revīzija
6fa147f042
1 mainītis faili ar 290 papildinājumiem un 0 dzēšanām
  1. 290 0
      specs/tech-architecture/flare-storage-nas.md

+ 290 - 0
specs/tech-architecture/flare-storage-nas.md

@@ -0,0 +1,290 @@
+# FLARE storage: expanded checkpoint on node NVMe, everything else on NAS (2026-11-10)
+
+Goal, in the operator's words: **only the expanded checkpoint stays on the sparks
+NVMe** (260 GiB for `DeepSeek-V4.1-Flash` tp4) and **all other model artefacts live
+on the NAS**. This document is written to be executed by someone with no context
+beyond it. Every command is copy-pasteable and every phase has a gate.
+
+## 1. Measured baseline (2026-11-10, read-only)
+
+| node | role | `models/` | `packages/` | rank cmdline cache |
+|---|---|---|---|---|
+| dgx8 | sparks rank 3 | 261 GiB | — | `--package-cache /home/kilork/.cache/flare` |
+| dgx1 | prod4 rank 0 | 261 GiB | 477 GiB | `--package-cache /home/kilork/.cache/flare` |
+
+Anatomy of one model directory — **this split is the whole plan**:
+
+```
+~/.cache/flare/models/checkpoint-expanded-scales/<version>/
+  model.safetensors      35 MiB      signed envelope (manifest + locations)
+  deployments/<pkg>/rank-<N>/  788 KiB (rank 0) … 1.8 GiB (rank 1-3)
+  weights/              260 GiB      ← the expanded checkpoint
+```
+
+Package store (CLI side, rank 0 only):
+
+```
+~/.cache/flare/packages/<hash>-<rev>/
+  model.safetensors      35 MiB
+  deployments/          7.2 GiB
+  weights/              470 GiB      ← second copy of the same content
+```
+
+NAS: `/mnt/nas` = 33 TiB, 1.1 TiB used, `/mnt/nas/ai/flare` effectively empty.
+
+### What the move actually saves — read before committing
+
+- sparks ranks 1–3: **1.8 GiB each** (261 → 260 GiB). Small.
+- rank 0 (prod4): **477 GiB** (the duplicate package store) plus 7.2 GiB of deployments.
+- Future model versions stop multiplying: only their `weights/` ever lands on NVMe.
+
+If the intent is "NVMe holds one checkpoint and nothing else", this plan delivers
+it. If the intent was primarily disk relief on sparks, the win is small and the
+cheap part (phase 2, the package store) may be enough.
+
+## 2. Target layout
+
+```
+# every rank node, local NVMe
+/home/kilork/.cache/flare/models/<model>/<version>/
+    weights/              real directory                    260 GiB
+    model.safetensors     bind mount → NAS/nodes/<node>/<version>/model.safetensors
+    deployments/          bind mount → NAS/nodes/<node>/<version>/deployments
+
+# NAS
+/mnt/nas/ai/flare/
+    store/                paths.package_cache — CLI store, packages/, blobs/, staging/,
+                          refs/ (rank 0's 477 GiB lives here, once)
+    nodes/<node>/<model>/<version>/{model.safetensors,deployments/}
+                          per-node small artefacts (bind targets)
+```
+
+Only `weights/` is allowed on node NVMe. Everything else resolves through the
+bind mounts, so FLARE's view of a model directory is unchanged.
+
+## 3. Why bind mounts and not symlinks
+
+FLARE's own error strings include *"a package primary escapes its directory"* and
+*"staged entry is not a regular file"* — a symlink pointing outside the model
+directory is exactly what those checks reject. Bind mounts are invisible to the
+process: the path stays inside the directory and the entry is a real file. Do not
+substitute symlinks.
+
+**Known risk:** if FLARE ever replaces `deployments/` by rename (`rm -rf` +
+`mv`), the mount point disappears and writes fall back to local disk. Phase 3 is a
+gate that measures this; if it happens, fall back to **option D**: leave
+`deployments/` local (1.8 GiB) and move only the package store. Do not fight it.
+
+## 4. Phases
+
+Each phase is independently revertible. Do not start a phase whose gate the
+previous phase failed.
+
+### Phase 0 — prerequisites (no FLARE restart yet)
+
+1. **Rebuild the runtime before any agent restart.** The installed binary is the
+   PPA build (`0.1.0~git20261105002735.0009b89d-1`), which lacks PR #184. The
+   running agents still hold the PR-184 inode; restarting without the rebuild
+   drops the io_uring Engram reader on dgx1/2/3/8. See `references/build.md` in
+   the `flare` skill, or `flare-upstream.pin` + `pr184-on-main` here. Verify:
+   `strings <binary> | grep -c "embedding read worker panicked"` → `1`.
+2. Confirm NAS health and space, from every node that will use it:
+   ```bash
+   ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
+   ```
+3. Record the current state of each node for rollback:
+   ```bash
+   du -sh ~/.cache/flare/* | sort -h > /tmp/flare-before-$(hostname).txt
+   findmnt -no TARGET,SOURCE,OPTIONS /mnt/nas
+   pgrep -af "[f]lare run" | head -1
+   ```
+4. **dgx8 currently has the NAS in fstab but not mounted.** Fix that first
+   (`sudo mount /mnt/nas`), and make it non-blocking for boot by adding `nofail`
+   and a timeout to the fstab line (see phase 1).
+
+Gate: binary rebuilt and staged, NAS mounted on every participating node.
+
+### Phase 1 — make the NAS mount boot-proof
+
+The existing line (all nodes) is:
+
+```
+nas.lan:/sna  /mnt/nas  nfs4  rw,hard,rsize=1048576,wsize=1048576,timeo=600,retrans=5,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0
+```
+
+Two problems for this plan: a NAS hiccup blocks boot, and `hard` + long `timeo`
+means a stale mount hangs reads (this NAS has already produced a
+`server not responding` outage that needed reboots).
+
+Change to:
+
+```
+nas.lan:/sna  /mnt/nas  nfs4  rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0
+```
+
+```bash
+sudo cp /etc/fstab /etc/fstab.bak-$(date +%F)
+sudo sed -i 's|^\(nas.lan:/sna.*\)$|# replaced by the storage plan\nnas.lan:/sna  /mnt/nas  nfs4  rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0|' /etc/fstab
+sudo systemctl daemon-reload
+sudo umount /mnt/nas 2>/dev/null; sudo mount /mnt/nas
+findmnt -no TARGET,OPTIONS /mnt/nas | grep -o 'nofail'
+```
+
+Gate: mount present, `nofail` in effect, a `sudo mount -a` is a no-op.
+
+### Phase 2 — package store (the 477 GiB) to NAS
+
+Low risk, no bind mounts, no FLARE code path surprises.
+
+```bash
+sudo mkdir -p /mnt/nas/ai/flare/store
+sudo chown kilork:kilork /mnt/nas/ai/flare/store
+# rsync -H is mandatory: format-2 packages are hardlink stores
+rsync -aH --info=progress2 ~/.cache/flare/packages/ /mnt/nas/ai/flare/store/
+```
+
+Point the CLI at it, in `~/.config/flare/config.toml` on the node that downloads
+(rank 0 / controller):
+
+```toml
+[paths]
+package_cache = "/mnt/nas/ai/flare/store"
+```
+
+Then remove the local copy only after a verified download check:
+
+```bash
+flare models list --model-cache /mnt/nas/ai/flare/store   # must list the version
+rm -rf ~/.cache/flare/packages
+```
+
+Gate: `flare models list` sees the version on NAS, `df -h /` shows the space back,
+and the next `orbit run` still starts (see phase 4 gate for the full test).
+
+**Open item to verify, not assume:** ranks are launched with an explicit
+`--package-cache /home/kilork/.cache/flare` that does **not** come from
+`[paths]` (the agent synthesises it). Check the launched cmdline after a start:
+
+```bash
+pgrep -af "[f]lare run" | head -1 | tr ' ' '\n' | grep -A1 -e package-cache -e model-cache
+```
+
+If that path needs to change, it is a **launcher** change
+(`[flare].extra_flags` / `[flare.clusters.<name>].extra_flags` in the sm121
+config, `src/native/flare_orbit.rs::extra_flags_for`) — never a hand edit on a
+node.
+
+### Phase 3 — validate the bind-mount model on an idle node (gate)
+
+Use **dgx4**: it is idle (no ranks), and now boots from the new MP600 Mini.
+
+1. Copy one model's small artefacts from a running node:
+   ```bash
+   ssh dgx8.lan 'tar -C ~/.cache/flare/models/<model>/<version> -cf - model.safetensors deployments' \
+     | tar -C /tmp -xf -
+   ```
+2. Build a fake local model dir whose `weights/` is a symlink to nothing and whose
+   `deployments/` is a **bind mount from the NAS copy**:
+   ```bash
+   mkdir -p ~/.cache/flare/models/<model>/<version>/weights
+   mv /tmp/model.safetensors /tmp/deployments ~/.cache/flare/models/<model>/<version>/
+   sudo mkdir -p /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>
+   sudo mv ~/.cache/flare/models/<model>/<version>/deployments /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>/
+   sudo mount --bind /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>/deployments \
+        ~/.cache/flare/models/<model>/<version>/deployments
+   ```
+3. Ask FLARE what it sees:
+   ```bash
+   flare inspect ~/.cache/flare/models/<model>/<version>/model.safetensors | head -20
+   flare models list
+   ```
+4. If a stage/launch is possible on dgx4 (`flare run --stage-only` with a dummy
+   agent), run it and then check the mount survived:
+   ```bash
+   findmnt -no TARGET ~/.cache/flare/models/<model>/<version>/deployments
+   ```
+   **Empty output = FLARE replaced the directory → take option D.**
+
+Gate: `flare inspect` shows the deployment, and `findmnt` still reports the bind
+mount after any stage operation.
+
+### Phase 4 — roll out to sparks (window; ranks stop)
+
+Repeat per node in rank order, one node at a time, verifying before moving on.
+
+1. Stop the group (needs a working controller; see the re-pair note in
+   `flare-bringup.md`):
+   ```bash
+   flare orbit stop <group> --cluster sparks
+   ```
+2. Move the node's small artefacts to NAS and bind them back:
+   ```bash
+   V=gb10-tp4-<build>; M=checkpoint-expanded-scales
+   D=/home/kilork/.cache/flare/models/$M/$V
+   N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V
+   sudo mkdir -p "$N"
+   rsync -aH --remove-source-files "$D/model.safetensors" "$N/"
+   rsync -aH "$D/deployments/" "$N/deployments/"
+   rm -rf "$D/deployments"
+   sudo mount --bind "$N/deployments" "$D/deployments"
+   ```
+   Persist it in `/etc/fstab` (bind mounts are ordinary fstab entries and are
+   mounted before the agent starts):
+   ```
+   /mnt/nas/ai/flare/nodes/dgx5/checkpoint-expanded-scales/gb10-tp4-<build>/deployments  /home/kilork/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-<build>/deployments  none  bind,nofail,x-systemd.requires=/mnt/nas,x-systemd.requires-mounts-for=/mnt/nas  0 0
+   ```
+3. Start the group and verify on the node:
+   ```bash
+   findmnt -no TARGET,SOURCE ~/.cache/flare/models/$M/$V/deployments
+   du -sh ~/.cache/flare/models/$M/$V/* | sort -h   # weights ~260G, rest tiny
+   pgrep -af "[f]lare run" | head -1
+   ```
+4. Only then move to the next node.
+
+Gate: ranks serve, `findmnt` shows the bind mounts, `du` shows ~260 GiB of
+`weights/` and nothing else of size.
+
+### Phase 5 — prod4 (dgx1–dgx4), same steps
+
+Same procedure with `--cluster prod4`. dgx1 is rank 0 and is the one that gains the
+477 GiB; it is also the node whose `packages/` phase 2 removed.
+
+### Phase 6 — verification and rollback
+
+Verification, per node:
+
+```bash
+du -sh ~/.cache/flare/models/*/*/*          # weights dominates
+findmnt -no TARGET,SOURCE | grep deployments
+df -h / /mnt/nas
+flare orbit status --cluster sparks         # agents reachable, groups listed
+curl -s localhost:<port>/v1/models | head  # API answers (port from the rank cmdline)
+```
+
+Rollback (per node, no data loss — the NAS copy is authoritative):
+
+```bash
+sudo umount ~/.cache/flare/models/$M/$V/deployments
+sudo sed -i "\|$N/deployments|d" /etc/fstab
+rsync -aH /mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V/deployments/ ~/.cache/flare/models/$M/$V/deployments/
+flare orbit run <group> --cluster sparks
+```
+
+## 5. Risks and their fallbacks
+
+| risk | detection | fallback |
+|---|---|---|
+| FLARE replaces `deployments/` by rename; bind mount lost | `findmnt` empty after a stage | **Option D**: deployments local, only the package store moves |
+| NAS stall at boot | node boots with `/mnt/nas` missing | `nofail` (phase 1) keeps boot going; agent then writes deployments locally — re-sync after recovery |
+| NAS outage during serving | reads hang (`hard` mount) | bind mounts are only touched at load; keep `timeo=100,retrans=3`; do not put `staging/` on NAS |
+| hardlink-blind copy duplicates the package | `du` doubles after a copy | always `rsync -aH` |
+| mixed flare binaries after a restart | rendezvous `incompatible protocol` | phase 0 rebuild first, restart all ranks together |
+
+## 6. What must be measured, not assumed
+
+1. Whether the agent's `--package-cache` path can be redirected by config or needs
+   a launcher `extra_flags` change (phase 2 note).
+2. Whether FLARE tolerates the bind mount across a real stage (phase 3 gate).
+3. Real NAS read throughput for `deployments/` at load — 1.8 GiB over NFS at
+   1 GbE is ~15 s per rank; measure before rolling to prod, and keep `weights/`
+   local regardless.