# FLARE storage: expanded checkpoint on node NVMe, everything else on NAS (2026-11-10) Goal, in the operator's words: **only the expanded checkpoint stays on the sparks NVMe** (260 GiB for `DeepSeek-V4.1-Flash` tp4) and **all other model artefacts live on the NAS**. This document is written to be executed by someone with no context beyond it. Every command is copy-pasteable and every phase has a gate. ## Milestones - [x] NAS mount made boot-proof on every participating node (`nofail`, `timeo=100`, `x-systemd.mount-timeout=30`) — verified live, serving untouched - [x] dgx8's missing NAS mount restored (it was in fstab but never mounted) - [x] dgx1's stale NFS state cleared without a reboot; node pinned to the protocol that works there (4.0) — see the correction in Phase 1 - [x] Pre-migration state of dgx1/2/3/4/8 captured locally and on the NAS - [x] NAS throughput measured (was open item 6.3) - [ ] PR #184 binary rebuilt and staged (Phase 0.1) — needs dgx5 powered on - [ ] Package store (477 GiB) on the NAS, local copy removed, config pointed at it (Phase 2) - [ ] Bind-mount layout validated on an idle node (Phase 3 gate) - [ ] Sparks rolled out (Phase 4, needs a window) - [ ] prod4 rolled out (Phase 5, needs a window) - [ ] Verification matrix run and rollback drilled (Phase 6) ## Round plan - [x] **R1 (Q1)** — Phase 0.2–0.4 + Phase 1 on dgx1/2/3/4/8 - [ ] **R2 (Q1)** — Phase 2 first half: `rsync -aH` the 477 GiB store to the NAS in the background, then verify with `flare models list --model-cache` - [ ] **R3 (Q1)** — Phase 2 second half: point `[paths] package_cache` at the NAS, remove the local store, confirm the launched rank cmdline (open item 6.1) - [ ] **R4 (Q1)** — PR #184 rebuild and staging (needs dgx5 on; prerequisite for any restart) - [ ] **R5 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4 - [ ] **R6 (Q2)** — Phase 4 sparks rollout (window; ranks stop) - [ ] **R7 (Q2)** — Phase 5 prod4 rollout (window; ranks stop) - [ ] **R8 (Q3)** — Phase 6 verification matrix and rollback drill ## Checks Run from the controller; every node must report `fstab=1 live=1`: ```bash for h in dgx1 dgx2 dgx3 dgx4 dgx5 dgx6 dgx7 dgx8; do printf "%-6s " $h ssh -o BatchMode=yes -o ConnectTimeout=5 $h.lan 'printf "%s fstab=%s live=%s ranks=%s\n" \ "$(hostname -s)" "$(grep /mnt/nas /etc/fstab | grep -c nofail)" \ "$(findmnt -no OPTIONS /mnt/nas | grep -c "timeo=100")" "$(pgrep -fc "[f]lare run")"' \ 2>/dev/null || echo "off"; done ``` ## 1. Measured baseline (2026-11-10, read-only) | node | role | `models/` | `packages/` | rank cmdline cache | |---|---|---|---|---| | dgx8 | sparks rank 3 | 261 GiB | — | `--package-cache /home/kilork/.cache/flare` | | dgx1 | prod4 rank 0 | 261 GiB | 477 GiB | `--package-cache /home/kilork/.cache/flare` | Anatomy of one model directory — **this split is the whole plan**: ``` ~/.cache/flare/models/checkpoint-expanded-scales// model.safetensors 35 MiB signed envelope (manifest + locations) deployments//rank-/ 788 KiB (rank 0) … 1.8 GiB (rank 1-3) weights/ 260 GiB ← the expanded checkpoint ``` Package store (CLI side, rank 0 only): ``` ~/.cache/flare/packages/-/ model.safetensors 35 MiB deployments/ 7.2 GiB weights/ 470 GiB ← second copy of the same content ``` NAS: `/mnt/nas` = 33 TiB, 1.1 TiB used, `/mnt/nas/ai/flare` effectively empty. ### What the move actually saves — read before committing - sparks ranks 1–3: **1.8 GiB each** (261 → 260 GiB). Small. - rank 0 (prod4): **477 GiB** (the duplicate package store) plus 7.2 GiB of deployments. - Future model versions stop multiplying: only their `weights/` ever lands on NVMe. If the intent is "NVMe holds one checkpoint and nothing else", this plan delivers it. If the intent was primarily disk relief on sparks, the win is small and the cheap part (phase 2, the package store) may be enough. ## 2. Target layout ``` # every rank node, local NVMe /home/kilork/.cache/flare/models/// weights/ real directory 260 GiB model.safetensors bind mount → NAS/nodes///model.safetensors deployments/ bind mount → NAS/nodes///deployments # NAS /mnt/nas/ai/flare/ store/ paths.package_cache — CLI store, packages/, blobs/, staging/, refs/ (rank 0's 477 GiB lives here, once) nodes////{model.safetensors,deployments/} per-node small artefacts (bind targets) ``` Only `weights/` is allowed on node NVMe. Everything else resolves through the bind mounts, so FLARE's view of a model directory is unchanged. ## 3. Why bind mounts and not symlinks FLARE's own error strings include *"a package primary escapes its directory"* and *"staged entry is not a regular file"* — a symlink pointing outside the model directory is exactly what those checks reject. Bind mounts are invisible to the process: the path stays inside the directory and the entry is a real file. Do not substitute symlinks. **Known risk:** if FLARE ever replaces `deployments/` by rename (`rm -rf` + `mv`), the mount point disappears and writes fall back to local disk. Phase 3 is a gate that measures this; if it happens, fall back to **option D**: leave `deployments/` local (1.8 GiB) and move only the package store. Do not fight it. ## 4. Phases Each phase is independently revertible. Do not start a phase whose gate the previous phase failed. ### Phase 0 — prerequisites (no FLARE restart yet) 1. **Rebuild the runtime before any agent restart.** The installed binary is the PPA build (`0.1.0~git20261105002735.0009b89d-1`), which lacks PR #184. The running agents still hold the PR-184 inode; restarting without the rebuild drops the io_uring Engram reader on dgx1/2/3/8. See `references/build.md` in the `flare` skill, or `flare-upstream.pin` + `pr184-on-main` here. Verify: `strings | grep -c "embedding read worker panicked"` → `1`. 2. Confirm NAS health and space, from every node that will use it: ```bash ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1 ``` 3. Record the current state of each node for rollback — **done 2026-11-11**, files are on each node and in `/mnt/nas/ai/flare/` (`flare-before-gx10-dgxN.txt`). Hostnames are `gx10-dgxN`, **not** the `dgxN.lan` aliases — build filenames from `hostname -s` and do not assume the alias resolves to a filename: ```bash du -sh ~/.cache/flare/* | sort -h > /tmp/flare-before-$(hostname -s).txt findmnt -no TARGET,SOURCE,OPTIONS /mnt/nas awk 'NR>1' /proc/fs/nfsfs/servers | sort -u | wc -l # NFS client IDs tr '\0' ' ' < /proc/$(pgrep -f "[f]lare run" | head -1)/cmdline # exact launch command ``` The captured rank cmdline is the recovery recipe if a node has to be restarted outside `orbit` (the controller is currently unpaired), e.g. rank 0 on dgx1: `flare run ~/.cache/flare/packages/7cf40bec…-35842073/model.safetensors --package-cache ~/.cache/flare --rank 0 --rendezvous 0.0.0.0:29800 --max-concurrency 32 --mtp-draft-tokens 5 --mtp-adaptive-verification --embedding-table-storage disk --embedding-table-cache-size 1073741824 --embedding-table-resident-scales --transfer-streams 1 --host 0.0.0.0 --port 30000 --served-model-name deepseek-v4.1-flash --publisher-keys … --dangerously-approve-key … --dangerously-accept-publisher local-inference-lab --gpus 0` 4. ~~dgx8 has the NAS in fstab but not mounted~~ **fixed 2026-11-11** — mounted at `vers=4.1` and made non-blocking with the phase 1 options. Gate: binary rebuilt and staged, NAS mounted on every participating node. ### Phase 1 — make the NAS mount boot-proof The existing line (all nodes) is: ``` nas.lan:/sna /mnt/nas nfs4 rw,hard,rsize=1048576,wsize=1048576,timeo=600,retrans=5,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0 ``` Two problems for this plan: a NAS hiccup blocks boot, and `hard` + long `timeo` means a stale mount hangs reads (this NAS has already produced a `server not responding` outage that needed reboots). Applied everywhere, **with one per-node exception** (see below): ``` nas.lan:/sna /mnt/nas nfs4 rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0 ``` Do **not** use a blanket `sed` over the `nas.lan:/sna` line — it silently rewrites the protocol version too. Rewrite the entry by its mount-point field and pass the protocol per node — use **`host-scripts/set-nas-mount-opts.sh [remount]`** (proven idempotent on dgx4 on 2026-11-11; it backs up fstab, rewrites the entry, and plain-`umount`s before mounting). For reference, it is equivalent to: ```bash sudo cp -a /etc/fstab /etc/fstab.bak-$(date +%F-%H%M) sudo python3 - "$VERS" <<'PY' import sys vers = sys.argv[1] path, out, hit = "/etc/fstab", [], False for line in open(path).read().splitlines(True): f = line.split() if len(f) >= 3 and f[1] == "/mnt/nas": opts = ("rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576," "timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=" + vers) out.append("nas.lan:/sna\t/mnt/nas\tnfs4\t%s\t0\t0\n" % opts); hit = True else: out.append(line) assert hit, "no /mnt/nas entry in fstab" open(path, "w").write("".join(out)) PY sudo systemctl daemon-reload sudo umount /mnt/nas && sudo mount /mnt/nas # umount fails safely if something holds it findmnt -no OPTIONS /mnt/nas | grep -o 'timeo=100' ``` `nofail` is an fstab-only option: it never appears in live mount options, so verify the *live* change with `timeo=100`, not with `nofail`. **dgx1 exception — applied and verified 2026-11-11.** dgx1 cannot mount NFSv4.1 at all: a fresh `vers=4.1` mount hangs even with a single clean client and with `nconnect=1`, while dgx2/3/4/8 mount 4.1 instantly. Its fstab is therefore pinned to `vers=4.0` with the same other options. Its two stale client IDs were cleared without a reboot (`umount`, not `umount -l`, with zero open files — that is the procedure; a reboot was not needed). Gate: mount present, `nofail` in effect, a `sudo mount -a` is a no-op. **Passed on dgx1/2/3/4/8 (2026-11-11):** live options show `timeo=100`, `mount -a` returns 0 without remounting, all rank PIDs kept their 7-day uptime and the API on `dgx1:30000` still answered. dgx5–7 are off and still need this. ### Phase 2 — package store (the 477 GiB) to NAS Low risk, no bind mounts, no FLARE code path surprises. ```bash sudo mkdir -p /mnt/nas/ai/flare/store sudo chown kilork:kilork /mnt/nas/ai/flare/store # rsync -H is mandatory: format-2 packages are hardlink stores rsync -aH --info=progress2 ~/.cache/flare/packages/ /mnt/nas/ai/flare/store/ ``` Point the CLI at it, in `~/.config/flare/config.toml` on the node that downloads (rank 0 / controller): ```toml [paths] package_cache = "/mnt/nas/ai/flare/store" ``` Then remove the local copy only after a verified download check: ```bash flare models list --model-cache /mnt/nas/ai/flare/store # must list the version rm -rf ~/.cache/flare/packages ``` Gate: `flare models list` sees the version on NAS, `df -h /` shows the space back, and the next `orbit run` still starts (see phase 4 gate for the full test). **Open item to verify, not assume:** ranks are launched with an explicit `--package-cache /home/kilork/.cache/flare` that does **not** come from `[paths]` (the agent synthesises it). Check the launched cmdline after a start: ```bash pgrep -af "[f]lare run" | head -1 | tr ' ' '\n' | grep -A1 -e package-cache -e model-cache ``` If that path needs to change, it is a **launcher** change (`[flare].extra_flags` / `[flare.clusters.].extra_flags` in the sm121 config, `src/native/flare_orbit.rs::extra_flags_for`) — never a hand edit on a node. ### Phase 3 — validate the bind-mount model on an idle node (gate) Use **dgx4**: it is idle (no ranks), and now boots from the new MP600 Mini. 1. Copy one model's small artefacts from a running node: ```bash ssh dgx8.lan 'tar -C ~/.cache/flare/models// -cf - model.safetensors deployments' \ | tar -C /tmp -xf - ``` 2. Build a fake local model dir whose `weights/` is a symlink to nothing and whose `deployments/` is a **bind mount from the NAS copy**: ```bash mkdir -p ~/.cache/flare/models///weights mv /tmp/model.safetensors /tmp/deployments ~/.cache/flare/models/// sudo mkdir -p /mnt/nas/ai/flare/nodes/dgx4// sudo mv ~/.cache/flare/models///deployments /mnt/nas/ai/flare/nodes/dgx4/// sudo mount --bind /mnt/nas/ai/flare/nodes/dgx4///deployments \ ~/.cache/flare/models///deployments ``` 3. Ask FLARE what it sees: ```bash flare inspect ~/.cache/flare/models///model.safetensors | head -20 flare models list ``` 4. If a stage/launch is possible on dgx4 (`flare run --stage-only` with a dummy agent), run it and then check the mount survived: ```bash findmnt -no TARGET ~/.cache/flare/models///deployments ``` **Empty output = FLARE replaced the directory → take option D.** Gate: `flare inspect` shows the deployment, and `findmnt` still reports the bind mount after any stage operation. ### Phase 4 — roll out to sparks (window; ranks stop) Repeat per node in rank order, one node at a time, verifying before moving on. 1. Stop the group (needs a working controller; see the re-pair note in `flare-bringup.md`): ```bash flare orbit stop --cluster sparks ``` 2. Move the node's small artefacts to NAS and bind them back: ```bash V=gb10-tp4-; M=checkpoint-expanded-scales D=/home/kilork/.cache/flare/models/$M/$V N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V sudo mkdir -p "$N" rsync -aH --remove-source-files "$D/model.safetensors" "$N/" rsync -aH "$D/deployments/" "$N/deployments/" rm -rf "$D/deployments" sudo mount --bind "$N/deployments" "$D/deployments" ``` Persist it in `/etc/fstab` (bind mounts are ordinary fstab entries and are mounted before the agent starts): ``` /mnt/nas/ai/flare/nodes/dgx5/checkpoint-expanded-scales/gb10-tp4-/deployments /home/kilork/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-/deployments none bind,nofail,x-systemd.requires=/mnt/nas,x-systemd.requires-mounts-for=/mnt/nas 0 0 ``` 3. Start the group and verify on the node: ```bash findmnt -no TARGET,SOURCE ~/.cache/flare/models/$M/$V/deployments du -sh ~/.cache/flare/models/$M/$V/* | sort -h # weights ~260G, rest tiny pgrep -af "[f]lare run" | head -1 ``` 4. Only then move to the next node. Gate: ranks serve, `findmnt` shows the bind mounts, `du` shows ~260 GiB of `weights/` and nothing else of size. ### Phase 5 — prod4 (dgx1–dgx4), same steps Same procedure with `--cluster prod4`. dgx1 is rank 0 and is the one that gains the 477 GiB; it is also the node whose `packages/` phase 2 removed. ### Phase 6 — verification and rollback Verification, per node: ```bash du -sh ~/.cache/flare/models/*/*/* # weights dominates findmnt -no TARGET,SOURCE | grep deployments df -h / /mnt/nas flare orbit status --cluster sparks # agents reachable, groups listed curl -s localhost:/v1/models | head # API answers (port from the rank cmdline) ``` Rollback (per node, no data loss — the NAS copy is authoritative): ```bash sudo umount ~/.cache/flare/models/$M/$V/deployments sudo sed -i "\|$N/deployments|d" /etc/fstab rsync -aH /mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V/deployments/ ~/.cache/flare/models/$M/$V/deployments/ flare orbit run --cluster sparks ``` ## 5. Risks and their fallbacks | risk | detection | fallback | |---|---|---| | FLARE replaces `deployments/` by rename; bind mount lost | `findmnt` empty after a stage | **Option D**: deployments local, only the package store moves | | NAS stall at boot | node boots with `/mnt/nas` missing | `nofail` (phase 1) keeps boot going; agent then writes deployments locally — re-sync after recovery | | NAS outage during serving | reads hang (`hard` mount) | bind mounts are only touched at load; keep `timeo=100,retrans=3`; do not put `staging/` on NAS | | hardlink-blind copy duplicates the package | `du` doubles after a copy | always `rsync -aH` | | mixed flare binaries after a restart | rendezvous `incompatible protocol` | phase 0 rebuild first, restart all ranks together | ## 6. What must be measured, not assumed 1. Whether the agent's `--package-cache` path can be redirected by config or needs a launcher `extra_flags` change (phase 2 note). 2. Whether FLARE tolerates the bind mount across a real stage (phase 3 gate). 3. ~~Real NAS read throughput~~ **measured 2026-11-11** (`dd iflag=direct`, 2 GiB, so no page-cache lying): the NAS is **10 GbE, not 1 GbE**. | node | write 2 GiB | read 1 MiB | read 8 KiB | |---|---|---|---| | dgx8 (v4.1) | 659 MiB/s | 731 MiB/s | 28 MiB/s | | dgx1 (v4.0) | 663 MiB/s | 822 MiB/s | 32 MiB/s | So a 1.8 GiB `deployments/` directory reads in **2–3 s**, and the protocol choice costs nothing. The real constraint is small-file/short-read work: at 8 KiB the same share runs at ~30 MiB/s, so a `deployments/` split into many small files would load ~25× slower than the bulk figure suggests — count the files in a rank deployment before trusting the 2–3 s number, and keep `weights/` local regardless.