Goal, in the operator's words: only the expanded checkpoint stays on the sparks
NVMe (260 GiB for DeepSeek-V4.1-Flash tp4) and all other model artefacts live
on the NAS. This document is written to be executed by someone with no context
beyond it. Every command is copy-pasteable and every phase has a gate.
nofail,
timeo=100, x-systemd.mount-timeout=30) — verified live, serving untouchedrsync -aH the 477 GiB store to the NAS in
the background, then verify with flare models list --model-cache[paths] package_cache at the NAS,
remove the local store, confirm the launched rank cmdline (open item 6.1)Run from the controller; every node must report fstab=1 live=1:
for h in dgx1 dgx2 dgx3 dgx4 dgx5 dgx6 dgx7 dgx8; do printf "%-6s " $h
ssh -o BatchMode=yes -o ConnectTimeout=5 $h.lan 'printf "%s fstab=%s live=%s ranks=%s\n" \
"$(hostname -s)" "$(grep /mnt/nas /etc/fstab | grep -c nofail)" \
"$(findmnt -no OPTIONS /mnt/nas | grep -c "timeo=100")" "$(pgrep -fc "[f]lare run")"' \
2>/dev/null || echo "off"; done
| node | role | models/ |
packages/ |
rank cmdline cache |
|---|---|---|---|---|
| dgx8 | sparks rank 3 | 261 GiB | — | --package-cache /home/kilork/.cache/flare |
| dgx1 | prod4 rank 0 | 261 GiB | 477 GiB | --package-cache /home/kilork/.cache/flare |
Anatomy of one model directory — this split is the whole plan:
~/.cache/flare/models/checkpoint-expanded-scales/<version>/
model.safetensors 35 MiB signed envelope (manifest + locations)
deployments/<pkg>/rank-<N>/ 788 KiB (rank 0) … 1.8 GiB (rank 1-3)
weights/ 260 GiB ← the expanded checkpoint
Package store (CLI side, rank 0 only):
~/.cache/flare/packages/<hash>-<rev>/
model.safetensors 35 MiB
deployments/ 7.2 GiB
weights/ 470 GiB ← second copy of the same content
NAS: /mnt/nas = 33 TiB, 1.1 TiB used, /mnt/nas/ai/flare effectively empty.
weights/ ever lands on NVMe.If the intent is "NVMe holds one checkpoint and nothing else", this plan delivers it. If the intent was primarily disk relief on sparks, the win is small and the cheap part (phase 2, the package store) may be enough.
# every rank node, local NVMe
/home/kilork/.cache/flare/models/<model>/<version>/
weights/ real directory 260 GiB
model.safetensors bind mount → NAS/nodes/<node>/<version>/model.safetensors
deployments/ bind mount → NAS/nodes/<node>/<version>/deployments
# NAS
/mnt/nas/ai/flare/
store/ paths.package_cache — CLI store, packages/, blobs/, staging/,
refs/ (rank 0's 477 GiB lives here, once)
nodes/<node>/<model>/<version>/{model.safetensors,deployments/}
per-node small artefacts (bind targets)
Only weights/ is allowed on node NVMe. Everything else resolves through the
bind mounts, so FLARE's view of a model directory is unchanged.
FLARE's own error strings include "a package primary escapes its directory" and "staged entry is not a regular file" — a symlink pointing outside the model directory is exactly what those checks reject. Bind mounts are invisible to the process: the path stays inside the directory and the entry is a real file. Do not substitute symlinks.
Known risk: if FLARE ever replaces deployments/ by rename (rm -rf +
mv), the mount point disappears and writes fall back to local disk. Phase 3 is a
gate that measures this; if it happens, fall back to option D: leave
deployments/ local (1.8 GiB) and move only the package store. Do not fight it.
Each phase is independently revertible. Do not start a phase whose gate the previous phase failed.
0.1.0~git20261105002735.0009b89d-1), which lacks PR #184. The
running agents still hold the PR-184 inode; restarting without the rebuild
drops the io_uring Engram reader on dgx1/2/3/8. See references/build.md in
the flare skill, or flare-upstream.pin + pr184-on-main here. Verify:
strings <binary> | grep -c "embedding read worker panicked" → 1.Confirm NAS health and space, from every node that will use it:
ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
Record the current state of each node for rollback — done 2026-11-11, files
are on each node and in /mnt/nas/ai/flare/ (flare-before-gx10-dgxN.txt).
Hostnames are gx10-dgxN, not the dgxN.lan aliases — build filenames from
hostname -s and do not assume the alias resolves to a filename:
du -sh ~/.cache/flare/* | sort -h > /tmp/flare-before-$(hostname -s).txt
findmnt -no TARGET,SOURCE,OPTIONS /mnt/nas
awk 'NR>1' /proc/fs/nfsfs/servers | sort -u | wc -l # NFS client IDs
tr '\0' ' ' < /proc/$(pgrep -f "[f]lare run" | head -1)/cmdline # exact launch command
The captured rank cmdline is the recovery recipe if a node has to be restarted
outside orbit (the controller is currently unpaired), e.g. rank 0 on dgx1:
flare run ~/.cache/flare/packages/7cf40bec…-35842073/model.safetensors
--package-cache ~/.cache/flare --rank 0 --rendezvous 0.0.0.0:29800
--max-concurrency 32 --mtp-draft-tokens 5 --mtp-adaptive-verification
--embedding-table-storage disk --embedding-table-cache-size 1073741824
--embedding-table-resident-scales --transfer-streams 1 --host 0.0.0.0 --port 30000
--served-model-name deepseek-v4.1-flash --publisher-keys … --dangerously-approve-key …
--dangerously-accept-publisher local-inference-lab --gpus 0
dgx8 has the NAS in fstab but not mounted fixed 2026-11-11 — mounted at
vers=4.1 and made non-blocking with the phase 1 options.
Gate: binary rebuilt and staged, NAS mounted on every participating node.
The existing line (all nodes) is:
nas.lan:/sna /mnt/nas nfs4 rw,hard,rsize=1048576,wsize=1048576,timeo=600,retrans=5,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0
Two problems for this plan: a NAS hiccup blocks boot, and hard + long timeo
means a stale mount hangs reads (this NAS has already produced a
server not responding outage that needed reboots).
Applied everywhere, with one per-node exception (see below):
nas.lan:/sna /mnt/nas nfs4 rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0
Do not use a blanket sed over the nas.lan:/sna line — it silently rewrites
the protocol version too. Rewrite the entry by its mount-point field and pass the
protocol per node — use host-scripts/set-nas-mount-opts.sh <vers> [remount]
(proven idempotent on dgx4 on 2026-11-11; it backs up fstab, rewrites the entry, and
plain-umounts before mounting). For reference, it is equivalent to:
sudo cp -a /etc/fstab /etc/fstab.bak-$(date +%F-%H%M)
sudo python3 - "$VERS" <<'PY'
import sys
vers = sys.argv[1]
path, out, hit = "/etc/fstab", [], False
for line in open(path).read().splitlines(True):
f = line.split()
if len(f) >= 3 and f[1] == "/mnt/nas":
opts = ("rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,"
"timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=" + vers)
out.append("nas.lan:/sna\t/mnt/nas\tnfs4\t%s\t0\t0\n" % opts); hit = True
else:
out.append(line)
assert hit, "no /mnt/nas entry in fstab"
open(path, "w").write("".join(out))
PY
sudo systemctl daemon-reload
sudo umount /mnt/nas && sudo mount /mnt/nas # umount fails safely if something holds it
findmnt -no OPTIONS /mnt/nas | grep -o 'timeo=100'
nofail is an fstab-only option: it never appears in live mount options, so verify
the live change with timeo=100, not with nofail.
dgx1 exception — applied and verified 2026-11-11. dgx1 cannot mount NFSv4.1 at
all: a fresh vers=4.1 mount hangs even with a single clean client and with
nconnect=1, while dgx2/3/4/8 mount 4.1 instantly. Its fstab is therefore pinned
to vers=4.0 with the same other options. Its two stale client IDs were cleared
without a reboot (umount, not umount -l, with zero open files — that is the
procedure; a reboot was not needed).
Gate: mount present, nofail in effect, a sudo mount -a is a no-op. Passed on
dgx1/2/3/4/8 (2026-11-11): live options show timeo=100, mount -a returns 0
without remounting, all rank PIDs kept their 7-day uptime and the API on
dgx1:30000 still answered. dgx5–7 are off and still need this.
Low risk, no bind mounts, no FLARE code path surprises.
sudo mkdir -p /mnt/nas/ai/flare/store
sudo chown kilork:kilork /mnt/nas/ai/flare/store
# rsync -H is mandatory: format-2 packages are hardlink stores
rsync -aH --info=progress2 ~/.cache/flare/packages/ /mnt/nas/ai/flare/store/
Point the CLI at it, in ~/.config/flare/config.toml on the node that downloads
(rank 0 / controller):
[paths]
package_cache = "/mnt/nas/ai/flare/store"
Then remove the local copy only after a verified download check:
flare models list --model-cache /mnt/nas/ai/flare/store # must list the version
rm -rf ~/.cache/flare/packages
Gate: flare models list sees the version on NAS, df -h / shows the space back,
and the next orbit run still starts (see phase 4 gate for the full test).
Open item to verify, not assume: ranks are launched with an explicit
--package-cache /home/kilork/.cache/flare that does not come from
[paths] (the agent synthesises it). Check the launched cmdline after a start:
pgrep -af "[f]lare run" | head -1 | tr ' ' '\n' | grep -A1 -e package-cache -e model-cache
If that path needs to change, it is a launcher change
([flare].extra_flags / [flare.clusters.<name>].extra_flags in the sm121
config, src/native/flare_orbit.rs::extra_flags_for) — never a hand edit on a
node.
Use dgx4: it is idle (no ranks), and now boots from the new MP600 Mini.
Copy one model's small artefacts from a running node:
ssh dgx8.lan 'tar -C ~/.cache/flare/models/<model>/<version> -cf - model.safetensors deployments' \
| tar -C /tmp -xf -
Build a fake local model dir whose weights/ is a symlink to nothing and whose
deployments/ is a bind mount from the NAS copy:
mkdir -p ~/.cache/flare/models/<model>/<version>/weights
mv /tmp/model.safetensors /tmp/deployments ~/.cache/flare/models/<model>/<version>/
sudo mkdir -p /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>
sudo mv ~/.cache/flare/models/<model>/<version>/deployments /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>/
sudo mount --bind /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>/deployments \
~/.cache/flare/models/<model>/<version>/deployments
Ask FLARE what it sees:
flare inspect ~/.cache/flare/models/<model>/<version>/model.safetensors | head -20
flare models list
If a stage/launch is possible on dgx4 (flare run --stage-only with a dummy
agent), run it and then check the mount survived:
findmnt -no TARGET ~/.cache/flare/models/<model>/<version>/deployments
Empty output = FLARE replaced the directory → take option D.
Gate: flare inspect shows the deployment, and findmnt still reports the bind
mount after any stage operation.
Repeat per node in rank order, one node at a time, verifying before moving on.
Stop the group (needs a working controller; see the re-pair note in
flare-bringup.md):
flare orbit stop <group> --cluster sparks
Move the node's small artefacts to NAS and bind them back:
V=gb10-tp4-<build>; M=checkpoint-expanded-scales
D=/home/kilork/.cache/flare/models/$M/$V
N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V
sudo mkdir -p "$N"
rsync -aH --remove-source-files "$D/model.safetensors" "$N/"
rsync -aH "$D/deployments/" "$N/deployments/"
rm -rf "$D/deployments"
sudo mount --bind "$N/deployments" "$D/deployments"
Persist it in /etc/fstab (bind mounts are ordinary fstab entries and are
mounted before the agent starts):
/mnt/nas/ai/flare/nodes/dgx5/checkpoint-expanded-scales/gb10-tp4-<build>/deployments /home/kilork/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-<build>/deployments none bind,nofail,x-systemd.requires=/mnt/nas,x-systemd.requires-mounts-for=/mnt/nas 0 0
Start the group and verify on the node:
findmnt -no TARGET,SOURCE ~/.cache/flare/models/$M/$V/deployments
du -sh ~/.cache/flare/models/$M/$V/* | sort -h # weights ~260G, rest tiny
pgrep -af "[f]lare run" | head -1
Only then move to the next node.
Gate: ranks serve, findmnt shows the bind mounts, du shows ~260 GiB of
weights/ and nothing else of size.
Same procedure with --cluster prod4. dgx1 is rank 0 and is the one that gains the
477 GiB; it is also the node whose packages/ phase 2 removed.
Verification, per node:
du -sh ~/.cache/flare/models/*/*/* # weights dominates
findmnt -no TARGET,SOURCE | grep deployments
df -h / /mnt/nas
flare orbit status --cluster sparks # agents reachable, groups listed
curl -s localhost:<port>/v1/models | head # API answers (port from the rank cmdline)
Rollback (per node, no data loss — the NAS copy is authoritative):
sudo umount ~/.cache/flare/models/$M/$V/deployments
sudo sed -i "\|$N/deployments|d" /etc/fstab
rsync -aH /mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V/deployments/ ~/.cache/flare/models/$M/$V/deployments/
flare orbit run <group> --cluster sparks
| risk | detection | fallback |
|---|---|---|
FLARE replaces deployments/ by rename; bind mount lost |
findmnt empty after a stage |
Option D: deployments local, only the package store moves |
| NAS stall at boot | node boots with /mnt/nas missing |
nofail (phase 1) keeps boot going; agent then writes deployments locally — re-sync after recovery |
| NAS outage during serving | reads hang (hard mount) |
bind mounts are only touched at load; keep timeo=100,retrans=3; do not put staging/ on NAS |
| hardlink-blind copy duplicates the package | du doubles after a copy |
always rsync -aH |
| mixed flare binaries after a restart | rendezvous incompatible protocol |
phase 0 rebuild first, restart all ranks together |
--package-cache path can be redirected by config or needs
a launcher extra_flags change (phase 2 note).dd iflag=direct, 2 GiB,
so no page-cache lying): the NAS is 10 GbE, not 1 GbE.| node | write 2 GiB | read 1 MiB | read 8 KiB | |---|---|---|---| | dgx8 (v4.1) | 659 MiB/s | 731 MiB/s | 28 MiB/s | | dgx1 (v4.0) | 663 MiB/s | 822 MiB/s | 32 MiB/s |
So a 1.8 GiB deployments/ directory reads in 2–3 s, and the protocol
choice costs nothing. The real constraint is small-file/short-read work: at
8 KiB the same share runs at ~30 MiB/s, so a deployments/ split into many
small files would load ~25× slower than the bulk figure suggests — count the
files in a rank deployment before trusting the 2–3 s number, and keep weights/
local regardless.