flare-storage-nas.md 12 KB

FLARE storage: expanded checkpoint on node NVMe, everything else on NAS (2026-11-10)

Goal, in the operator's words: only the expanded checkpoint stays on the sparks NVMe (260 GiB for DeepSeek-V4.1-Flash tp4) and all other model artefacts live on the NAS. This document is written to be executed by someone with no context beyond it. Every command is copy-pasteable and every phase has a gate.

1. Measured baseline (2026-11-10, read-only)

node role models/ packages/ rank cmdline cache
dgx8 sparks rank 3 261 GiB — --package-cache /home/kilork/.cache/flare
dgx1 prod4 rank 0 261 GiB 477 GiB --package-cache /home/kilork/.cache/flare

Anatomy of one model directory — this split is the whole plan:

~/.cache/flare/models/checkpoint-expanded-scales/<version>/
  model.safetensors      35 MiB      signed envelope (manifest + locations)
  deployments/<pkg>/rank-<N>/  788 KiB (rank 0) … 1.8 GiB (rank 1-3)
  weights/              260 GiB      ← the expanded checkpoint

Package store (CLI side, rank 0 only):

~/.cache/flare/packages/<hash>-<rev>/
  model.safetensors      35 MiB
  deployments/          7.2 GiB
  weights/              470 GiB      ← second copy of the same content

NAS: /mnt/nas = 33 TiB, 1.1 TiB used, /mnt/nas/ai/flare effectively empty.

What the move actually saves — read before committing

  • sparks ranks 1–3: 1.8 GiB each (261 → 260 GiB). Small.
  • rank 0 (prod4): 477 GiB (the duplicate package store) plus 7.2 GiB of deployments.
  • Future model versions stop multiplying: only their weights/ ever lands on NVMe.

If the intent is "NVMe holds one checkpoint and nothing else", this plan delivers it. If the intent was primarily disk relief on sparks, the win is small and the cheap part (phase 2, the package store) may be enough.

2. Target layout

# every rank node, local NVMe
/home/kilork/.cache/flare/models/<model>/<version>/
    weights/              real directory                    260 GiB
    model.safetensors     bind mount → NAS/nodes/<node>/<version>/model.safetensors
    deployments/          bind mount → NAS/nodes/<node>/<version>/deployments

# NAS
/mnt/nas/ai/flare/
    store/                paths.package_cache — CLI store, packages/, blobs/, staging/,
                          refs/ (rank 0's 477 GiB lives here, once)
    nodes/<node>/<model>/<version>/{model.safetensors,deployments/}
                          per-node small artefacts (bind targets)

Only weights/ is allowed on node NVMe. Everything else resolves through the bind mounts, so FLARE's view of a model directory is unchanged.

3. Why bind mounts and not symlinks

FLARE's own error strings include "a package primary escapes its directory" and "staged entry is not a regular file" — a symlink pointing outside the model directory is exactly what those checks reject. Bind mounts are invisible to the process: the path stays inside the directory and the entry is a real file. Do not substitute symlinks.

Known risk: if FLARE ever replaces deployments/ by rename (rm -rf + mv), the mount point disappears and writes fall back to local disk. Phase 3 is a gate that measures this; if it happens, fall back to option D: leave deployments/ local (1.8 GiB) and move only the package store. Do not fight it.

4. Phases

Each phase is independently revertible. Do not start a phase whose gate the previous phase failed.

Phase 0 — prerequisites (no FLARE restart yet)

  1. Rebuild the runtime before any agent restart. The installed binary is the PPA build (0.1.0~git20261105002735.0009b89d-1), which lacks PR #184. The running agents still hold the PR-184 inode; restarting without the rebuild drops the io_uring Engram reader on dgx1/2/3/8. See references/build.md in the flare skill, or flare-upstream.pin + pr184-on-main here. Verify: strings <binary> | grep -c "embedding read worker panicked" → 1.
  2. Confirm NAS health and space, from every node that will use it:

    ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
    
  3. Record the current state of each node for rollback:

    du -sh ~/.cache/flare/* | sort -h > /tmp/flare-before-$(hostname).txt
    findmnt -no TARGET,SOURCE,OPTIONS /mnt/nas
    pgrep -af "[f]lare run" | head -1
    
  4. dgx8 currently has the NAS in fstab but not mounted. Fix that first (sudo mount /mnt/nas), and make it non-blocking for boot by adding nofail and a timeout to the fstab line (see phase 1).

Gate: binary rebuilt and staged, NAS mounted on every participating node.

Phase 1 — make the NAS mount boot-proof

The existing line (all nodes) is:

nas.lan:/sna  /mnt/nas  nfs4  rw,hard,rsize=1048576,wsize=1048576,timeo=600,retrans=5,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0

Two problems for this plan: a NAS hiccup blocks boot, and hard + long timeo means a stale mount hangs reads (this NAS has already produced a server not responding outage that needed reboots).

Change to:

nas.lan:/sna  /mnt/nas  nfs4  rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0
sudo cp /etc/fstab /etc/fstab.bak-$(date +%F)
sudo sed -i 's|^\(nas.lan:/sna.*\)$|# replaced by the storage plan\nnas.lan:/sna  /mnt/nas  nfs4  rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0|' /etc/fstab
sudo systemctl daemon-reload
sudo umount /mnt/nas 2>/dev/null; sudo mount /mnt/nas
findmnt -no TARGET,OPTIONS /mnt/nas | grep -o 'nofail'

Gate: mount present, nofail in effect, a sudo mount -a is a no-op.

Phase 2 — package store (the 477 GiB) to NAS

Low risk, no bind mounts, no FLARE code path surprises.

sudo mkdir -p /mnt/nas/ai/flare/store
sudo chown kilork:kilork /mnt/nas/ai/flare/store
# rsync -H is mandatory: format-2 packages are hardlink stores
rsync -aH --info=progress2 ~/.cache/flare/packages/ /mnt/nas/ai/flare/store/

Point the CLI at it, in ~/.config/flare/config.toml on the node that downloads (rank 0 / controller):

[paths]
package_cache = "/mnt/nas/ai/flare/store"

Then remove the local copy only after a verified download check:

flare models list --model-cache /mnt/nas/ai/flare/store   # must list the version
rm -rf ~/.cache/flare/packages

Gate: flare models list sees the version on NAS, df -h / shows the space back, and the next orbit run still starts (see phase 4 gate for the full test).

Open item to verify, not assume: ranks are launched with an explicit --package-cache /home/kilork/.cache/flare that does not come from [paths] (the agent synthesises it). Check the launched cmdline after a start:

pgrep -af "[f]lare run" | head -1 | tr ' ' '\n' | grep -A1 -e package-cache -e model-cache

If that path needs to change, it is a launcher change ([flare].extra_flags / [flare.clusters.<name>].extra_flags in the sm121 config, src/native/flare_orbit.rs::extra_flags_for) — never a hand edit on a node.

Phase 3 — validate the bind-mount model on an idle node (gate)

Use dgx4: it is idle (no ranks), and now boots from the new MP600 Mini.

  1. Copy one model's small artefacts from a running node:

    ssh dgx8.lan 'tar -C ~/.cache/flare/models/<model>/<version> -cf - model.safetensors deployments' \
     | tar -C /tmp -xf -
    
  2. Build a fake local model dir whose weights/ is a symlink to nothing and whose deployments/ is a bind mount from the NAS copy:

    mkdir -p ~/.cache/flare/models/<model>/<version>/weights
    mv /tmp/model.safetensors /tmp/deployments ~/.cache/flare/models/<model>/<version>/
    sudo mkdir -p /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>
    sudo mv ~/.cache/flare/models/<model>/<version>/deployments /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>/
    sudo mount --bind /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>/deployments \
        ~/.cache/flare/models/<model>/<version>/deployments
    
  3. Ask FLARE what it sees:

    flare inspect ~/.cache/flare/models/<model>/<version>/model.safetensors | head -20
    flare models list
    
  4. If a stage/launch is possible on dgx4 (flare run --stage-only with a dummy agent), run it and then check the mount survived:

    findmnt -no TARGET ~/.cache/flare/models/<model>/<version>/deployments
    

    Empty output = FLARE replaced the directory → take option D.

Gate: flare inspect shows the deployment, and findmnt still reports the bind mount after any stage operation.

Phase 4 — roll out to sparks (window; ranks stop)

Repeat per node in rank order, one node at a time, verifying before moving on.

  1. Stop the group (needs a working controller; see the re-pair note in flare-bringup.md):

    flare orbit stop <group> --cluster sparks
    
  2. Move the node's small artefacts to NAS and bind them back:

    V=gb10-tp4-<build>; M=checkpoint-expanded-scales
    D=/home/kilork/.cache/flare/models/$M/$V
    N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V
    sudo mkdir -p "$N"
    rsync -aH --remove-source-files "$D/model.safetensors" "$N/"
    rsync -aH "$D/deployments/" "$N/deployments/"
    rm -rf "$D/deployments"
    sudo mount --bind "$N/deployments" "$D/deployments"
    

    Persist it in /etc/fstab (bind mounts are ordinary fstab entries and are mounted before the agent starts):

    /mnt/nas/ai/flare/nodes/dgx5/checkpoint-expanded-scales/gb10-tp4-<build>/deployments  /home/kilork/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-<build>/deployments  none  bind,nofail,x-systemd.requires=/mnt/nas,x-systemd.requires-mounts-for=/mnt/nas  0 0
    
  3. Start the group and verify on the node:

    findmnt -no TARGET,SOURCE ~/.cache/flare/models/$M/$V/deployments
    du -sh ~/.cache/flare/models/$M/$V/* | sort -h   # weights ~260G, rest tiny
    pgrep -af "[f]lare run" | head -1
    
  4. Only then move to the next node.

Gate: ranks serve, findmnt shows the bind mounts, du shows ~260 GiB of weights/ and nothing else of size.

Phase 5 — prod4 (dgx1–dgx4), same steps

Same procedure with --cluster prod4. dgx1 is rank 0 and is the one that gains the 477 GiB; it is also the node whose packages/ phase 2 removed.

Phase 6 — verification and rollback

Verification, per node:

du -sh ~/.cache/flare/models/*/*/*          # weights dominates
findmnt -no TARGET,SOURCE | grep deployments
df -h / /mnt/nas
flare orbit status --cluster sparks         # agents reachable, groups listed
curl -s localhost:<port>/v1/models | head  # API answers (port from the rank cmdline)

Rollback (per node, no data loss — the NAS copy is authoritative):

sudo umount ~/.cache/flare/models/$M/$V/deployments
sudo sed -i "\|$N/deployments|d" /etc/fstab
rsync -aH /mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V/deployments/ ~/.cache/flare/models/$M/$V/deployments/
flare orbit run <group> --cluster sparks

5. Risks and their fallbacks

risk detection fallback
FLARE replaces deployments/ by rename; bind mount lost findmnt empty after a stage Option D: deployments local, only the package store moves
NAS stall at boot node boots with /mnt/nas missing nofail (phase 1) keeps boot going; agent then writes deployments locally — re-sync after recovery
NAS outage during serving reads hang (hard mount) bind mounts are only touched at load; keep timeo=100,retrans=3; do not put staging/ on NAS
hardlink-blind copy duplicates the package du doubles after a copy always rsync -aH
mixed flare binaries after a restart rendezvous incompatible protocol phase 0 rebuild first, restart all ranks together

6. What must be measured, not assumed

  1. Whether the agent's --package-cache path can be redirected by config or needs a launcher extra_flags change (phase 2 note).
  2. Whether FLARE tolerates the bind mount across a real stage (phase 3 gate).
  3. Real NAS read throughput for deployments/ at load — 1.8 GiB over NFS at 1 GbE is ~15 s per rank; measure before rolling to prod, and keep weights/ local regardless.