flare-storage-nas.md 36 KB

FLARE storage: expanded checkpoint on node NVMe, everything else on NAS (2026-11-10)

Goal, in the operator's words: only the expanded checkpoint stays on the sparks NVMe (260 GiB for DeepSeek-V4.1-Flash tp4) and all other model artefacts live on the NAS. This document is written to be executed by someone with no context beyond it. Every command is copy-pasteable and every phase has a gate.

Milestones

  • NAS mount made boot-proof on every participating node (nofail, timeo=100, x-systemd.mount-timeout=30) — verified live, serving untouched
  • dgx8's missing NAS mount restored (it was in fstab but never mounted)
  • dgx1's stale NFS state cleared without a reboot; node pinned to the protocol that works there (4.0) — see the correction in Phase 1
  • Pre-migration state of dgx1/2/3/4/8 captured locally and on the NAS
  • NAS throughput measured (was open item 6.3)
  • Stale model data purged from the nodes — 783 GiB freed, dgx1 from 98 % full (22 G free) to 283 G free; serving never interrupted
  • Launch parameters aligned with the current gb10 package recipe; API port stays 30000 (operator directive), --mtp-adaptive-verification-alpha 0.75 added
  • Every node on the v0.1.0 runtime — staged identically on dgx1/2/3/4/8 at /usr/local/share/hyperflare/v0.1.0/flare (archive 7ba5ba43…, binary e3fe697e… 26,054,848 B, same sha256 on all five), nothing in /usr/bin or the agent path touched yet. Remaining: dgx5–7 (off) + the temp+mv swap and one restart window; PR #184 rebuild dropped (§6 item 5)
  • Fresh gb10 package acquired onto the NAS store and deployed; superseded build deleted (Phase 2R — replaces the "migrate the old store" plan)
  • Bind-mount layout validated on an idle node (Phase 3 gate)
  • Sparks rolled out (Phase 4, needs a window)
  • prod4 rolled out (Phase 5, needs a window)
  • Verification matrix run and rollback drilled (Phase 6)

Round plan

  • R1 (Q1) — Phase 0.2–0.4 + Phase 1 on dgx1/2/3/4/8
  • R2 (Q1) — stale-data purge instead of migration (operator decision): 783 GiB freed on dgx1/2/3, all ranks stayed up; also identified that every local package is a superseded build from a deleted repo (Phase 2R)
  • R3 (Q1) — Phase 2R first half: download the current gb10 package (474.7 GiB) into the NAS store with flare download … --revision gb10, verify with flare models list --model-cache /mnt/nas/ai/flare/store. In flight 2026-11-11 22:07 on dgx4; the Nov-5 apt binary could not do it at all (§ Phase 0.1), the tagged v0.1.0 binary does it at 55–78 MiB/s ⇒ ETA ≈ 2 h. Also done in this round: v0.1.0 staged on dgx1/2/3/4/8, and SGLang (Ripple eval server) stopped on dgx4 — 103 GiB → 4 GiB RAM used
  • R4 (Q1) — Phase 2R second half: fix dgx1's package_cache drift (it points at an empty ~/flare-pkgs/gb10), confirm the launched rank cmdline (open item 6.1)
  • R5 (Q1) — Phase 0.1 as rewritten: pre-flight that the packaged release is the binary every node starts into at cutover (no PR-184 rebuild — dropped)
  • R6 (Q2) — Phase 3 bind-mount validation gate on idle dgx4
  • R7 (Q2) — Phase 4 sparks rollout (window; ranks stop)
  • R8 (Q2) — Phase 5 prod4 rollout (window; ranks stop)
  • R9 (Q3) — Phase 6 verification matrix and rollback drill

Checks

Only rank 0 serves the API. dgx1:30000/v1/models answering is the health probe; curl on dgx2/dgx3/dgx8 returns 000 there by design (peers have no listening port) — do not read that as an outage. Peer health is pgrep -fc "[f]lare run" = 1.

The runtime check a round should run (staged-vs-running, never inferred from flare --version, which prints 0.1.0 for every build):

for h in dgx1 dgx2 dgx3 dgx4 dgx8; do printf "%-6s " $h
  ssh -o BatchMode=yes $h.lan 'printf "staged=%s installed=%s running=%s\n" \
    "$(sha256sum /usr/local/share/hyperflare/v0.1.0/flare 2>/dev/null | cut -c1-8)" \
    "$(sha256sum /usr/bin/flare | cut -c1-8)" \
    "$(readlink /proc/$(pgrep -f "[f]lare run" | head -1)/exe 2>/dev/null)"' 2>/dev/null || echo off
done

Expected today: staged=e3fe697e installed=03c8e463 running=…flare-agent/flare (deleted). All three equal to e3fe697e (and no (deleted)) is the gate for this milestone.

For the NAS mount, run from the controller; every node must report fstab=1 live=1:

for h in dgx1 dgx2 dgx3 dgx4 dgx5 dgx6 dgx7 dgx8; do printf "%-6s " $h
  ssh -o BatchMode=yes -o ConnectTimeout=5 $h.lan 'printf "%s fstab=%s live=%s ranks=%s\n" \
    "$(hostname -s)" "$(grep /mnt/nas /etc/fstab | grep -c nofail)" \
    "$(findmnt -no OPTIONS /mnt/nas | grep -c "timeo=100")" "$(pgrep -fc "[f]lare run")"' \
  2>/dev/null || echo "off"; done

1. Measured baseline (2026-11-10, read-only)

node role models/ packages/ rank cmdline cache
dgx8 sparks rank 3 261 GiB — --package-cache /home/kilork/.cache/flare
dgx1 prod4 rank 0 261 GiB 477 GiB --package-cache /home/kilork/.cache/flare

Anatomy of one model directory — this split is the whole plan:

~/.cache/flare/models/checkpoint-expanded-scales/<version>/
  model.safetensors      35 MiB      signed envelope (manifest + locations)
  deployments/<pkg>/rank-<N>/  788 KiB (rank 0) … 1.8 GiB (rank 1-3)
  weights/              260 GiB      ← the expanded checkpoint

Package store (CLI side, rank 0 only):

~/.cache/flare/packages/<hash>-<rev>/
  model.safetensors      35 MiB
  deployments/          7.2 GiB
  weights/              470 GiB      ← second copy of the same content

NAS: /mnt/nas = 33 TiB, 1.1 TiB used, /mnt/nas/ai/flare effectively empty.

What the move actually saves — read before committing

  • sparks ranks 1–3: 1.8 GiB each (261 → 260 GiB). Small.
  • rank 0 (prod4): 477 GiB (the duplicate package store) plus 7.2 GiB of deployments.
  • Future model versions stop multiplying: only their weights/ ever lands on NVMe.

If the intent is "NVMe holds one checkpoint and nothing else", this plan delivers it. If the intent was primarily disk relief on sparks, the win is small and the cheap part (phase 2, the package store) may be enough.

2. Target layout

# every rank node, local NVMe
/home/kilork/.cache/flare/models/<model>/<version>/
    weights/              real directory                    260 GiB
    model.safetensors     bind mount → NAS/nodes/<node>/<version>/model.safetensors
    deployments/          bind mount → NAS/nodes/<node>/<version>/deployments

# NAS
/mnt/nas/ai/flare/
    store/                paths.package_cache — CLI store, packages/, blobs/, staging/,
                          refs/ (rank 0's 477 GiB lives here, once)
    nodes/<node>/<model>/<version>/{model.safetensors,deployments/}
                          per-node small artefacts (bind targets)

Only weights/ is allowed on node NVMe. Everything else resolves through the bind mounts, so FLARE's view of a model directory is unchanged.

3. Why bind mounts and not symlinks

FLARE's own error strings include "a package primary escapes its directory" and "staged entry is not a regular file" — a symlink pointing outside the model directory is exactly what those checks reject. Bind mounts are invisible to the process: the path stays inside the directory and the entry is a real file. Do not substitute symlinks.

Known risk: if FLARE ever replaces deployments/ by rename (rm -rf + mv), the mount point disappears and writes fall back to local disk. Phase 3 is a gate that measures this; if it happens, fall back to option D: leave deployments/ local (1.8 GiB) and move only the package store. Do not fight it.

4. Phases

Each phase is independently revertible. Do not start a phase whose gate the previous phase failed.

Phase 0 — prerequisites (no FLARE restart yet)

  1. Every node must be running the packaged release build before any cutover (rewritten 2026-11-11, operator directive: “the flare itself must be the latest version and properly updated on all nodes”). The release is installed and identical everywhere — Installed == Candidate == 0.1.0~git20261105002735.0009b89d-1, /usr/bin/flare = 03c8e463 on dgx1/2/3/4/8 — but dgx1/2/3/8 execute deleted inodes (the PR #184 binaries), so only dgx4 actually runs the release. readlink /proc/<pid>/exe showing (deleted) is the symptom; the fix is a restart onto the packaged file, never cp. The PR #184 rebuild is dropped: PR #184 is not in the release (strings -a /usr/bin/flare | grep -c "embedding read worker panicked" → 0), and the current gb10 package cannot be loaded by anything older than 20261105002735 — so the release build wins. Revisit only if/when the PR is merged upstream. Keep the rollback binaries (flare.orig-20261106) until the cutover is verified.

Corrected 2026-11-11 (R3): “the latest apt build” is no longer the latest runtime, and it cannot acquire the current package.

  • The project was renamed again: NVIDIA/spark-local-inference → 301 → NVIDIA/hyperflare-inference (default branch main, pushed 2026-11-11, 801 ★). (github.com/hyperflare/hyperflare is an unrelated 2017 repo — do not use it.)
  • Upstream now ships first-class release archives from version tags (commit 94fbce5f6 “Publish release archives from version tags”). Tag v0.1.0, published 2026-11-11T19:31Z, assets: flare-linux-arm64.tgz 11,427,812 B, sha256 7ba5ba4387cfe56a2e8205d711625addf26633219ec670ca33c23fd046d22d84 (unpacks to flare/flare, 26,054,848 B, plus LICENSE/NOTICE/THIRD_PARTY_NOTICES). Contents of v0.1.0 include PR #29 Multi-host disk prefix caching, PR #35/#36 requested logprob token IDs / System One scoring, ARM SHA-256 acceleration. PR #184 is still not merged (marker string → 0 in the v0.1.0 binary too).
  • The APT repo (ppa.local-inference-lab.ai, still the documented channel in the gb10 README, and its install stanza is unchanged: lil-flare-archive.asc + flare.sources + apt install flare-inference) lags: after a clean apt-get update, apt-cache madison flare-inference offers only 20261105002735 / 20261030013211 / 20261027170349. There is no newer apt build.
  • Measured consequence, not speculation: with the installed Nov-5 binary the acquisition of the current gb10 revision never gets past Verify package — 0 B for >120 s, holding locks/entries/checkpoint-expanded-scales--gb10-tp4-…lock and locks/staging/<pid>-0.lock, making one TLS handshake after another to 13.36.183.231 (HF's eu-west-3 LFS CDN) with no payload transfer. The same command with the v0.1.0 binary streams immediately (Acquire core, 55–58 MiB/s, 5 GiB in 90 s). So: runtime channel = GitHub release archives (primary), source build (backup); apt is only a floor.
  • v0.1.0 is not yet packaged by us, so until the cutover the fleet still runs Nov-5 while acquiring with the tagged release binary. Two consequences: install the chosen build on every node in the window (one identical binary, temp+mv), and record the sha256 of whatever we install.

Staged 2026-11-11 (R3), zero disruption: the verified archive is unpacked on dgx1, dgx2, dgx3, dgx4, dgx8 to /usr/local/share/hyperflare/v0.1.0/ (flare, LICENSE, NOTICE, THIRD_PARTY_NOTICES.md, archive.sha256, binary.sha256). Every node independently re-downloaded and checked archive sha256 7ba5ba43…; the resulting binary is 26,054,848 B, sha256 e3fe697eab1f97ff… — identical on all five, and each runs flare --version → flare 0.1.0. /usr/bin/flare and ~/.local/lib/flare-agent/flare were not touched, so nothing serving can be affected. dgx5–7 are off and still need this.

The swap in the window is deliberately boring, and note that /usr/bin/flare is not what runs — the agent executes ~/.local/lib/flare-agent/flare:

   # per node, in the window (rename, never cp — see the SIGBUS rule)
   sudo install -m 0755 /usr/local/share/hyperflare/v0.1.0/flare /usr/bin/flare.tmp-v010
   sudo mv /usr/bin/flare.tmp-v010 /usr/bin/flare
   # the agent's own copy is what the unit starts; refresh it the same way, then
   # restart the ranks together (one group, one window). Verify afterwards:
   readlink /proc/$(pgrep -f "[f]lare run" | head -1)/exe   # must NOT say (deleted)
   sha256sum /usr/bin/flare ~/.local/lib/flare-agent/flare  # both must be e3fe697e…
  1. Confirm NAS health and space, from every node that will use it:

    ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
    
  2. Record the current state of each node for rollback — done 2026-11-11, files are on each node and in /mnt/nas/ai/flare/ (flare-before-gx10-dgxN.txt). Hostnames are gx10-dgxN, not the dgxN.lan aliases — build filenames from hostname -s and do not assume the alias resolves to a filename:

    du -sh ~/.cache/flare/* | sort -h > /tmp/flare-before-$(hostname -s).txt
    findmnt -no TARGET,SOURCE,OPTIONS /mnt/nas
    awk 'NR>1' /proc/fs/nfsfs/servers | sort -u | wc -l    # NFS client IDs
    tr '\0' ' ' < /proc/$(pgrep -f "[f]lare run" | head -1)/cmdline   # exact launch command
    

    The captured rank cmdline is the recovery recipe if a node has to be restarted outside orbit (the controller is currently unpaired), e.g. rank 0 on dgx1: flare run ~/.cache/flare/packages/7cf40bec…-35842073/model.safetensors --package-cache ~/.cache/flare --rank 0 --rendezvous 0.0.0.0:29800 --max-concurrency 32 --mtp-draft-tokens 5 --mtp-adaptive-verification --embedding-table-storage disk --embedding-table-cache-size 1073741824 --embedding-table-resident-scales --transfer-streams 1 --host 0.0.0.0 --port 30000 --served-model-name deepseek-v4.1-flash --publisher-keys … --dangerously-approve-key … --dangerously-accept-publisher local-inference-lab --gpus 0

  3. dgx8 has the NAS in fstab but not mounted fixed 2026-11-11 — mounted at vers=4.1 and made non-blocking with the phase 1 options.

Gate: binary rebuilt and staged, NAS mounted on every participating node.

Phase 1 — make the NAS mount boot-proof

The existing line (all nodes) is:

nas.lan:/sna  /mnt/nas  nfs4  rw,hard,rsize=1048576,wsize=1048576,timeo=600,retrans=5,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0

Two problems for this plan: a NAS hiccup blocks boot, and hard + long timeo means a stale mount hangs reads (this NAS has already produced a server not responding outage that needed reboots).

Applied everywhere, with one per-node exception (see below):

nas.lan:/sna  /mnt/nas  nfs4  rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0

Do not use a blanket sed over the nas.lan:/sna line — it silently rewrites the protocol version too. Rewrite the entry by its mount-point field and pass the protocol per node — use host-scripts/set-nas-mount-opts.sh <vers> [remount] (proven idempotent on dgx4 on 2026-11-11; it backs up fstab, rewrites the entry, and plain-umounts before mounting). For reference, it is equivalent to:

sudo cp -a /etc/fstab /etc/fstab.bak-$(date +%F-%H%M)
sudo python3 - "$VERS" <<'PY'
import sys
vers = sys.argv[1]
path, out, hit = "/etc/fstab", [], False
for line in open(path).read().splitlines(True):
    f = line.split()
    if len(f) >= 3 and f[1] == "/mnt/nas":
        opts = ("rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,"
                "timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=" + vers)
        out.append("nas.lan:/sna\t/mnt/nas\tnfs4\t%s\t0\t0\n" % opts); hit = True
    else:
        out.append(line)
assert hit, "no /mnt/nas entry in fstab"
open(path, "w").write("".join(out))
PY
sudo systemctl daemon-reload
sudo umount /mnt/nas && sudo mount /mnt/nas     # umount fails safely if something holds it
findmnt -no OPTIONS /mnt/nas | grep -o 'timeo=100'

nofail is an fstab-only option: it never appears in live mount options, so verify the live change with timeo=100, not with nofail.

dgx1 exception — applied and verified 2026-11-11. dgx1 cannot mount NFSv4.1 at all: a fresh vers=4.1 mount hangs even with a single clean client and with nconnect=1, while dgx2/3/4/8 mount 4.1 instantly. Its fstab is therefore pinned to vers=4.0 with the same other options. Its two stale client IDs were cleared without a reboot (umount, not umount -l, with zero open files — that is the procedure; a reboot was not needed).

Gate: mount present, nofail in effect, a sudo mount -a is a no-op. Passed on dgx1/2/3/4/8 (2026-11-11): live options show timeo=100, mount -a returns 0 without remounting, all rank PIDs kept their 7-day uptime and the API on dgx1:30000 still answered. dgx5–7 are off and still need this.

Phase 2 — package store (the 477 GiB) to NAS — SUPERSEDED, do not run

Superseded on 2026-11-11 (R2). Two independent reasons:

  1. The operator rejected it: “Syncing those old flare cache to store does not make much sense honestly.” Copying a superseded build onto the NAS is not migration, it is archiving garbage.
  2. It is superseded by fact: the local package is an out-of-date build whose source repo no longer exists (see Phase 2R). The rsync was started, reached 105 GiB, was killed, and the partial copy was deleted.

What replaced it: purge the stale data (done, below) and acquire the current revision onto the NAS store (Phase 2R).

Purge performed on 2026-11-11

Every node held the Nov-2 build checkpoint-expanded-scales/gb10-tp4-202611021936 (261 GiB each) with nothing holding it open — verified three ways per node before deleting (/proc/<rank>/fd, /proc/<rank>/maps, and lsof +D):

node before after freed
dgx1 848 G used, 22 G free (98 %) 587 G used, 283 G free 261 G
dgx2 664 G used, 206 G free 403 G used, 467 G free 261 G
dgx3 661 G used, 209 G free 400 G used, 470 G free 261 G

Total 783 GiB freed. Rank PIDs were unchanged (7-day uptime) and dgx1:30000/v1/models kept answering. Note dgx1's root is 915 GiB, not 1.8 TB like dgx4 — the 261 GiB was 28 % of that disk.

Still present and not deletable while serving:

  • dgx1 ~/.cache/flare/packages/7cf40bec…-35842073 — 477 G, and rank 0 has its weights/engram-*.safetensors (92 G each) open. Space returns only when the rank exits, so this one dies in the Phase 5 window, not before.
  • dgx2/dgx3 models/…202611032142 and dgx8 models/…202611022100 (261 G each) — the live ranks read these.

For reference, the original (not executed) procedure was:

sudo mkdir -p /mnt/nas/ai/flare/store
sudo chown kilork:kilork /mnt/nas/ai/flare/store
# rsync -H is mandatory: format-2 packages are hardlink stores
rsync -aH --info=progress2 ~/.cache/flare/packages/ /mnt/nas/ai/flare/store/

Point the CLI at it, in ~/.config/flare/config.toml on the node that downloads (rank 0 / controller):

[paths]
package_cache = "/mnt/nas/ai/flare/store"

Then remove the local copy only after a verified download check:

flare models list --model-cache /mnt/nas/ai/flare/store   # must list the version
rm -rf ~/.cache/flare/packages

Gate: flare models list sees the version on NAS, df -h / shows the space back, and the next orbit run still starts (see phase 4 gate for the full test).

Open item to verify, not assume: ranks are launched with an explicit --package-cache /home/kilork/.cache/flare that does not come from [paths] (the agent synthesises it). Check the launched cmdline after a start:

pgrep -af "[f]lare run" | head -1 | tr ' ' '\n' | grep -A1 -e package-cache -e model-cache

If that path needs to change, it is a launcher change ([flare].extra_flags / [flare.clusters.<name>].extra_flags in the sm121 config, src/native/flare_orbit.rs::extra_flags_for) — never a hand edit on a node.

Phase 2R — replace, don't migrate (2026-11-11)

What we are actually serving is superseded, and its source is gone. The acquired package's manifest (read out of the flare.manifest tensor inside model.safetensors — it is protobuf, not JSON, but the strings are legible) names:

source   https://huggingface.co/local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE2
model    checkpoint-expanded-scales   identity gb10   publisher local-inference-lab

That repo answers 404 with a valid token (401 unauthenticated is HF masking a missing repo). The current source is the same publisher without the 2, and gb10 is a branch of it:

repo local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE (public, no token needed)
revision branch gb10 — last commit 2026-11-05T01:42:45 “Publish the GB10 TP4 package with vocabulary-parallel sampling and an FP8 Markov head”
size 187 files, 474.7 GiB (we hold 184 files / 477 GiB from a Nov-3 pull)
requires flare-inference 0.1.0~git20261105002735.0009b89d-1 or newer — “earlier versions cannot load it”

That version is exactly what the fleet has installed on every node. The binaries running are the older PR-184 inodes, so no node can load the new package until the Nov-5-based rebuild (R5) lands — the same conclusion #11 reached, now with a functional reason instead of a performance one.

Download into the NAS store (this is what the store is for — fresh acquisitions, not old caches). The hf token, when one is needed, is export HF_TOKEN=… in ~/.zshenv on the nodes — there is no token on the mac:

# Verified working form (run on dgx4, the idle node, with the NAS mounted):
~/hyperflare/flare/flare download local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE \
  --revision gb10 \
  --tensor-parallel 4 \
  --model-cache /mnt/nas/ai/flare/store \
  --publisher-keys ~/.cache/flare/publishers/937c0143f6856d3ee8ad164f62e3d804bd825d7a8750d0accb9a5849f703a8c1.json \
  --token-file ~/.config/hf/token
flare models list --model-cache /mnt/nas/ai/flare/store     # must list the new version

Three things the upstream example does not tell you (all hit in R3):

  1. --tensor-parallel 4 is mandatory here. Their example works because they run it on the controller of a four-Spark group; on a lone Spark the default is the local GPU count (1), and the acquisition dies with Error: "package exports no deployment with tensor-parallel size 1" after fetching the 35 MiB envelope.
  2. Publisher trust must be non-interactive. flare download has no --dangerously-approve-key / --dangerously-accept-publisher flags (those exist on orbit run only) — it takes --publisher-keys <json> and --trust-store <path>. The two files to copy from a node that already accepted the key: ~/.config/flare/trust.toml (fleet-uniform: dgx1 and dgx2 have the same file, 270 B, sha256 26e5ae90568572a28cbea9fb4b274ddca4127f4edf00b2f23086a2079db78464, containing the 937c0143… fingerprint) and ~/.cache/flare/publishers/937c0143….json (219 B, sha256 c8215a75db94b6dc4ee2992789acb675527caf2806380b80b3b98d1c79698a38, from dgx1 — dgx2/dgx4's publishers/ directories are empty). Copy them with base64 -w0 and compare sha256 on both ends; do not overwrite a trust store without checking it first (dgx4 had its own copy of the same file).
  3. A token file helps even for a public repo — --token-file ~/.config/hf/token (mode 600, built from export HF_TOKEN=… in ~/.zshenv on the node) keeps the LFS transport off anonymous rate limits over 475 GiB. --model-cache is a flag; it does not touch [paths] in config, so it causes no drift on the node.

Everything lands under /mnt/nas/ai/flare/store/ as transport/hub/v1/{blobs,repos}/… plus staging/, locks/, refs/, models/ — the transport/ default comes from --cache-dir (“parent directory for the shared Hugging Face download transport cache [default: the model storage directory]”), so all 475 GiB is written once, on the NAS, and renames inside it are cheap.

Progress of the acquisition started 2026-11-11 22:07 on dgx4 (detached, survives the session — setsid nohup):

tail -c 400 ~/flare-download-gb10.log          # symlink to the newest log
du -sh /mnt/nas/ai/flare/store                 # bytes landed on the NAS
pgrep -af "[f]lare download"                   # still alive?

Observed rate 55–58 MiB/s (WAN-bound, not NAS-bound) ⇒ 474.7 GiB ≈ 2.5 h. Measured inbound on enP7s7: 1482 MiB / 20 s ≈ 74 MiB/s.

The upstream launch recipe (SPARK-DS4-GUIDE.md / the gb10 README) is worth aligning our launcher flags with — note it differs from what prod4 runs today:

flare orbit run "spark1:${package_path}" \
  --cluster sparks --name deepseek1017 \
  --max-concurrency 32 --mtp-draft-tokens 5 --mtp-adaptive-verification \
  --mtp-adaptive-verification-alpha 0.75 \
  --embedding-table-storage disk \
  --embedding-table-resident-scales --embedding-table-cache-size 1GiB \
  --host 0.0.0.0 --port 8000 --detach

(Differences from our running rank 0: it adds --mtp-adaptive-verification-alpha 0.75, uses port 8000 not 30000, and names the group. Orbit ships each peer only the data it needs — “You do not need four full downloads.”)

Order of work: R5 rebuild → download fresh to NAS (R3) → window: stop ranks, delete the superseded 477 G on dgx1 and the 261 G dirs everywhere, start on the new package → then the bind-mount layout (Phase 3+) on top of the new version dirs. Doing the bind-mount work on the old version dirs would be wasted: the paths contain the version name, and every fstab entry would have to be rewritten.

Phase 3 — validate the bind-mount model on an idle node (gate)

Do this before touching dgx1–dgx3.

Use dgx4: it is idle (no ranks), and now boots from the new MP600 Mini.

  1. Copy one model's small artefacts from a running node:

    ssh dgx8.lan 'tar -C ~/.cache/flare/models/<model>/<version> -cf - model.safetensors deployments' \
     | tar -C /tmp -xf -
    
  2. Build a fake local model dir whose weights/ is a symlink to nothing and whose deployments/ is a bind mount from the NAS copy:

    mkdir -p ~/.cache/flare/models/<model>/<version>/weights
    mv /tmp/model.safetensors /tmp/deployments ~/.cache/flare/models/<model>/<version>/
    sudo mkdir -p /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>
    sudo mv ~/.cache/flare/models/<model>/<version>/deployments /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>/
    sudo mount --bind /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>/deployments \
        ~/.cache/flare/models/<model>/<version>/deployments
    
  3. Ask FLARE what it sees:

    flare inspect ~/.cache/flare/models/<model>/<version>/model.safetensors | head -20
    flare models list
    
  4. If a stage/launch is possible on dgx4 (flare run --stage-only with a dummy agent), run it and then check the mount survived:

    findmnt -no TARGET ~/.cache/flare/models/<model>/<version>/deployments
    

    Empty output = FLARE replaced the directory → take option D.

Gate: flare inspect shows the deployment, and findmnt still reports the bind mount after any stage operation.

Phase 4 — roll out to sparks (window; ranks stop)

Repeat per node in rank order, one node at a time, verifying before moving on.

  1. Stop the group (needs a working controller; see the re-pair note in flare-bringup.md):

    flare orbit stop <group> --cluster sparks
    
  2. Move the node's small artefacts to NAS and bind them back:

    V=gb10-tp4-<build>; M=checkpoint-expanded-scales
    D=/home/kilork/.cache/flare/models/$M/$V
    N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V
    sudo mkdir -p "$N"
    rsync -aH --remove-source-files "$D/model.safetensors" "$N/"
    rsync -aH "$D/deployments/" "$N/deployments/"
    rm -rf "$D/deployments"
    sudo mount --bind "$N/deployments" "$D/deployments"
    

    Persist it in /etc/fstab (bind mounts are ordinary fstab entries and are mounted before the agent starts):

    /mnt/nas/ai/flare/nodes/dgx5/checkpoint-expanded-scales/gb10-tp4-<build>/deployments  /home/kilork/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-<build>/deployments  none  bind,nofail,x-systemd.requires=/mnt/nas,x-systemd.requires-mounts-for=/mnt/nas  0 0
    
  3. Start the group and verify on the node:

    findmnt -no TARGET,SOURCE ~/.cache/flare/models/$M/$V/deployments
    du -sh ~/.cache/flare/models/$M/$V/* | sort -h   # weights ~260G, rest tiny
    pgrep -af "[f]lare run" | head -1
    
  4. Only then move to the next node.

Gate: ranks serve, findmnt shows the bind mounts, du shows ~260 GiB of weights/ and nothing else of size.

Phase 5 — prod4 (dgx1–dgx4), same steps

Same procedure with --cluster prod4. dgx1 is rank 0 and is the one that gains the 477 GiB; it is also the node whose packages/ phase 2 removed.

Phase 6 — verification and rollback

Verification, per node:

du -sh ~/.cache/flare/models/*/*/*          # weights dominates
findmnt -no TARGET,SOURCE | grep deployments
df -h / /mnt/nas
flare orbit status --cluster sparks         # agents reachable, groups listed
curl -s localhost:<port>/v1/models | head  # API answers (port from the rank cmdline)

Rollback (per node, no data loss — the NAS copy is authoritative):

sudo umount ~/.cache/flare/models/$M/$V/deployments
sudo sed -i "\|$N/deployments|d" /etc/fstab
rsync -aH /mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V/deployments/ ~/.cache/flare/models/$M/$V/deployments/
flare orbit run <group> --cluster sparks

Runtime parameters (2026-11-11)

Decision: take the launch parameters from the current gb10 package, not from what the lane happens to run today — with one deliberate exception, the API port stays 30000 (every lane here serves on 30000; 8000 in their example is just their host convention).

Their recipe vs. the live rank 0 command line on dgx1 (tr '\0' ' ' < /proc/$(pgrep -f "[f]lare run" | head -1)/cmdline):

flag upstream gb10 live action
--max-concurrency 32 32 keep
--mtp-draft-tokens 5 5 keep
--mtp-adaptive-verification yes yes keep
--mtp-adaptive-verification-alpha 0.75 absent added to both recipes
--embedding-table-storage disk disk keep
--embedding-table-resident-scales yes yes keep
--embedding-table-cache-size 1GiB 1073741824 equivalent (both = 1 GiB)
--transfer-streams 1 absent 1 keep ours — RoCE workaround, flare-roce-poison.md
--host 0.0.0.0 0.0.0.0 keep
--port 8000 30000 keep 30000
--served-model-name deepseek-v4.1-flash absent set keep ours (client-facing name)

alpha 0.75 is theirs on purpose: “favor longer verification rounds, which raises decode throughput with four or more concurrent requests and coding throughput” — i.e. it helps exactly the lane shape we run (c32, TP4).

They also document memory/KV knobs (--memory-limit=-12G, --kv-cache-limit memory|context|concurrency) and recommend leaving both at the default memory: “Every GiB reserved or cached away from the default reduces the KV capacity that Flare can plan.” Their stated budget — ~11.4 GiB KV per GPU, ~13.1 M tokens, 17–18 GiB of 121 GiB left free — matches what we measure, so no change. The 1 GiB Engram row cache is their recommended default; they suggest 4–8 GiB only if memory is spare and the startup Plan cache line confirms room, and our own 48 GiB experiment failed with “selected concurrency and verification width exceed available memory” — so 1 GiB stays.

5. Risks and their fallbacks

risk detection fallback
FLARE replaces deployments/ by rename; bind mount lost findmnt empty after a stage Option D: deployments local, only the package store moves
NAS stall at boot node boots with /mnt/nas missing nofail (phase 1) keeps boot going; agent then writes deployments locally — re-sync after recovery
NAS outage during serving reads hang (hard mount) bind mounts are only touched at load; keep timeo=100,retrans=3; do not put staging/ on NAS
hardlink-blind copy duplicates the package du doubles after a copy always rsync -aH
mixed flare binaries after a restart rendezvous incompatible protocol phase 0 rebuild first, restart all ranks together

6. What must be measured, not assumed

  1. Whether the agent's --package-cache path can be redirected by config or needs a launcher extra_flags change (phase 2 note). Partly answered 2026-11-11, and it is worse than assumed: on dgx1 the CLI's configured storage is /home/kilork/flare-pkgs/gb10, an empty 68 K skeleton — flare models list says “No local model versions found” and flare models inspect cannot see the live package id — while the real 477 G store is in ~/.cache/flare/packages because rank 0 is launched with --package-cache /home/kilork/.cache/flare. So redirection works, but CLI and agent are already pointed at different trees on one node. Unify them (launcher local_dir + agent flag) before either points at the NAS, or the NAS store will inherit the same split-brain.
  2. Whether FLARE tolerates the bind mount across a real stage (phase 3 gate).
  3. Real NAS read throughput measured 2026-11-11 (dd iflag=direct, 2 GiB, so no page-cache lying): the NAS is 10 GbE, not 1 GbE.

| node | write 2 GiB | read 1 MiB | read 8 KiB | |---|---|---|---| | dgx8 (v4.1) | 659 MiB/s | 731 MiB/s | 28 MiB/s | | dgx1 (v4.0) | 663 MiB/s | 822 MiB/s | 32 MiB/s |

So a 1.8 GiB deployments/ directory reads in 2–3 s, and the protocol choice costs nothing. The real constraint is small-file/short-read work: at 8 KiB the same share runs at ~30 MiB/s, so a deployments/ split into many small files would load ~25× slower than the bulk figure suggests — count the files in a rank deployment before trusting the 2–3 s number, and keep weights/ local regardless.

  1. Package revision currency must be verified against the source before any storage work (2026-11-11: the fleet served a superseded Nov-3 build whose source repo now 404s — see Phase 2R). To read what a local package is, do not trust the tree layout; read the identity out of the payload (the flare.manifest tensor is protobuf, but the strings are legible):

    strings -a <pkg>/model.safetensors | grep -aoE "https://huggingface.co/[^ ]*" | sort -u
    

    and compare with https://huggingface.co/api/models/<repo>/commits/<revision>. The HF API masks a missing repo as 401 when unauthenticated, so always re-check with a token before concluding a repo is gated. A valid token is export HF_TOKEN=… in ~/.zshenv on the nodes — there is none on the mac.

  2. flare-inference must be the latest release and actually running on every node. Measured 2026-11-11: on dgx1/2/3/4/8 Installed == Candidate == 0.1.0~git20261105002735.0009b89d-1 and /usr/bin/flare is the same file (03c8e463) everywhere — but dgx1/2/3/8 execute deleted inodes (the PR #184 binaries), so only dgx4 is truly on the release build. The package's ≥ 20261105002735 requirement makes this a hard gate: no node can load the current gb10 revision until it restarts onto the packaged binary. PR #184 is not in the release (strings -a /usr/bin/flare | grep -c "embedding read worker panicked" → 0), so running the latest build means giving up the patched Engram reader until it is merged upstream.

  3. Node root disks are not uniform — dgx1 is 915 GiB, dgx4 is 1.8 TiB. dgx1 was the one at 98 % full; a plan sized from dgx4's disk would have missed the whole reason this migration exists.

  4. Package stores across dgx1–dgx3 share one id (7cf40bec…-35842073), and the layout here is flat: du -l reports 0 hardlinks / 184 files, so rsync -H was never load-bearing on this fleet.