Goal, in the operator's words: only the expanded checkpoint stays on the sparks
NVMe (260 GiB for DeepSeek-V4.1-Flash tp4) and all other model artefacts live
on the NAS. This document is written to be executed by someone with no context
beyond it. Every command is copy-pasteable and every phase has a gate.
nofail,
timeo=100, x-systemd.mount-timeout=30) — verified live, serving untouchedgb10 package recipe; API port
stays 30000 (operator directive), --mtp-adaptive-verification-alpha 0.75 added/usr/local/share/hyperflare/v0.1.0/flare (archive 7ba5ba43…, binary
e3fe697e… 26,054,848 B, same sha256 on all five). Swapped and running on dgx4
(2026-11-12: /usr/bin + agent copy both e3fe697e, agent exe not (deleted),
flare.orig-20261112 backups) — the swap is rehearsed and known-good. Remaining:
dgx1/2/3/8 (needs the window, they hold the ranks) and dgx5–7 (off).
PR #184 rebuild dropped (§6 item 5). Note ~/.local/lib/flare-agent/flare is
not dpkg-owned — apt never updates what the unit actually runsgb10 core acquired onto the NAS store and verified —
checkpoint-expanded-scales/gb10-tp4-202611120031, 474.66 GiB, TP 4 · ready,
signatures verified, visible from dgx1/dgx4 over NFS (Phase 2R)deployments/ (TP1 test bed), negative control failed correctly, no
writes/rename by FLARE, fstab bind survives reboot with Requires=mnt-nas.mount.
Remaining for the layout: apply it to the TP4 rank slices in the window (R7/R8).deployments/ was local (see the measurement caveats). Since
R4 pointed package_cache at the NAS store, confirm on an idle node that a rank
still starts when /mnt/nas is unavailable — if it does not, the NAS is on the
rank restart path and that must be documented as an availability dependency.gb10 core acquired and verified on the NAS store:
models/checkpoint-expanded-scales/gb10-tp4-202611120031, 474.66 GiB,
payload v2, deployment db20209d361f473b8452f24ad517fa0d TP 4 · ready,
474.63 GiB in 181 hardlinked objects, signatures verified by the publisher,
flare models list correct from dgx1 (prod) and dgx4. 2 h 8 min, avg
63 MiB/s (peak 104). Note the gate command is flare models list /
--core-cache, not --model-cache (see the vocabulary section)package_cache now /mnt/nas/ai/flare/store on dgx4 + dgx1
(backups kept; dgx1 rank untouched, api=200, same PID). Answered the config
question: v0.1.0 still honours paths.package_cache, the launcher never passes
the cache flag, and the flag renamed --package-cache → --core-cache — see the
new “v0.1.0 vocabulary and paths” section. Still open: confirm the launched
cmdline after the first restart (needs the window)/usr/bin/flare and ~/.local/lib/flare-agent/flare both e3fe697e,
agent restarted (systemctl --user restart flare-agent.service, new pid,
readlink /proc/<pid>/exe without (deleted)), Nov-5 backups kept at both
paths as flare.orig-20261112. The mechanism behind the fleet's deleted inodes is
proven: dpkg -S ~/.local/lib/flare-agent/flare → owned by no package, so apt
refreshes /usr/bin/flare and never the copy the unit ExecStarts. dgx4 is
therefore ahead of dgx1/2/3/8 — do not add it to a group until the window swaps
the others (one identical binary per running group)--stage-only passes Verify core through a NAS bind, mount survives
five stage attempts. Also found: the plan's Phase 3 procedure was invalid
(a standalone model-dir copy is rejected in v0.1.0) and there is a data-loss
trap in Phase 4 cleanup — see “Phase 3 gate results”. No data was lost.deployments/
bind-mounted from the NAS, and the negative control (bind an empty NAS dir →
rank refuses to start) proves the data comes through the bind. Across a complete
load+serve cycle FLARE wrote nothing into the bind (0 files newer than rank
start) and never replaced it (same inode, still mounted after serving and after
reboot) ⇒ option D is not needed. Also validated the fstab-persisted bind
across a reboot of dgx4: mounted at boot, its unit provably
Requires=mnt-nas.mount, and a rank came up from it 22 s after boot.
Steady-state inference touches the NAS 0 MiB.Only rank 0 serves the API. dgx1:30000/v1/models answering is the health probe;
curl on dgx2/dgx3/dgx8 returns 000 there by design (peers have no listening port) —
do not read that as an outage. Peer health is pgrep -fc "[f]lare run" = 1.
The runtime check a round should run (staged-vs-running, never inferred from
flare --version, which prints 0.1.0 for every build):
for h in dgx1 dgx2 dgx3 dgx4 dgx8; do printf "%-6s " $h
ssh -o BatchMode=yes $h.lan 'A=$(pgrep -f "[f]lare run" | head -1); B=$(pgrep -f "[f]lare agent" | head -1)
printf "staged=%s /usrbin=%s agent=%s exe=%s\n" \
"$(sha256sum /usr/local/share/hyperflare/v0.1.0/flare 2>/dev/null | cut -c1-8)" \
"$(sha256sum /usr/bin/flare | cut -c1-8)" \
"$(sha256sum ~/.local/lib/flare-agent/flare | cut -c1-8)" \
"$(readlink /proc/${A:-$B}/exe 2>/dev/null)"' 2>/dev/null || echo off
done
agent= is the column that matters. The unit starts
%h/.local/lib/flare-agent/flare, and dpkg owns no such file, so /usrbin= can move
while the runtime does not. Expected now (dgx4 swapped, ranks still on the Nov-5 build):
dgx1 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
dgx4 staged=e3fe697e /usrbin=e3fe697e agent=e3fe697e exe=/home/kilork/.local/lib/flare-agent/flare
Gate for the milestone: agent=e3fe697e on every node, and no exe ending in
(deleted). A node reporting agent=03c8e463 after an apt upgrade is not updated.
For the NAS mount, run from the controller; every node must report fstab=1 live=1:
for h in dgx1 dgx2 dgx3 dgx4 dgx5 dgx6 dgx7 dgx8; do printf "%-6s " $h
ssh -o BatchMode=yes -o ConnectTimeout=5 $h.lan 'printf "%s fstab=%s live=%s ranks=%s\n" \
"$(hostname -s)" "$(grep /mnt/nas /etc/fstab | grep -c nofail)" \
"$(findmnt -no OPTIONS /mnt/nas | grep -c "timeo=100")" "$(pgrep -fc "[f]lare run")"' \
2>/dev/null || echo "off"; done
| node | role | models/ |
packages/ |
rank cmdline cache |
|---|---|---|---|---|
| dgx8 | sparks rank 3 | 261 GiB | — | --package-cache /home/kilork/.cache/flare |
| dgx1 | prod4 rank 0 | 261 GiB | 477 GiB | --package-cache /home/kilork/.cache/flare |
Anatomy of one model directory — this split is the whole plan:
~/.cache/flare/models/checkpoint-expanded-scales/<version>/
model.safetensors 35 MiB signed envelope (manifest + locations)
deployments/<pkg>/rank-<N>/ 788 KiB (rank 0) … 1.8 GiB (rank 1-3)
weights/ 260 GiB ← the expanded checkpoint
Package store (CLI side, rank 0 only):
~/.cache/flare/packages/<hash>-<rev>/
model.safetensors 35 MiB
deployments/ 7.2 GiB
weights/ 470 GiB ← second copy of the same content
NAS: /mnt/nas = 33 TiB, 1.1 TiB used, /mnt/nas/ai/flare effectively empty.
weights/ ever lands on NVMe.If the intent is "NVMe holds one checkpoint and nothing else", this plan delivers it. If the intent was primarily disk relief on sparks, the win is small and the cheap part (phase 2, the package store) may be enough.
# every rank node, local NVMe
/home/kilork/.cache/flare/models/<model>/<version>/
weights/ real directory 260 GiB
model.safetensors bind mount → NAS/nodes/<node>/<version>/model.safetensors
deployments/ bind mount → NAS/nodes/<node>/<version>/deployments
# NAS
/mnt/nas/ai/flare/
store/ paths.package_cache — CLI store, packages/, blobs/, staging/,
refs/ (rank 0's 477 GiB lives here, once)
nodes/<node>/<model>/<version>/{model.safetensors,deployments/}
per-node small artefacts (bind targets)
Only weights/ is allowed on node NVMe. Everything else resolves through the
bind mounts, so FLARE's view of a model directory is unchanged.
FLARE's own error strings include "a package primary escapes its directory" and "staged entry is not a regular file" — a symlink pointing outside the model directory is exactly what those checks reject. Bind mounts are invisible to the process: the path stays inside the directory and the entry is a real file. Do not substitute symlinks.
Known risk: if FLARE ever replaces deployments/ by rename (rm -rf +
mv), the mount point disappears and writes fall back to local disk. Phase 3 is a
gate that measures this; if it happens, fall back to option D: leave
deployments/ local (1.8 GiB) and move only the package store. Do not fight it.
Each phase is independently revertible. Do not start a phase whose gate the previous phase failed.
Installed == Candidate ==
0.1.0~git20261105002735.0009b89d-1, /usr/bin/flare = 03c8e463 on
dgx1/2/3/4/8 — but dgx1/2/3/8 execute deleted inodes (the PR #184 binaries), so
only dgx4 actually runs the release. readlink /proc/<pid>/exe showing (deleted)
is the symptom; the fix is a restart onto the packaged file, never cp.
The PR #184 rebuild is dropped: PR #184 is not in the release (strings -a
/usr/bin/flare | grep -c "embedding read worker panicked" → 0), and the current
gb10 package cannot be loaded by anything older than 20261105002735 — so the
release build wins. Revisit only if/when the PR is merged upstream.
Keep the rollback binaries (flare.orig-20261106) until the cutover is verified.Corrected 2026-11-11 (R3): “the latest apt build” is no longer the latest runtime, and it cannot acquire the current package.
NVIDIA/spark-local-inference → 301 →
NVIDIA/hyperflare-inference (default branch main, pushed 2026-11-11, 801 ★).
(github.com/hyperflare/hyperflare is an unrelated 2017 repo — do not use it.)94fbce5f6 “Publish release archives from version tags”). Tag v0.1.0,
published 2026-11-11T19:31Z, assets: flare-linux-arm64.tgz 11,427,812 B,
sha256 7ba5ba4387cfe56a2e8205d711625addf26633219ec670ca33c23fd046d22d84
(unpacks to flare/flare, 26,054,848 B, plus LICENSE/NOTICE/THIRD_PARTY_NOTICES).
Contents of v0.1.0 include PR #29 Multi-host disk prefix caching, PR #35/#36
requested logprob token IDs / System One scoring, ARM SHA-256 acceleration.
PR #184 is still not merged (marker string → 0 in the v0.1.0 binary too).ppa.local-inference-lab.ai, still the documented channel in the
gb10 README, and its install stanza is unchanged: lil-flare-archive.asc +
flare.sources + apt install flare-inference) lags: after a clean
apt-get update, apt-cache madison flare-inference offers only
20261105002735 / 20261030013211 / 20261027170349. There is no newer apt build.gb10 revision never gets past Verify package —
0 B for >120 s, holding locks/entries/checkpoint-expanded-scales--gb10-tp4-…lock
and locks/staging/<pid>-0.lock, making one TLS handshake after another to
13.36.183.231 (HF's eu-west-3 LFS CDN) with no payload transfer. The same
command with the v0.1.0 binary streams immediately (Acquire core, 55–58 MiB/s,
5 GiB in 90 s). So: runtime channel = GitHub release archives (primary), source
build (backup); apt is only a floor.v0.1.0 is not yet packaged by us, so until the cutover the fleet still runs
Nov-5 while acquiring with the tagged release binary. Two consequences: install
the chosen build on every node in the window (one identical binary, temp+mv),
and record the sha256 of whatever we install.Staged 2026-11-11 (R3), zero disruption: the verified archive is unpacked on
dgx1, dgx2, dgx3, dgx4, dgx8 to /usr/local/share/hyperflare/v0.1.0/
(flare, LICENSE, NOTICE, THIRD_PARTY_NOTICES.md, archive.sha256,
binary.sha256). Every node independently re-downloaded and checked
archive sha256 7ba5ba43…; the resulting binary is 26,054,848 B,
sha256 e3fe697eab1f97ff… — identical on all five, and each runs
flare --version → flare 0.1.0. /usr/bin/flare and
~/.local/lib/flare-agent/flare were not touched, so nothing serving can be
affected. dgx5–7 are off and still need this.
Use host-scripts/flare-swap.sh <staged-binary> — written and tested for this
(2026-11-12: verified idempotent no-op on dgx4, and it correctly refused on dgx8
with a rank running, exit 1, tmp-swap files count 0, nothing disturbed). It swaps
both paths by temp+mv, keeps a dated *.orig-<YYYYMMDD> backup at each, restarts
only the agent unit, and verifies sha256 and /proc/<pid>/exe for (deleted)
before exiting 0. Inside the window it is one command per node:
ssh dgxN.lan 'bash -s -- /usr/local/share/hyperflare/v0.1.0/flare' < host-scripts/flare-swap.sh
# ranks are already stopped in the window; the script still refuses unless FORCE=1
FORCE=1 ssh dgxN.lan 'bash -s -- /usr/local/share/hyperflare/v0.1.0/flare' < host-scripts/flare-swap.sh
The swap is deliberately boring, and note that /usr/bin/flare is not what runs —
the agent executes ~/.local/lib/flare-agent/flare (and dpkg owns neither copy):
# per node, in the window (rename, never cp — see the SIGBUS rule)
sudo install -m 0755 /usr/local/share/hyperflare/v0.1.0/flare /usr/bin/flare.tmp-v010
sudo mv /usr/bin/flare.tmp-v010 /usr/bin/flare
# the agent's own copy is what the unit starts; refresh it the same way, then
# restart the ranks together (one group, one window). Verify afterwards:
readlink /proc/$(pgrep -f "[f]lare run" | head -1)/exe # must NOT say (deleted)
sha256sum /usr/bin/flare ~/.local/lib/flare-agent/flare # both must be e3fe697e…
Confirm NAS health and space, from every node that will use it:
ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
Record the current state of each node for rollback — done 2026-11-11, files
are on each node and in /mnt/nas/ai/flare/ (flare-before-gx10-dgxN.txt).
Hostnames are gx10-dgxN, not the dgxN.lan aliases — build filenames from
hostname -s and do not assume the alias resolves to a filename:
du -sh ~/.cache/flare/* | sort -h > /tmp/flare-before-$(hostname -s).txt
findmnt -no TARGET,SOURCE,OPTIONS /mnt/nas
awk 'NR>1' /proc/fs/nfsfs/servers | sort -u | wc -l # NFS client IDs
tr '\0' ' ' < /proc/$(pgrep -f "[f]lare run" | head -1)/cmdline # exact launch command
The captured rank cmdline is the recovery recipe if a node has to be restarted
outside orbit (the controller is currently unpaired), e.g. rank 0 on dgx1:
flare run ~/.cache/flare/packages/7cf40bec…-35842073/model.safetensors
--package-cache ~/.cache/flare --rank 0 --rendezvous 0.0.0.0:29800
--max-concurrency 32 --mtp-draft-tokens 5 --mtp-adaptive-verification
--embedding-table-storage disk --embedding-table-cache-size 1073741824
--embedding-table-resident-scales --transfer-streams 1 --host 0.0.0.0 --port 30000
--served-model-name deepseek-v4.1-flash --publisher-keys … --dangerously-approve-key …
--dangerously-accept-publisher local-inference-lab --gpus 0
dgx8 has the NAS in fstab but not mounted fixed 2026-11-11 — mounted at
vers=4.1 and made non-blocking with the phase 1 options.
Pre-window disk math — do this before scheduling the window (2026-11-12).
The acquired core's weights/ is 470 G, i.e. bigger than the old 260 GiB the plan
was written against, and per-node free space is the one thing a window cannot recover:
| node | root | free now | superseded data freed by the window | note |
|---|---|---|---|---|
| dgx1 | 915 G | 283 G | 477 G package store (open by rank 0 until restart) | tight: ~760 G free afterwards |
| dgx2 | 1.8 T | 467 G | 261 G model dir | comfortable |
| dgx3 | 1.8 T | 470 G | 261 G model dir | comfortable |
| dgx4 | 1.8 T | 1.5 T | — (idle) | the staging test bed |
| dgx8 | 1.8 T | ~470 G | 261 G model dir | comfortable |
A peer's local footprint is not the store total: on the old build dgx2 held 261 G
where the package store was 477 G, and a rank slice is only 3 files / 1.3 GiB. The exact
new per-rank number is unknown until a real stage happens — measure it on dgx4 first
(1.5 T free), because if a rank needs the full 470 G locally then dgx1 lands at ~40 %
free and must not receive any further local copies. Never start a window whose aggregate
weights/ exceeds a node's free space after its own deletion: a rank that cannot write
its staging tree takes the whole group down.
Gate: binary staged and verified on every node (## Checks probe), agent binary swapped
wherever ranks are not running, NAS mounted on every participating node, and the table
above re-measured the day of the window.
The existing line (all nodes) is:
nas.lan:/sna /mnt/nas nfs4 rw,hard,rsize=1048576,wsize=1048576,timeo=600,retrans=5,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0
Two problems for this plan: a NAS hiccup blocks boot, and hard + long timeo
means a stale mount hangs reads (this NAS has already produced a
server not responding outage that needed reboots).
Applied everywhere, with one per-node exception (see below):
nas.lan:/sna /mnt/nas nfs4 rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0
Do not use a blanket sed over the nas.lan:/sna line — it silently rewrites
the protocol version too. Rewrite the entry by its mount-point field and pass the
protocol per node — use host-scripts/set-nas-mount-opts.sh <vers> [remount]
(proven idempotent on dgx4 on 2026-11-11; it backs up fstab, rewrites the entry, and
plain-umounts before mounting). For reference, it is equivalent to:
sudo cp -a /etc/fstab /etc/fstab.bak-$(date +%F-%H%M)
sudo python3 - "$VERS" <<'PY'
import sys
vers = sys.argv[1]
path, out, hit = "/etc/fstab", [], False
for line in open(path).read().splitlines(True):
f = line.split()
if len(f) >= 3 and f[1] == "/mnt/nas":
opts = ("rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,"
"timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=" + vers)
out.append("nas.lan:/sna\t/mnt/nas\tnfs4\t%s\t0\t0\n" % opts); hit = True
else:
out.append(line)
assert hit, "no /mnt/nas entry in fstab"
open(path, "w").write("".join(out))
PY
sudo systemctl daemon-reload
sudo umount /mnt/nas && sudo mount /mnt/nas # umount fails safely if something holds it
findmnt -no OPTIONS /mnt/nas | grep -o 'timeo=100'
nofail is an fstab-only option: it never appears in live mount options, so verify
the live change with timeo=100, not with nofail.
dgx1 exception — applied and verified 2026-11-11. dgx1 cannot mount NFSv4.1 at
all: a fresh vers=4.1 mount hangs even with a single clean client and with
nconnect=1, while dgx2/3/4/8 mount 4.1 instantly. Its fstab is therefore pinned
to vers=4.0 with the same other options. Its two stale client IDs were cleared
without a reboot (umount, not umount -l, with zero open files — that is the
procedure; a reboot was not needed).
Gate: mount present, nofail in effect, a sudo mount -a is a no-op. Passed on
dgx1/2/3/4/8 (2026-11-11): live options show timeo=100, mount -a returns 0
without remounting, all rank PIDs kept their 7-day uptime and the API on
dgx1:30000 still answered. dgx5–7 are off and still need this.
Superseded on 2026-11-11 (R2). Two independent reasons:
- The operator rejected it: “Syncing those old flare cache to store does not make much sense honestly.” Copying a superseded build onto the NAS is not migration, it is archiving garbage.
- It is superseded by fact: the local package is an out-of-date build whose source repo no longer exists (see Phase 2R). The rsync was started, reached 105 GiB, was killed, and the partial copy was deleted.
What replaced it: purge the stale data (done, below) and acquire the current revision onto the NAS store (Phase 2R).
Every node held the Nov-2 build checkpoint-expanded-scales/gb10-tp4-202611021936
(261 GiB each) with nothing holding it open — verified three ways per node before
deleting (/proc/<rank>/fd, /proc/<rank>/maps, and lsof +D):
| node | before | after | freed |
|---|---|---|---|
| dgx1 | 848 G used, 22 G free (98 %) | 587 G used, 283 G free | 261 G |
| dgx2 | 664 G used, 206 G free | 403 G used, 467 G free | 261 G |
| dgx3 | 661 G used, 209 G free | 400 G used, 470 G free | 261 G |
Total 783 GiB freed. Rank PIDs were unchanged (7-day uptime) and dgx1:30000/v1/models
kept answering. Note dgx1's root is 915 GiB, not 1.8 TB like dgx4 — the 261 GiB
was 28 % of that disk.
Still present and not deletable while serving:
~/.cache/flare/packages/7cf40bec…-35842073 — 477 G, and rank 0 has its
weights/engram-*.safetensors (92 G each) open. Space returns only when the rank
exits, so this one dies in the Phase 5 window, not before.models/…202611032142 and dgx8 models/…202611022100 (261 G each) — the
live ranks read these.For reference, the original (not executed) procedure was:
sudo mkdir -p /mnt/nas/ai/flare/store
sudo chown kilork:kilork /mnt/nas/ai/flare/store
# rsync -H is mandatory: format-2 packages are hardlink stores
rsync -aH --info=progress2 ~/.cache/flare/packages/ /mnt/nas/ai/flare/store/
Point the CLI at it, in ~/.config/flare/config.toml on the node that downloads
(rank 0 / controller):
[paths]
package_cache = "/mnt/nas/ai/flare/store"
Then remove the local copy only after a verified download check:
flare models list --model-cache /mnt/nas/ai/flare/store # must list the version
rm -rf ~/.cache/flare/packages
Gate: flare models list sees the version on NAS, df -h / shows the space back,
and the next orbit run still starts (see phase 4 gate for the full test).
Open item to verify, not assume: ranks are launched with an explicit
--package-cache /home/kilork/.cache/flare that does not come from
[paths] (the agent synthesises it). Check the launched cmdline after a start:
pgrep -af "[f]lare run" | head -1 | tr ' ' '\n' | grep -A1 -e package-cache -e model-cache
If that path needs to change, it is a launcher change
([flare].extra_flags / [flare.clusters.<name>].extra_flags in the sm121
config, src/native/flare_orbit.rs::extra_flags_for) — never a hand edit on a
node.
What we are actually serving is superseded, and its source is gone. The acquired
package's manifest (read out of the flare.manifest tensor inside model.safetensors
— it is protobuf, not JSON, but the strings are legible) names:
source https://huggingface.co/local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE2
model checkpoint-expanded-scales identity gb10 publisher local-inference-lab
That repo answers 404 with a valid token (401 unauthenticated is HF masking a
missing repo). The current source is the same publisher without the 2, and
gb10 is a branch of it:
| repo | local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE (public, no token needed) |
| revision | branch gb10 — last commit 2026-11-05T01:42:45 “Publish the GB10 TP4 package with vocabulary-parallel sampling and an FP8 Markov head” |
| size | 187 files, 474.7 GiB (we hold 184 files / 477 GiB from a Nov-3 pull) |
| requires | flare-inference 0.1.0~git20261105002735.0009b89d-1 or newer — “earlier versions cannot load it” |
That version is exactly what the fleet has installed on every node. The binaries running are the older PR-184 inodes, so no node can load the new package until the Nov-5-based rebuild (R5) lands — the same conclusion #11 reached, now with a functional reason instead of a performance one.
Download into the NAS store (this is what the store is for — fresh acquisitions,
not old caches). The hf token, when one is needed, is export HF_TOKEN=… in
~/.zshenv on the nodes — there is no token on the mac:
# Verified working form (run on dgx4, the idle node, with the NAS mounted):
~/hyperflare/flare/flare download local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE \
--revision gb10 \
--tensor-parallel 4 \
--model-cache /mnt/nas/ai/flare/store \
--publisher-keys ~/.cache/flare/publishers/937c0143f6856d3ee8ad164f62e3d804bd825d7a8750d0accb9a5849f703a8c1.json \
--token-file ~/.config/hf/token
flare models list --model-cache /mnt/nas/ai/flare/store # must list the new version
Three things the upstream example does not tell you (all hit in R3):
--tensor-parallel 4 is mandatory here. Their example works because they run it
on the controller of a four-Spark group; on a lone Spark the default is the local
GPU count (1), and the acquisition dies with
Error: "package exports no deployment with tensor-parallel size 1" after fetching
the 35 MiB envelope.flare download has no
--dangerously-approve-key / --dangerously-accept-publisher flags (those exist on
orbit run only) — it takes --publisher-keys <json> and --trust-store <path>.
The two files to copy from a node that already accepted the key:
~/.config/flare/trust.toml (fleet-uniform: dgx1 and dgx2 have the same file,
270 B, sha256 26e5ae90568572a28cbea9fb4b274ddca4127f4edf00b2f23086a2079db78464,
containing the 937c0143… fingerprint) and
~/.cache/flare/publishers/937c0143….json (219 B,
sha256 c8215a75db94b6dc4ee2992789acb675527caf2806380b80b3b98d1c79698a38, from dgx1
— dgx2/dgx4's publishers/ directories are empty). Copy them with base64 -w0 and
compare sha256 on both ends; do not overwrite a trust store without checking it
first (dgx4 had its own copy of the same file).--token-file ~/.config/hf/token
(mode 600, built from export HF_TOKEN=… in ~/.zshenv on the node) keeps the LFS
transport off anonymous rate limits over 475 GiB. --model-cache is a flag; it
does not touch [paths] in config, so it causes no drift on the node.Everything lands under /mnt/nas/ai/flare/store/ as
transport/hub/v1/{blobs,repos}/… plus staging/, locks/, refs/, models/ —
the transport/ default comes from --cache-dir (“parent directory for the shared
Hugging Face download transport cache [default: the model storage directory]”), so all
475 GiB is written once, on the NAS, and renames inside it are cheap.
Progress of the acquisition started 2026-11-11 22:07 on dgx4 (detached, survives the
session — setsid nohup):
tail -c 400 ~/flare-download-gb10.log # symlink to the newest log
du -sh /mnt/nas/ai/flare/store # bytes landed on the NAS
pgrep -af "[f]lare download" # still alive?
Observed rate 55 → 90 MiB/s as the transfer warms up (WAN-bound, not NAS-bound:
inbound on enP7s7 measured 1482 MiB / 20 s ≈ 74 MiB/s while the CLI reported
86 MiB/s) ⇒ 474.7 GiB in roughly 1.5 h from 22:07. Trust the CLI's own elapsed
timer in the log, not the wall clock: node clocks are UTC and the mac is not, so
date on the two disagree by about an hour.
The upstream launch recipe (SPARK-DS4-GUIDE.md / the gb10 README) is worth
aligning our launcher flags with — note it differs from what prod4 runs today:
flare orbit run "spark1:${package_path}" \
--cluster sparks --name deepseek1017 \
--max-concurrency 32 --mtp-draft-tokens 5 --mtp-adaptive-verification \
--mtp-adaptive-verification-alpha 0.75 \
--embedding-table-storage disk \
--embedding-table-resident-scales --embedding-table-cache-size 1GiB \
--host 0.0.0.0 --port 8000 --detach
(Differences from our running rank 0: it adds --mtp-adaptive-verification-alpha 0.75,
uses port 8000 not 30000, and names the group. Orbit ships each peer only the data it
needs — “You do not need four full downloads.”)
Order of work: R5 rebuild → download fresh to NAS (R3) → window: stop ranks, delete the superseded 477 G on dgx1 and the 261 G dirs everywhere, start on the new package → then the bind-mount layout (Phase 3+) on top of the new version dirs. Doing the bind-mount work on the old version dirs would be wasted: the paths contain the version name, and every fstab entry would have to be rewritten.
Version ids are mutable; deployment ids are not. The core was first acquired as
gb10-tp4-202611112207. That version's materialisation was destroyed by a recursive delete through a live bind (see “Phase 3 gate results”), and re-runningdownloadre-linked it from local blobs asgb10-tp4-202611120031— which is the id used by every command in this document now. The deployment iddb20209d361f473b8452f24ad517fa0d(TP 4 · ready) is content-derived and did not change, as did the inventory (474.66 GiB,181 hardlinked objects). Recovery cost: 34 MB of network, the remaining 509.6 GB re-read from local blobs, 1251 s. Current state:1 model · 1 version,[DEFAULT]=gb10-tp4-202611120031.
store/…/models/checkpoint-expanded-scales/gb10-tp4-202611120031/
model.safetensors 33 MiB mode 0444 (read-only)
weights/ 470 G
deployments/db20209d361f473b8452f24ad517fa0d/
rank-0 rank-1 rank-2 rank-3 1.3 G each, 5.1 G total
flare models list: 1 model · 1 version, Format payload v2,
Disk used 474.66 GiB, Shared files 474.63 GiB · 181 hardlinked objects,
Deployment db20209d361f473b8452f24ad517fa0d · TP 4 · ready,
Context / prefill 1,048,576 / 8,192 tokens, Pages / state rows 128 / 448.
flare models inspect checkpoint-expanded-scales: name DeepSeek-V4.1-Flash,
publisher local-inference-lab (signatures verified), sm_121, vocab 129,280,
concurrency 32 · 4 ranks, speculation mtp · widths 1,2,3,4,5,6.
Acquisition accounting: network 509,628,494,618 B, local source 509,653,222,270 B,
verification 509,646,855,013 B, materialized 0 B, elapsed 7682 s (2 h 08 m),
average 63 MiB/s (peak 104 MiB/s measured on du). No 2× leftover: the transport
cache was folded into blobs/ — it peaked at 475 G next to a growing blobs/, and at
the end transport/ is 88 K and blobs/ is the whole 475 G. So the earlier 2×-peak
warning applies to the middle of the run only, and models gc was not needed.
This changes the Phase 3 arithmetic, in our favour. A rank slice is
three files: initial-state.safetensors 1,321,056,820 B,
kernels.safetensors 39,429,126 B, graphs.safetensors 4,207,295 B. Measured directly on the
real files from dgx4 (dd iflag=direct, so no page-cache lying — note this
dd has no status=nocache, do not add it): the whole 1.3 GiB slice reads in
1.29 s at 1.1 GB/s (initial-state 1.242 s, kernels 0.038 s, graphs 0.006 s
at 689 MB/s). nconnect=8 with 4 MiB reads beats the earlier 1 MiB estimate of
660–820 MiB/s. So a NAS-backed deployments/ costs ~1.3 s per rank at load — the
small-file fear in §6 item 3 does not apply to this package, and Phase 3 has every
prospect of passing on the numbers, not by hope. model.safetensors is mode 0444, so
bind-mount it read-write and do not expect to replace it.
The runtime we are moving to renamed the artefact. Measured against both binaries on
dgx4 (/usr/bin/flare = Nov-5, /usr/local/share/hyperflare/v0.1.0/flare):
| Nov-5 build (installed) | v0.1.0 (staged) | |
|---|---|---|
| artefact name | package | core (“Serve a local core”, “Download a prepared core”) |
| serve/orbit flag | --package-cache |
--core-cache (run, orbit run, models list) |
| download flag | --model-cache |
--model-cache (unchanged — the two flags coexist, do not “fix” one into the other) |
| default store root | ~/.cache/flare/packages |
~/.cache/flare/cores |
| config key | paths.package_cache |
still paths.package_cache — verified: with package_cache set, models list reports that path under both binaries |
| store root layout | blobs/ refs/ staging/ locks/ sources/ origins/ models/ |
same, plus transport/ (the HF download cache) |
Consequences that actually bite:
migrat*-with-core/package string. An upgraded node simply looks somewhere else
unless paths.package_cache says otherwise. Since the fleet pins
package_cache in ~/.config/flare/config.toml, that key is what keeps the
upgrade from orphaning 474 GiB — do not let a node end up without it.grep -rn "package-cache" src/ → only a comment in flare_package.rs); the agent
synthesises it. So the flag rename travels with the binary swap, not with our
config. Confirm in the window anyway:
tr '\0' ' ' < /proc/$(pgrep -f "[f]lare run" | head -1)/cmdline.flare models is now a store manager — list, inspect, download, downloads,
select, remove, gc. Use models remove / models gc for the superseded build
instead of rm -rf, and models select to pin the default version.
models downloads inspects the transport cache. Open question for the
verification step: whether transport/ keeps a second copy of the payload (⇒
~950 GiB on the NAS until models gc); check du -sh /mnt/nas/ai/flare/store/*
and models downloads once the acquisition lands. Half-answered mid-flight: at
39 GiB downloaded, all of it sits in transport/ (40 K in locks, 24 K in
staging, 8 K in blobs) — so materialisation into models/ happens only after
the whole payload is cached, and staging/trees/<pid>-<n> accumulates one tree per
attempt (the killed Nov-5 attempt is still there as 1800521-0).flare trust exists now (revoke), for publisher trust management.23:14 while dgx4 22:14 for the same
instant). Never compare timestamps across nodes; filename your backups with the
local node's clock and compare mtime within a node only.Applied on dgx4 (the acquiring node) and dgx1 (prod rank 0):
package_cache = "/mnt/nas/ai/flare/store" # NAS store (2026-11-11, R4); was /home/kilork/flare-pkgs/gb10 (empty skeleton)
Backups: ~/.config/flare/config.toml.bak-2026-11-11-2214-nas-store (dgx4),
…-2314-nas-store (dgx1). Verify: flare models list prints
Storage /mnt/nas/ai/flare/store. dgx1's rank was untouched — same PID
(533700), 7.5-day uptime, api=200 — because the rank's cache path comes from the
agent's explicit flag, not from this key.
What the old value was doing: it deliberately equalled the recipe's local_dir
(/home/kilork/flare-pkgs/gb10) so that the launcher's rsync mirror would see
src == dst and skip copying, materialising the ~500 GB package once. In practice
the mirror never filled up (the agent writes to ~/.cache/flare/packages), so the
key pointed at a 68 K empty skeleton and flare models list reported “No local model
versions found” while 477 G sat elsewhere. With the store on the NAS, the
local_dir mirror is obsolete — retire it in the Phase 5 window rather than now, so
the launcher keeps a working path until the cutover.
Do this before touching dgx1–dgx3.
Use dgx4: it is idle (no ranks), and now boots from the new MP600 Mini.
The paths for this gate now exist — do not invent them. Source (NAS):
/mnt/nas/ai/flare/store/models/checkpoint-expanded-scales/gb10-tp4-202611120031/
with model.safetensors (0444), weights/ (470 G) and
deployments/db20209d361f473b8452f24ad517fa0d/rank-<N>/ (3 files, 1.3 G each).
Put the local copy at
~/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-202611120031/
(v0.1.0's default root is ~/.cache/flare/cores, but this fleet pins
paths.package_cache, so use whatever that key says on the node under test).
Copy one model's small artefacts from a running node:
ssh dgx8.lan 'tar -C ~/.cache/flare/models/<model>/<version> -cf - model.safetensors deployments' \
| tar -C /tmp -xf -
Build a fake local model dir whose weights/ is a symlink to nothing and whose
deployments/ is a bind mount from the NAS copy:
mkdir -p ~/.cache/flare/models/<model>/<version>/weights
mv /tmp/model.safetensors /tmp/deployments ~/.cache/flare/models/<model>/<version>/
sudo mkdir -p /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>
sudo mv ~/.cache/flare/models/<model>/<version>/deployments /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>/
sudo mount --bind /mnt/nas/ai/flare/nodes/dgx4/<model>/<version>/deployments \
~/.cache/flare/models/<model>/<version>/deployments
Ask FLARE what it sees:
flare inspect ~/.cache/flare/models/<model>/<version>/model.safetensors | head -20
flare models list
If a stage/launch is possible on dgx4 (flare run --stage-only with a dummy
agent), run it and then check the mount survived:
findmnt -no TARGET ~/.cache/flare/models/<model>/<version>/deployments
Empty output = FLARE replaced the directory → take option D.
Gate: flare inspect shows the deployment, and findmnt still reports the bind
mount after any stage operation.
What passed. With the exact Phase 4 shape — per-node artefacts on the NAS under
nodes/gx10-dgx4/<model>/<version>/deployments/<id>/, bind-mounted over the model
directory's deployments/ inside a store root — flare run --stage-only passed
Verify core and reached Wait for tensor-parallel hosts (the furthest a lone node
can get). The bind was unchanged before and after: same inode, same mount,
findmnt still reporting the NAS source. v0.1.0's inventory layer is also bind-safe:
flare models list over a locally-assembled root reported the full inventory
(474.66 GiB, 181 hardlinked objects) while blobs/ and weights/ were bind mounts
onto a different device (st_dev 80 = NFS vs 66306 = local NVMe).
Correction to what R5 concluded here (2026-11-12, R6). R5 blamed a missing store-root scaffolding for the
open core directory: No such file or directoryfailure. That was a misattribution. The real cause, proved by the negative control below: that error is what v0.1.0 reports whendeployments/<deployment-id>/rank-Nis missing or one level off. R5's failing binds pointed atdeployments/<id>asdeployments/, so the expectedrank-Nlevel vanished. Bind a directory that contains<id>/rank-Nand the very same tree verifies and serves. A store does indeed have no metadata files (purely structural), and the model dir does have to live inside its store — but do not go copyingblobs/ refs/ staging/around on the strength of the old wording; that was never what the error meant.
The plan's original Phase 3 step 1–2 procedure was still wrong in another way: a copied
model.safetensors alone is not a core. Working test shape (verified):
S=/mnt/nas/ai/flare/store; R2=$HOME/flare-bindtest # any local store root
mkdir -p $R2 && sudo mount --bind $S $R2 # root scaffolding
M=checkpoint-expanded-scales; V=gb10-tp4-202611120031
D=$R2/models/$M/$V; DEP=db20209d361f473b8452f24ad517fa0d
N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V # per-node artefacts on NAS
sudo mount --bind $N/deployments $D/deployments # per-node slice on top
flare run $D/model.safetensors --package-cache $R2 --rank 0 \
--rendezvous 127.0.0.1:29814 --gpus 0 --stage-only
findmnt -no TARGET,SOURCE $D/deployments # must still print the bind
Data-loss trap — this is the most important line in the document. A rm -rf over a
tree that still has live bind mounts deletes through them, into the NAS. That is how
the freshly acquired core's models/<m>/<v>/weights/ + deployments/ materialisation
was destroyed during test cleanup on dgx4: the rmdir of the two mount points failed
with EBUSY (which looks like “it refused”), while every file inside them was deleted
on the NAS. Unmount first, always, or make the refusal real:
findmnt -rno TARGET | grep -F "$D" | sort -r | xargs -r -n1 sudo umount # deepest first
sudo rm -rf --one-file-system "$D" # belt and braces
Nothing was lost, because the store is content-addressed: blobs/ still held the whole
475 G (link count dropped to 1), and nodes/gx10-dgx4/ still held all four
deployment slices (5.2 G). Recovery is just re-running flare download into the same
store — all blobs are local, so it re-materialises the hardlink farm instead of
re-downloading. Do not panic at Disk used 32.4 MiB in models list; check
du -sh <store>/blobs first.
Three measurement gotchas that look like corruption and are not:
du -sh a b c deduplicates hardlinks across its arguments — measured
du -sh $S/models $S/blobs as 475G + 6.2M, i.e. “the blobs vanished”, when in
fact each directory alone is 475 G. Measure one directory per call.actimeo=600 dir caching makes a live artefact look missing while test binds
are stacked (cp: cannot stat …/rank-0: No such file or directory); the file was
there the whole time. Unmount the binds before debugging.cp -al between two NAS paths failed with Invalid cross-device link even though
both were under /mnt/nas. Use cp -a for per-node artefacts (1.3 GiB, cheap).What is still unproven (R6b). strace shows FLARE opens with dirfd-relative paths —
openat(7, "tp4", O_RDONLY|O_NOFOLLOW|O_DIRECTORY) — so grepping strace for full paths
undercounts reads; the positive test that a deployment file was read through the
bind needs a completed stage, which needs the other three ranks. The O_NOFOLLOW in
that same line is the escape check, and it is why symlinks remain banned here. If a
completed stage turns out to rename over deployments/ or to hardlink into it
(cross-device hardlinks fail with EXDEV, as just demonstrated on NFS), take option D
without arguing: leave deployments/ local (1.3 GiB) and move only the package store.
Every “does FLARE tolerate X” question above was answered on one idle node with a 25 GB single-GPU core instead of waiting for the TP4 group. Keep this recipe — it is the cheapest way to test a storage or launch change:
# local-inference-lab/Qwen3.8-27B-FLARE, branch gb10 → 25.5 GB, 317 s at ~80 MiB/s
R=$HOME/teststore; mkdir -p $R
flare download local-inference-lab/Qwen3.8-27B-FLARE --tensor-parallel 1 \
--model-cache $R --publisher-keys ~/.cache/flare/publishers/937c0143*.json \
--token-file ~/.config/hf/token
# → model Qwen3.8-27B-NVIDIA-NVFP4-QAD-MXFP8 · gb10-tp1-<ts> · payload v2
# 23.77 GiB · 69 hardlinked objects · Deployment gb10-tp1 · TP 1 · ready
# context/prefill 262,144 / 4,096 · weights 22 G · deployments 2.5 G · envelope 18 M
flare run $R/models/<model>/<version>/model.safetensors --package-cache $R --gpus 0 \
--host 127.0.0.1 --port 30080 --publisher-keys ~/.cache/flare/publishers/937c0143*.json
curl -s http://127.0.0.1:30080/v1/models # 200 in ~20 s; chat completions work
Gotchas measured on the way: --rank and --rendezvous are rejected for a single-host
core (--rendezvous applies only to a multi-host core), the deployment id here is
human-readable (gb10-tp1) instead of a hash, this model streams into
reasoning_content with content: null, a 262k context makes the rank reserve ~109 GiB
of RAM (MemAvailable 117 → 8 GiB), and after pkill of a rank allow ~10 s for the
port to free before restarting (unmapping 109 GiB is not instant — restarting after 3 s
produced silent failures here).
Proven with it (deployments/ moved to nodes/<node>/<model>/<version>/deployments
on the NAS and bind-mounted over the model directory's deployments/):
api=200, 20 s (warm) / 21 s (cold, after drop_caches on the idle node);[FAILED] Verify core …
open core directory: No such file or directory, i.e. no hidden fallback to disk;find $D/deployments -newermt "$START" → 0 files,
inode identical, mount intact after serving, after restart, and after reboot;Boot persistence (fstab) — validated by rebooting idle dgx4, the exact stanza Phase 4
installs, and nothing else in /etc/fstab touched (backup /etc/fstab.bak-20261112-fsbind):
<mnt-nas>/ai/flare/nodes/<node>/<model>/<version>/deployments /home/kilork/.cache/flare/models/<model>/<version>/deployments none bind,nofail,x-systemd.mount-timeout=30,x-systemd.requires=/mnt/nas,x-systemd.requires-mounts-for=/mnt/nas 0 0
After reboot: findmnt shows the bind, systemctl show <bind-unit> -p Requires contains
mnt-nas.mount (so the bind can never be mounted before the NFS root), and a rank
started from it 22 s after boot with the bind intact. Removing it again: delete the
fstab line, daemon-reload, unmount deepest-first, then rm -rf --one-file-system
(see the trap above).
Repeat per node in rank order, one node at a time, verifying before moving on.
Stop the group (needs a working controller; see the re-pair note in
flare-bringup.md):
flare orbit stop <group> --cluster sparks
Move the node's small artefacts to NAS and bind them back:
V=gb10-tp4-<build>; M=checkpoint-expanded-scales
D=/home/kilork/.cache/flare/models/$M/$V
N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V
sudo mkdir -p "$N"
rsync -aH --remove-source-files "$D/model.safetensors" "$N/"
rsync -aH "$D/deployments/" "$N/deployments/" # cp -a/rsync, never cp -al: NAS↔NAS hardlinks can EXDEV
# NOTHING here may run while a bind is mounted at the target — rm deletes through it:
findmnt -rno TARGET | grep -F "$D" | sort -r | xargs -r -n1 sudo umount
rm -rf --one-file-system "$D/deployments"
sudo mount --bind "$N/deployments" "$D/deployments"
Persist it in /etc/fstab (bind mounts are ordinary fstab entries and are
mounted before the agent starts):
/mnt/nas/ai/flare/nodes/dgx5/checkpoint-expanded-scales/gb10-tp4-<build>/deployments /home/kilork/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-<build>/deployments none bind,nofail,x-systemd.requires=/mnt/nas,x-systemd.requires-mounts-for=/mnt/nas 0 0
Start the group and verify on the node:
findmnt -no TARGET,SOURCE ~/.cache/flare/models/$M/$V/deployments
du -sh ~/.cache/flare/models/$M/$V/* | sort -h # weights ~260G, rest tiny
pgrep -af "[f]lare run" | head -1
Only then move to the next node.
Gate: ranks serve, findmnt shows the bind mounts, du shows ~260 GiB of
weights/ and nothing else of size.
Same procedure with --cluster prod4. dgx1 is rank 0 and is the one that gains the
477 GiB; it is also the node whose packages/ phase 2 removed.
Verification, per node:
du -sh ~/.cache/flare/models/*/*/* # weights dominates
findmnt -no TARGET,SOURCE | grep deployments
df -h / /mnt/nas
flare orbit status --cluster sparks # agents reachable, groups listed
curl -s localhost:<port>/v1/models | head # API answers (port from the rank cmdline)
Rollback (per node, no data loss — the NAS copy is authoritative):
sudo umount ~/.cache/flare/models/$M/$V/deployments
sudo sed -i "\|$N/deployments|d" /etc/fstab
rsync -aH /mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V/deployments/ ~/.cache/flare/models/$M/$V/deployments/
flare orbit run <group> --cluster sparks
Decision: take the launch parameters from the current gb10 package, not from what
the lane happens to run today — with one deliberate exception, the API port stays
30000 (every lane here serves on 30000; 8000 in their example is just their host
convention).
Their recipe vs. the live rank 0 command line on dgx1 (tr '\0' ' ' <
/proc/$(pgrep -f "[f]lare run" | head -1)/cmdline):
| flag | upstream gb10 |
live | action |
|---|---|---|---|
--max-concurrency |
32 |
32 |
keep |
--mtp-draft-tokens |
5 |
5 |
keep |
--mtp-adaptive-verification |
yes | yes | keep |
--mtp-adaptive-verification-alpha |
0.75 |
absent | added to both recipes |
--embedding-table-storage |
disk |
disk |
keep |
--embedding-table-resident-scales |
yes | yes | keep |
--embedding-table-cache-size |
1GiB |
1073741824 |
equivalent (both = 1 GiB) |
--transfer-streams 1 |
absent | 1 |
keep ours — RoCE workaround, flare-roce-poison.md |
--host |
0.0.0.0 |
0.0.0.0 |
keep |
--port |
8000 |
30000 |
keep 30000 |
--served-model-name deepseek-v4.1-flash |
absent | set | keep ours (client-facing name) |
alpha 0.75 is theirs on purpose: “favor longer verification rounds, which raises
decode throughput with four or more concurrent requests and coding throughput” — i.e.
it helps exactly the lane shape we run (c32, TP4).
They also document memory/KV knobs (--memory-limit=-12G, --kv-cache-limit
memory|context|concurrency) and recommend leaving both at the default memory:
“Every GiB reserved or cached away from the default reduces the KV capacity that
Flare can plan.” Their stated budget — ~11.4 GiB KV per GPU, ~13.1 M tokens, 17–18 GiB
of 121 GiB left free — matches what we measure, so no change. The 1 GiB Engram row
cache is their recommended default; they suggest 4–8 GiB only if memory is spare and
the startup Plan cache line confirms room, and our own 48 GiB experiment failed with
“selected concurrency and verification width exceed available memory” — so 1 GiB
stays.
| risk | detection | fallback |
|---|---|---|
FLARE replaces deployments/ by rename; bind mount lost |
findmnt empty after a stage |
Option D: deployments local, only the package store moves |
| NAS stall at boot | node boots with /mnt/nas missing |
nofail (phase 1) keeps boot going; agent then writes deployments locally — re-sync after recovery |
| NAS outage during serving | reads hang (hard mount) |
bind mounts are only touched at load; keep timeo=100,retrans=3; do not put staging/ on NAS |
| hardlink-blind copy duplicates the package | du doubles after a copy |
always rsync -aH |
| mixed flare binaries after a restart | rendezvous incompatible protocol |
phase 0 rebuild first, restart all ranks together |
--package-cache path can be redirected by config or needs
a launcher extra_flags change (phase 2 note). Partly answered 2026-11-11, and it
is worse than assumed: on dgx1 the CLI's configured storage is
/home/kilork/flare-pkgs/gb10, an empty 68 K skeleton — flare models list
says “No local model versions found” and flare models inspect cannot see the
live package id — while the real 477 G store is in ~/.cache/flare/packages
because rank 0 is launched with --package-cache /home/kilork/.cache/flare. So
redirection works, but CLI and agent are already pointed at different trees on
one node. Unify them (launcher local_dir + agent flag) before either points
at the NAS, or the NAS store will inherit the same split-brain.dd iflag=direct, 2 GiB,
so no page-cache lying): the NAS is 10 GbE, not 1 GbE.| node | write 2 GiB | read 1 MiB | read 8 KiB | |---|---|---|---| | dgx8 (v4.1) | 659 MiB/s | 731 MiB/s | 28 MiB/s | | dgx1 (v4.0) | 663 MiB/s | 822 MiB/s | 32 MiB/s |
So a 1.8 GiB deployments/ directory reads in 2–3 s, and the protocol
choice costs nothing. The real constraint is small-file/short-read work: at
8 KiB the same share runs at ~30 MiB/s, so a deployments/ split into many
small files would load ~25× slower than the bulk figure suggests — count the
files in a rank deployment before trusting the 2–3 s number, and keep weights/
local regardless.
Package revision currency must be verified against the source before any storage
work (2026-11-11: the fleet served a superseded Nov-3 build whose source repo now
404s — see Phase 2R). To read what a local package is, do not trust the tree
layout; read the identity out of the payload (the flare.manifest tensor is
protobuf, but the strings are legible):
strings -a <pkg>/model.safetensors | grep -aoE "https://huggingface.co/[^ ]*" | sort -u
and compare with https://huggingface.co/api/models/<repo>/commits/<revision>.
The HF API masks a missing repo as 401 when unauthenticated, so always re-check
with a token before concluding a repo is gated. A valid token is
export HF_TOKEN=… in ~/.zshenv on the nodes — there is none on the mac.
flare-inference must be the latest release and actually running on every node.
Measured 2026-11-11: on dgx1/2/3/4/8 Installed == Candidate ==
0.1.0~git20261105002735.0009b89d-1 and /usr/bin/flare is the same file
(03c8e463) everywhere — but dgx1/2/3/8 execute deleted inodes (the PR #184
binaries), so only dgx4 is truly on the release build. The package's
≥ 20261105002735 requirement makes this a hard gate: no node can load the
current gb10 revision until it restarts onto the packaged binary.
PR #184 is not in the release (strings -a /usr/bin/flare | grep -c
"embedding read worker panicked" → 0), so running the latest build means giving up
the patched Engram reader until it is merged upstream.
Node root disks are not uniform — dgx1 is 915 GiB, dgx4 is 1.8 TiB. dgx1 was the one at 98 % full; a plan sized from dgx4's disk would have missed the whole reason this migration exists.
Package stores across dgx1–dgx3 share one id (7cf40bec…-35842073), and the layout
here is flat: du -l reports 0 hardlinks / 184 files, so rsync -H was
never load-bearing on this fleet.