Done and verified.
rw,hard,nofail,x-systemd.mount-timeout=30,…,timeo=100,retrans=3,… with a backup
/etc/fstab.bak-<date>-<time> on each node. Verified live (timeo=100 in
findmnt), sudo mount -a returns 0 without remounting.
Serving untouched: rank PIDs kept their 7-day uptime on dgx1/2/3/8 and
dgx1:30000/v1/models still answered afterwards.umount while it had zero open files. Then the real finding:
dgx1 cannot mount NFSv4.1 at all — a fresh vers=4.1 mount hangs with one
clean client and with nconnect=1, while dgx2/3/4/8 mount 4.1 instantly. Its
fstab is pinned to vers=4.0 (with the new timeouts). Do not "fix" it back to 4.1./tmp/flare-before-gx10-dgxN.txt and copied to /mnt/nas/ai/flare/. Includes the
exact rank launch cmdline (recovery recipe while the orbit controller is unpaired).deployments/ reads in
2–3 s; many-small-file deployments would load ~25× slower. v4.0 vs v4.1: no
measurable difference.Objective file updated: added ## Milestones, ## Round plan, ## Checks (the
check passes on all five powered nodes: fstab=1 live=1), rewrote Phase 1 (the
original blanket sed would have silently forced 4.1 onto dgx1 and hung it; and
nofail never shows up in live options, so verify with timeo=100), and filled in
§6.3.
R2 (Q1): Phase 2 first half — rsync -aH ~/.cache/flare/packages/
/mnt/nas/ai/flare/store/ from dgx1 (rank 0, the node holding the 477 GiB). At the
measured ~660 MiB/s write that is ~12 minutes, so run it in the foreground-ish
with --info=progress2 and then verify with
flare models list --model-cache /mnt/nas/ai/flare/store. Do not delete the
local copy in the same round — that is R3, gated on the models list result.
Watch out for:
echo 3 > /proc/sys/vm/drop_caches on a serving node — the 260 GiB of
weights are served from page cache. Use dd iflag=direct for any storage test.## Checks
command reports them as off).Goal quadrant: impact × effort — delete 783 GiB of dead data and find out whether the fleet is even on the current model build. Both landed.
The operator rejected migrating the old cache (“*Syncing those old flare cache to store does not make much sense honestly*”) and was right on the facts, harder than expected:
checkpoint-expanded-scales/gb10-tp4-202611021936
(261 GiB × 3 nodes) with nothing holding it open — verified per node via
/proc/<rank>/fd, /proc/<rank>/maps and lsof +D before deleting. Freed
783 GiB: dgx1 98 % full → 283 G free, dgx2 → 467 G, dgx3 → 470 G. Rank PIDs
untouched (7-day uptime), dgx1:30000/v1/models still answering.package_cache had silently drifted to an empty ~/flare-pkgs/gb10
skeleton: flare models list said “No local model versions found”, while the live
rank used ~/.cache/flare. Fix that before anything points at the NAS.The real finding: the fleet serves a superseded package from a repo that no longer
exists. The manifest inside model.safetensors names
…/DeepSeek-V4.1-Flash-1017-FLARE2 → 404 with a valid token (401 unauthenticated is
HF masking a missing repo). Current source: …/DeepSeek-V4.1-Flash-1017-FLARE, revision
= branch gb10, last commit 2026-11-05 “Publish the GB10 TP4 package with
vocabulary-parallel sampling and an FP8 Markov head”, 187 files / 474.7 GiB, and it
requires flare-inference ≥ 0.1.0~git20261105002735.0009b89d-1.
Consequences recorded in the plan (Phase 2R, ## Runtime parameters, §6 items 4–7):
0), and it
predates the package's minimum runtime → functionally unusable for the current
package, however fast. Prod = release build, everywhere. Installed+identical already
(03c8e463, Installed == Candidate on dgx1/2/3/4/8), but dgx1/2/3/8 execute
deleted inodes — only dgx4 truly runs the release; the restart window fixes that.--mtp-adaptive-verification-alpha 0.75 added to
both dsv41-flash-1017-4x-flare-gb10* recipes. --transfer-streams 1 (RoCE poison
workaround) and --served-model-name stay. --memory-limit / --kv-cache-limit:
upstream says keep the memory default, so nothing.flare download … --revision gb10 into the NAS store (474.7 GiB), verify
with flare models list --model-cache, then a window to cut over and delete the
superseded 477 G on dgx1 + 261 G dirs on the ranks.Skill updated (~/.agents/skills/flare): rule 1 now checks readlink /proc/<pid>/exe
for (deleted) — hash equality does not prove a node runs the installed build; rule 2
reversed to “prod runs the latest release build”, with the package-gates-runtime reason;
rule 5 adds “keep the flag set aligned with the package README, deviate only where
documented”.
Goal quadrant: impact × effort — acquire the current gb10 package onto the NAS
store (R3). The download is running; the surprise is that it could not have been run
with the runtime we had installed.
The Nov-5 apt build cannot acquire the current package. With /usr/bin/flare
(0.1.0~git20261105002735.0009b89d-1, installed == candidate on all five powered
nodes) the acquisition sits at Verify package, 0 B for minutes: it holds
store/locks/entries/checkpoint-expanded-scales--gb10-tp4-…lock and
store/locks/staging/<pid>-0.lock and makes one TLS handshake after another to
13.36.183.231 (HF's eu-west-3 LFS CDN) with rx_bytes flat. The same command
with upstream's tagged binary streams instantly. So the operator's instinct was right:
1105 is too old — and this is functional, not a lost optimisation.
Channels, established rather than assumed: github.com/NVIDIA/spark-local-inference
→ 301 → NVIDIA/hyperflare-inference (branch main, pushed 2026-11-11, 801 ★);
upstream publishes release archives from version tags (commit 94fbce5f6), tag
v0.1.0 (2026-11-11T19:31Z) with flare-linux-arm64.tgz = 11,427,812 B,
sha256 7ba5ba43…, unpacking to a 26,054,848 B flare. The APT repo lags: after a
clean apt-get update, apt-cache madison flare-inference offers only
20261105002735 / 20261030013211 / 20261027170349. Source build stays the backup path.
PR #184 is still not merged (embedding read worker panicked marker → 0 in v0.1.0),
which independently confirms dropping it.
What landed:
setsid nohup, v0.1.0 binary):
flare download … --revision gb10 --tensor-parallel 4 --model-cache
/mnt/nas/ai/flare/store --publisher-keys … --token-file … → 55–78 MiB/s into the NAS
store, 14 GiB by 22:10 (160 s). Upstream's example needs three things it does not
document: --tensor-parallel 4 (default is the local GPU count →
package exports no deployment with tensor-parallel size 1), non-interactive
publisher trust (flare download has no --dangerously-* flags: it wants
--publisher-keys + --trust-store; copied dgx1's key c8215a75… and the
fleet-uniform trust.toml 26e5ae90…), and --token-file for the LFS transport./usr/local/share/hyperflare/v0.1.0/flare — each node re-downloaded and checked the
archive sha256; resulting binary identical on all five (e3fe697e…, 26,054,848 B).
/usr/bin/flare and ~/.local/lib/flare-agent/flare untouched, so nothing serving
was touched. Documented the actual swap for the window (/usr/bin/flare is not what
runs — the agent executes ~/.local/lib/flare-agent/flare).pkill -f sglang killed my own ssh session (the pattern matches the remote
command line) — always self-exclude.## Checks runtime probe: staged=e3fe697e installed=03c8e463
running=…flare-agent/flare (deleted) on all four serving nodes, running= empty on
dgx4 (no rank) — exactly the three-way state the milestone needs to close.curl :30000 → 000 is normal),
so dgx2 api=000 is not an outage.Objective file: Phase 0.1 corrected with the channel table and the measured hang;
Phase 2R rewritten with the command that actually works plus the three undocumented
requirements; milestones/round plan updated; ## Checks gained the runtime probe.
Skill: SKILL.md rule 2 now ranks channels (GitHub tags → APT (lagging floor) →
source); references/build.md gained “Distribution channels (2026-11-11)” with the
staging recipe; references/troubleshooting.md gained the acquisition section (verify
hang, --tensor-parallel 4, publisher trust, token file, pkill self-match, rank-0-only
API).
Goal quadrant: impact × effort — close R3's verification and do R4 (the config drift) while the 475 GiB acquisition streams. R4 is done and verified; R3's gate is still streaming (see below).
The runtime we are moving to speaks a different vocabulary, measured not assumed.
Side by side on dgx4 (/usr/bin/flare Nov-5 vs the staged v0.1.0): “prepared package”
→ prepared “core”, --package-cache → --core-cache (on run, orbit run,
models list), default store root ~/.cache/flare/packages → ~/.cache/flare/cores,
and flare download keeps --model-cache (the two flags coexist — do not “fix” one
into the other). But paths.package_cache is unchanged: with that key set, both
binaries report the configured path from models list. And nothing migrates
automatically — neither binary has a migrat* string for core/package. So that one
config key is what prevents an upgrade from orphaning hundreds of GB.
R4 applied: package_cache = "/mnt/nas/ai/flare/store" on dgx4 (acquiring node)
and dgx1 (prod rank 0), replacing the value that pointed at
/home/kilork/flare-pkgs/gb10 — a 68 K empty skeleton. Backups
config.toml.bak-2026-11-11-2214-nas-store (dgx4) / …-2314-nas-store (dgx1).
Verified: flare models list → Storage /mnt/nas/ai/flare/store, and dgx1's rank is
untouched — same PID 533700, 7.5-day uptime, api=200 — because the rank's cache
path comes from the agent's explicit flag, not this key. The old value was deliberate
(it equalled the recipe's local_dir so the rsync mirror saw src == dst), but the
mirror never filled, so the key was pointing at nothing; retire local_dir in the
Phase 5 window, not now.
The launcher needs no change for the flag rename — grep -rn "package-cache" src/
finds only a comment; the agent synthesises the flag, so it travels with the binary
swap. Still to confirm in the window: the launched cmdline itself (open item 6.1).
Store mechanics learned mid-flight, which change the verification step: at 39 GiB
downloaded, all bytes sit in <store>/transport/ — models/, blobs/, refs/ are
still empty skeletons, and staging/trees/<pid>-<n> has one tree per attempt (the
killed Nov-5 attempt is still there). So materialisation happens only after the whole
payload is cached, expect a peak near 2× the payload, and flare models gc is the tool
for it — v0.1.0's flare models gained download · downloads · select · remove · gc,
i.e. real store management (models remove/gc beat rm -rf, models select pins a
default version). flare trust also exists now.
Node clocks disagree by an hour (dgx1 23:14 vs dgx4 22:14 at the same instant) —
never compare timestamps across nodes.
Acquisition state at last check: 38.1 GiB at 420 s, 93 MiB/s, alive, one process,
--tensor-parallel 4, target ≈ 23:32 UTC. R3's gate (models list showing the version)
is the next round's first action if it does not land this round.
Objective file: new “v0.1.0 vocabulary and paths” section (side-by-side table + the five
consequences that bite), “R4 — CLI store pointed at the NAS store” (applied value,
backup paths, verification, why the old value existed), R4 ticked. Skill
references/storage.md: same vocabulary table, the no-migration warning, the agent
synthesises-the-flag note, the models subcommands, and the transport-first layout with
its 2× peak.
gb10 core is on the NAS and verifiedR3's gate passed. The acquisition that started 22:07 UTC finished at 00:16 UTC
(7682 s, avg 63 MiB/s, peak 104) and flare models list now reports, from both dgx4
and dgx1 (prod) over NFS:
checkpoint-expanded-scales gb10-tp4-202611112207 [DEFAULT]
Format payload v2 · Disk used 474.66 GiB · Shared files 474.63 GiB · 181 hardlinked objects
Deployment db20209d361f473b8452f24ad517fa0d · TP 4 · ready
Context / prefill 1,048,576 / 8,192 tokens · Pages / state rows 128 / 448
models inspect confirms identity: DeepSeek-V4.1-Flash, local-inference-lab
(signatures verified), sm_121, vocab 129,280, concurrency 32 · 4 ranks,
speculation mtp · widths 1–6.
Two numbers in the plan were wrong and are now measured. (1) The store does not
keep a 2× copy: transport/ peaked at 475 G while blobs/ grew beside it, and at the
end transport/ is 88 K and blobs/ is the whole 475 G — so no models gc was needed,
and the 2× warning only applies mid-run. (2) A rank slice is three files —
initial-state.safetensors 1,321,056,820 B + kernels.safetensors 39,429,126 B +
graphs.safetensors 4,207,295 B = 1.3 GiB each, deployments/ 5.1 G total,
weights/ 470 G, model.safetensors 33 M at mode 0444. At our measured NAS read
rates that is ~2 s per rank, so the small-file worry in §6 item 3 simply does not
apply to this package — the bind-mount phase is viable on the numbers, not by hope.
Serving survived the 2 h of NAS writes untouched: dgx1/2/3/8 all ranks=1, dgx1
api=200, uptime continuing, every mount still timeo=100.
Milestones restructured honestly: acquisition done, deployment of the core + deletion of the superseded build still open (it needs the window — dgx1's 477 G and the ranks' 261 G dirs are what is actually serving).
Next: R6 (Phase 3 gate) on dgx4 — now unblocked, with real paths in the doc; then the window (R7/R8) which is also what lands v0.1.0 as the running binary and frees the superseded 477 G.
Round goal was R6 (Phase 3 gate) on idle dgx4. Half of it is now proven, and the plan gained its most important safety rule.
Proven on dgx4 with the staged v0.1.0 binary, using the real Phase 4 shape (per-node
artefacts under nodes/gx10-dgx4/<model>/<version>/deployments/<id>/ on the NAS,
bind-mounted over the model directory's deployments/ inside a store root):
flare run --stage-only passed Verify core and reached Wait for tensor-parallel
hosts, and the bind was byte-identical before and after five stage attempts (same inode,
findmnt still showing the NAS source). The inventory layer is bind-safe too: models
list over a locally-assembled root reported 474.66 GiB · 181 hardlinked objects while
blobs/ and weights/ were bind mounts onto a different device (NFS st_dev 80 vs
local NVMe 66306). strace explains the symlink ban precisely: the escape check is
openat(7, "tp4", O_RDONLY|O_NOFOLLOW|O_DIRECTORY).
The plan's own Phase 3 procedure was invalid and is rewritten. Copying
model.safetensors + deployments out of a store and pointing run/inspect at the
copy fails in v0.1.0 with open core directory: No such file or directory (os error 2) —
the envelope resolves its core within the store that contains it, and a store has no
metadata files at all (purely structural), so the model path must live inside a coherent
store root with blobs/ refs/ sources/ origins/ staging/ locks/ present. The verified
command shape is in the doc.
Then I destroyed the core's materialisation. Cleanup ran rm -rf over a tree that
still had two live bind mounts into the NAS store. rm unmounts nothing: the rmdir of
the mount points failed EBUSY — which reads like a refusal — while every file inside
them was deleted on the NAS. models list went to Disk used 32.4 MiB. No data was
lost: the store is content-addressed, so blobs/ still held the whole 475 G (link
count 1) and nodes/gx10-dgx4/ still held all four deployment slices. Recovery is
flare download into the same store, which re-links from local blobs; it is running now
(pid 2037594, ~/flare-rematerialise.log). There is no verify/repair subcommand —
download is the repair path.
Three false alarms that look like corruption, all documented so the next round does not
chase them: du -sh a b dedups hardlinks across arguments (that is how blobs read
as 6.2M before I had actually broken anything); NFS actimeo=600 dir caching made a
live rank-0 look missing while binds were stacked; and cp -al between two NAS paths
fails Invalid cross-device link — per-node artefacts need cp -a.
Rule added to the skill as hard rule 6 (unmount deepest-first before recursive deletes,
rm -rf --one-file-system, then blob-first triage), plus a store-triage section in
references/troubleshooting.md and a validated bind-layout section in
references/storage.md. R6 is split: R6a done, R6b needs a completed TP4
stage (peers ⇒ the window) to prove deployment reads traverse the bind and that FLARE
never renames over / hardlinks into deployments/ — either would take option D.
Serving untouched all round: dgx1/2/3/8 ranks=1, dgx1 api=200. dgx4 has 1.5 T free
and no ranks, and now holds the first real nodes/<node>/… artefacts (5.2 G).
R5 is done, and it is not just paperwork — it produced the pre-cutover runtime table and the mechanism behind the fleet's mystery:
dgx1 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
dgx2 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
dgx3 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
dgx4 staged=e3fe697e /usrbin=e3fe697e agent=e3fe697e exe=/home/kilork/.local/lib/flare-agent/flare
dgx8 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
Every serving node runs a (deleted) inode, i.e. production is running neither the
Nov-5 apt build nor v0.1.0 — it is the pre-apt (PR-184) binary held alive by the
process. The reason apt never fixed this is now proven instead of inferred: the unit
ExecStarts %h/.local/lib/flare-agent/flare and dpkg -S owns no such file, so every
apt upgrade moved /usr/bin/flare and left the running path alone. The ## Checks
probe prints agent= for exactly this reason — it is the only column describing what
executes.
Then the swap was rehearsed on idle dgx4 (temp+mv at both paths,
systemctl --user restart flare-agent.service, sha + readlink /proc/PID/exe
verification, flare.orig-20261112 backups) and turned into
host-scripts/flare-swap.sh so the window runs one reviewed command per node
instead of hand-typed lines. Its two tests both behaved: on dgx4 it was a verified
no-op with exit 0; on dgx8, with a rank running, it refused (exit 1) leaving shas
unchanged, zero tmp-swap files behind, and the rank untouched. That refusal is the
behaviour that protects prod during a window.
A trap for the window, found by reading the numbers again: the new core's weights/
is 470 G, not the 260 GiB this plan was written against. dgx1's root is 915 G with
283 G free, and its 477 G package store is only released when rank 0 restarts — so dgx1
comes to ~760 G free during the window, and if a rank turns out to need the full 470 G
locally, that is the only node where the arithmetic is close. Hence a new gate: measure
the real per-rank footprint on dgx4 (1.5 T free) before rolling sparks/prod4, and
re-measure the free-space table the day of the window.
Repair of my own damage is still running healthily on dgx4 (pid 2037594, read_bytes
+370 MB/s against a 475 G blob re-verification, NIC idle — the progress bar counts
network bytes only, which is why it reads 32.4 MiB the whole time). When it lands,
models list returns to 474.66 GiB.