Done and verified.
rw,hard,nofail,x-systemd.mount-timeout=30,…,timeo=100,retrans=3,… with a backup
/etc/fstab.bak-<date>-<time> on each node. Verified live (timeo=100 in
findmnt), sudo mount -a returns 0 without remounting.
Serving untouched: rank PIDs kept their 7-day uptime on dgx1/2/3/8 and
dgx1:30000/v1/models still answered afterwards.umount while it had zero open files. Then the real finding:
dgx1 cannot mount NFSv4.1 at all — a fresh vers=4.1 mount hangs with one
clean client and with nconnect=1, while dgx2/3/4/8 mount 4.1 instantly. Its
fstab is pinned to vers=4.0 (with the new timeouts). Do not "fix" it back to 4.1./tmp/flare-before-gx10-dgxN.txt and copied to /mnt/nas/ai/flare/. Includes the
exact rank launch cmdline (recovery recipe while the orbit controller is unpaired).deployments/ reads in
2–3 s; many-small-file deployments would load ~25× slower. v4.0 vs v4.1: no
measurable difference.Objective file updated: added ## Milestones, ## Round plan, ## Checks (the
check passes on all five powered nodes: fstab=1 live=1), rewrote Phase 1 (the
original blanket sed would have silently forced 4.1 onto dgx1 and hung it; and
nofail never shows up in live options, so verify with timeo=100), and filled in
§6.3.
R2 (Q1): Phase 2 first half — rsync -aH ~/.cache/flare/packages/
/mnt/nas/ai/flare/store/ from dgx1 (rank 0, the node holding the 477 GiB). At the
measured ~660 MiB/s write that is ~12 minutes, so run it in the foreground-ish
with --info=progress2 and then verify with
flare models list --model-cache /mnt/nas/ai/flare/store. Do not delete the
local copy in the same round — that is R3, gated on the models list result.
Watch out for:
echo 3 > /proc/sys/vm/drop_caches on a serving node — the 260 GiB of
weights are served from page cache. Use dd iflag=direct for any storage test.## Checks
command reports them as off).Goal quadrant: impact × effort — delete 783 GiB of dead data and find out whether the fleet is even on the current model build. Both landed.
The operator rejected migrating the old cache (“*Syncing those old flare cache to store does not make much sense honestly*”) and was right on the facts, harder than expected:
checkpoint-expanded-scales/gb10-tp4-202611021936
(261 GiB × 3 nodes) with nothing holding it open — verified per node via
/proc/<rank>/fd, /proc/<rank>/maps and lsof +D before deleting. Freed
783 GiB: dgx1 98 % full → 283 G free, dgx2 → 467 G, dgx3 → 470 G. Rank PIDs
untouched (7-day uptime), dgx1:30000/v1/models still answering.package_cache had silently drifted to an empty ~/flare-pkgs/gb10
skeleton: flare models list said “No local model versions found”, while the live
rank used ~/.cache/flare. Fix that before anything points at the NAS.The real finding: the fleet serves a superseded package from a repo that no longer
exists. The manifest inside model.safetensors names
…/DeepSeek-V4.1-Flash-1017-FLARE2 → 404 with a valid token (401 unauthenticated is
HF masking a missing repo). Current source: …/DeepSeek-V4.1-Flash-1017-FLARE, revision
= branch gb10, last commit 2026-11-05 “Publish the GB10 TP4 package with
vocabulary-parallel sampling and an FP8 Markov head”, 187 files / 474.7 GiB, and it
requires flare-inference ≥ 0.1.0~git20261105002735.0009b89d-1.
Consequences recorded in the plan (Phase 2R, ## Runtime parameters, §6 items 4–7):
0), and it
predates the package's minimum runtime → functionally unusable for the current
package, however fast. Prod = release build, everywhere. Installed+identical already
(03c8e463, Installed == Candidate on dgx1/2/3/4/8), but dgx1/2/3/8 execute
deleted inodes — only dgx4 truly runs the release; the restart window fixes that.--mtp-adaptive-verification-alpha 0.75 added to
both dsv41-flash-1017-4x-flare-gb10* recipes. --transfer-streams 1 (RoCE poison
workaround) and --served-model-name stay. --memory-limit / --kv-cache-limit:
upstream says keep the memory default, so nothing.flare download … --revision gb10 into the NAS store (474.7 GiB), verify
with flare models list --model-cache, then a window to cut over and delete the
superseded 477 G on dgx1 + 261 G dirs on the ranks.Skill updated (~/.agents/skills/flare): rule 1 now checks readlink /proc/<pid>/exe
for (deleted) — hash equality does not prove a node runs the installed build; rule 2
reversed to “prod runs the latest release build”, with the package-gates-runtime reason;
rule 5 adds “keep the flag set aligned with the package README, deviate only where
documented”.
Goal quadrant: impact × effort — acquire the current gb10 package onto the NAS
store (R3). The download is running; the surprise is that it could not have been run
with the runtime we had installed.
The Nov-5 apt build cannot acquire the current package. With /usr/bin/flare
(0.1.0~git20261105002735.0009b89d-1, installed == candidate on all five powered
nodes) the acquisition sits at Verify package, 0 B for minutes: it holds
store/locks/entries/checkpoint-expanded-scales--gb10-tp4-…lock and
store/locks/staging/<pid>-0.lock and makes one TLS handshake after another to
13.36.183.231 (HF's eu-west-3 LFS CDN) with rx_bytes flat. The same command
with upstream's tagged binary streams instantly. So the operator's instinct was right:
1105 is too old — and this is functional, not a lost optimisation.
Channels, established rather than assumed: github.com/NVIDIA/spark-local-inference
→ 301 → NVIDIA/hyperflare-inference (branch main, pushed 2026-11-11, 801 ★);
upstream publishes release archives from version tags (commit 94fbce5f6), tag
v0.1.0 (2026-11-11T19:31Z) with flare-linux-arm64.tgz = 11,427,812 B,
sha256 7ba5ba43…, unpacking to a 26,054,848 B flare. The APT repo lags: after a
clean apt-get update, apt-cache madison flare-inference offers only
20261105002735 / 20261030013211 / 20261027170349. Source build stays the backup path.
PR #184 is still not merged (embedding read worker panicked marker → 0 in v0.1.0),
which independently confirms dropping it.
What landed:
setsid nohup, v0.1.0 binary):
flare download … --revision gb10 --tensor-parallel 4 --model-cache
/mnt/nas/ai/flare/store --publisher-keys … --token-file … → 55–78 MiB/s into the NAS
store, 14 GiB by 22:10 (160 s). Upstream's example needs three things it does not
document: --tensor-parallel 4 (default is the local GPU count →
package exports no deployment with tensor-parallel size 1), non-interactive
publisher trust (flare download has no --dangerously-* flags: it wants
--publisher-keys + --trust-store; copied dgx1's key c8215a75… and the
fleet-uniform trust.toml 26e5ae90…), and --token-file for the LFS transport./usr/local/share/hyperflare/v0.1.0/flare — each node re-downloaded and checked the
archive sha256; resulting binary identical on all five (e3fe697e…, 26,054,848 B).
/usr/bin/flare and ~/.local/lib/flare-agent/flare untouched, so nothing serving
was touched. Documented the actual swap for the window (/usr/bin/flare is not what
runs — the agent executes ~/.local/lib/flare-agent/flare).pkill -f sglang killed my own ssh session (the pattern matches the remote
command line) — always self-exclude.## Checks runtime probe: staged=e3fe697e installed=03c8e463
running=…flare-agent/flare (deleted) on all four serving nodes, running= empty on
dgx4 (no rank) — exactly the three-way state the milestone needs to close.curl :30000 → 000 is normal),
so dgx2 api=000 is not an outage.Objective file: Phase 0.1 corrected with the channel table and the measured hang;
Phase 2R rewritten with the command that actually works plus the three undocumented
requirements; milestones/round plan updated; ## Checks gained the runtime probe.
Skill: SKILL.md rule 2 now ranks channels (GitHub tags → APT (lagging floor) →
source); references/build.md gained “Distribution channels (2026-11-11)” with the
staging recipe; references/troubleshooting.md gained the acquisition section (verify
hang, --tensor-parallel 4, publisher trust, token file, pkill self-match, rank-0-only
API).
Goal quadrant: impact × effort — close R3's verification and do R4 (the config drift) while the 475 GiB acquisition streams. R4 is done and verified; R3's gate is still streaming (see below).
The runtime we are moving to speaks a different vocabulary, measured not assumed.
Side by side on dgx4 (/usr/bin/flare Nov-5 vs the staged v0.1.0): “prepared package”
→ prepared “core”, --package-cache → --core-cache (on run, orbit run,
models list), default store root ~/.cache/flare/packages → ~/.cache/flare/cores,
and flare download keeps --model-cache (the two flags coexist — do not “fix” one
into the other). But paths.package_cache is unchanged: with that key set, both
binaries report the configured path from models list. And nothing migrates
automatically — neither binary has a migrat* string for core/package. So that one
config key is what prevents an upgrade from orphaning hundreds of GB.
R4 applied: package_cache = "/mnt/nas/ai/flare/store" on dgx4 (acquiring node)
and dgx1 (prod rank 0), replacing the value that pointed at
/home/kilork/flare-pkgs/gb10 — a 68 K empty skeleton. Backups
config.toml.bak-2026-11-11-2214-nas-store (dgx4) / …-2314-nas-store (dgx1).
Verified: flare models list → Storage /mnt/nas/ai/flare/store, and dgx1's rank is
untouched — same PID 533700, 7.5-day uptime, api=200 — because the rank's cache
path comes from the agent's explicit flag, not this key. The old value was deliberate
(it equalled the recipe's local_dir so the rsync mirror saw src == dst), but the
mirror never filled, so the key was pointing at nothing; retire local_dir in the
Phase 5 window, not now.
The launcher needs no change for the flag rename — grep -rn "package-cache" src/
finds only a comment; the agent synthesises the flag, so it travels with the binary
swap. Still to confirm in the window: the launched cmdline itself (open item 6.1).
Store mechanics learned mid-flight, which change the verification step: at 39 GiB
downloaded, all bytes sit in <store>/transport/ — models/, blobs/, refs/ are
still empty skeletons, and staging/trees/<pid>-<n> has one tree per attempt (the
killed Nov-5 attempt is still there). So materialisation happens only after the whole
payload is cached, expect a peak near 2× the payload, and flare models gc is the tool
for it — v0.1.0's flare models gained download · downloads · select · remove · gc,
i.e. real store management (models remove/gc beat rm -rf, models select pins a
default version). flare trust also exists now.
Node clocks disagree by an hour (dgx1 23:14 vs dgx4 22:14 at the same instant) —
never compare timestamps across nodes.
Acquisition state at last check: 38.1 GiB at 420 s, 93 MiB/s, alive, one process,
--tensor-parallel 4, target ≈ 23:32 UTC. R3's gate (models list showing the version)
is the next round's first action if it does not land this round.
Objective file: new “v0.1.0 vocabulary and paths” section (side-by-side table + the five
consequences that bite), “R4 — CLI store pointed at the NAS store” (applied value,
backup paths, verification, why the old value existed), R4 ticked. Skill
references/storage.md: same vocabulary table, the no-migration warning, the agent
synthesises-the-flag note, the models subcommands, and the transport-first layout with
its 2× peak.
gb10 core is on the NAS and verifiedR3's gate passed. The acquisition that started 22:07 UTC finished at 00:16 UTC
(7682 s, avg 63 MiB/s, peak 104) and flare models list now reports, from both dgx4
and dgx1 (prod) over NFS:
checkpoint-expanded-scales gb10-tp4-202611112207 [DEFAULT]
Format payload v2 · Disk used 474.66 GiB · Shared files 474.63 GiB · 181 hardlinked objects
Deployment db20209d361f473b8452f24ad517fa0d · TP 4 · ready
Context / prefill 1,048,576 / 8,192 tokens · Pages / state rows 128 / 448
models inspect confirms identity: DeepSeek-V4.1-Flash, local-inference-lab
(signatures verified), sm_121, vocab 129,280, concurrency 32 · 4 ranks,
speculation mtp · widths 1–6.
Two numbers in the plan were wrong and are now measured. (1) The store does not
keep a 2× copy: transport/ peaked at 475 G while blobs/ grew beside it, and at the
end transport/ is 88 K and blobs/ is the whole 475 G — so no models gc was needed,
and the 2× warning only applies mid-run. (2) A rank slice is three files —
initial-state.safetensors 1,321,056,820 B + kernels.safetensors 39,429,126 B +
graphs.safetensors 4,207,295 B = 1.3 GiB each, deployments/ 5.1 G total,
weights/ 470 G, model.safetensors 33 M at mode 0444. At our measured NAS read
rates that is ~2 s per rank, so the small-file worry in §6 item 3 simply does not
apply to this package — the bind-mount phase is viable on the numbers, not by hope.
Serving survived the 2 h of NAS writes untouched: dgx1/2/3/8 all ranks=1, dgx1
api=200, uptime continuing, every mount still timeo=100.
Milestones restructured honestly: acquisition done, deployment of the core + deletion of the superseded build still open (it needs the window — dgx1's 477 G and the ranks' 261 G dirs are what is actually serving).
Next: R6 (Phase 3 gate) on dgx4 — now unblocked, with real paths in the doc; then the window (R7/R8) which is also what lands v0.1.0 as the running binary and frees the superseded 477 G.
Round goal was R6 (Phase 3 gate) on idle dgx4. Half of it is now proven, and the plan gained its most important safety rule.
Proven on dgx4 with the staged v0.1.0 binary, using the real Phase 4 shape (per-node
artefacts under nodes/gx10-dgx4/<model>/<version>/deployments/<id>/ on the NAS,
bind-mounted over the model directory's deployments/ inside a store root):
flare run --stage-only passed Verify core and reached Wait for tensor-parallel
hosts, and the bind was byte-identical before and after five stage attempts (same inode,
findmnt still showing the NAS source). The inventory layer is bind-safe too: models
list over a locally-assembled root reported 474.66 GiB · 181 hardlinked objects while
blobs/ and weights/ were bind mounts onto a different device (NFS st_dev 80 vs
local NVMe 66306). strace explains the symlink ban precisely: the escape check is
openat(7, "tp4", O_RDONLY|O_NOFOLLOW|O_DIRECTORY).
The plan's own Phase 3 procedure was invalid and is rewritten. Copying
model.safetensors + deployments out of a store and pointing run/inspect at the
copy fails in v0.1.0 with open core directory: No such file or directory (os error 2) —
the envelope resolves its core within the store that contains it, and a store has no
metadata files at all (purely structural), so the model path must live inside a coherent
store root with blobs/ refs/ sources/ origins/ staging/ locks/ present. The verified
command shape is in the doc.
Then I destroyed the core's materialisation. Cleanup ran rm -rf over a tree that
still had two live bind mounts into the NAS store. rm unmounts nothing: the rmdir of
the mount points failed EBUSY — which reads like a refusal — while every file inside
them was deleted on the NAS. models list went to Disk used 32.4 MiB. No data was
lost: the store is content-addressed, so blobs/ still held the whole 475 G (link
count 1) and nodes/gx10-dgx4/ still held all four deployment slices. Recovery is
flare download into the same store, which re-links from local blobs; it is running now
(pid 2037594, ~/flare-rematerialise.log). There is no verify/repair subcommand —
download is the repair path.
Three false alarms that look like corruption, all documented so the next round does not
chase them: du -sh a b dedups hardlinks across arguments (that is how blobs read
as 6.2M before I had actually broken anything); NFS actimeo=600 dir caching made a
live rank-0 look missing while binds were stacked; and cp -al between two NAS paths
fails Invalid cross-device link — per-node artefacts need cp -a.
Rule added to the skill as hard rule 6 (unmount deepest-first before recursive deletes,
rm -rf --one-file-system, then blob-first triage), plus a store-triage section in
references/troubleshooting.md and a validated bind-layout section in
references/storage.md. R6 is split: R6a done, R6b needs a completed TP4
stage (peers ⇒ the window) to prove deployment reads traverse the bind and that FLARE
never renames over / hardlinks into deployments/ — either would take option D.
Serving untouched all round: dgx1/2/3/8 ranks=1, dgx1 api=200. dgx4 has 1.5 T free
and no ranks, and now holds the first real nodes/<node>/… artefacts (5.2 G).
R5 is done, and it is not just paperwork — it produced the pre-cutover runtime table and the mechanism behind the fleet's mystery:
dgx1 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
dgx2 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
dgx3 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
dgx4 staged=e3fe697e /usrbin=e3fe697e agent=e3fe697e exe=/home/kilork/.local/lib/flare-agent/flare
dgx8 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
Every serving node runs a (deleted) inode, i.e. production is running neither the
Nov-5 apt build nor v0.1.0 — it is the pre-apt (PR-184) binary held alive by the
process. The reason apt never fixed this is now proven instead of inferred: the unit
ExecStarts %h/.local/lib/flare-agent/flare and dpkg -S owns no such file, so every
apt upgrade moved /usr/bin/flare and left the running path alone. The ## Checks
probe prints agent= for exactly this reason — it is the only column describing what
executes.
Then the swap was rehearsed on idle dgx4 (temp+mv at both paths,
systemctl --user restart flare-agent.service, sha + readlink /proc/PID/exe
verification, flare.orig-20261112 backups) and turned into
host-scripts/flare-swap.sh so the window runs one reviewed command per node
instead of hand-typed lines. Its two tests both behaved: on dgx4 it was a verified
no-op with exit 0; on dgx8, with a rank running, it refused (exit 1) leaving shas
unchanged, zero tmp-swap files behind, and the rank untouched. That refusal is the
behaviour that protects prod during a window.
A trap for the window, found by reading the numbers again: the new core's weights/
is 470 G, not the 260 GiB this plan was written against. dgx1's root is 915 G with
283 G free, and its 477 G package store is only released when rank 0 restarts — so dgx1
comes to ~760 G free during the window, and if a rank turns out to need the full 470 G
locally, that is the only node where the arithmetic is close. Hence a new gate: measure
the real per-rank footprint on dgx4 (1.5 T free) before rolling sparks/prod4, and
re-measure the free-space table the day of the window.
Repair of my own damage is still running healthily on dgx4 (pid 2037594, read_bytes
+370 MB/s against a 475 G blob re-verification, NIC idle — the progress bar counts
network bytes only, which is why it reads 32.4 MiB the whole time). When it lands,
models list returns to 474.66 GiB.
Recovery confirmed inside the same round. The repair finished at 1251 s:
network 33,966,105 bytes · local source 509,653,222,270 · verification 509,646,855,013
— i.e. 34 MB re-downloaded, 509.6 GB re-read from local blobs. The store now shows
1 model · 1 version, [DEFAULT] gb10-tp4-202611120031, 474.66 GiB,
181 hardlinked objects, deployment db20209d361f473b8452f24ad517fa0d · TP 4 · ready.
Two durable facts: version ids are mutable (a re-materialisation mints a new one, so
scripts must query rather than hardcode) and deployment ids are content-derived and
stable (every reference to db20209d… in the plan stayed valid). flare models remove
wants MODEL@VERSION, not MODEL/VERSION. Per-node artefacts were rebuilt for the new
version (5.1 G). NAS at 1.6 T/33 T. Serving untouched throughout: dgx1/2/3/8 ranks=1,
dgx1 api=200.