Done and verified.
rw,hard,nofail,x-systemd.mount-timeout=30,…,timeo=100,retrans=3,… with a backup
/etc/fstab.bak-<date>-<time> on each node. Verified live (timeo=100 in
findmnt), sudo mount -a returns 0 without remounting.
Serving untouched: rank PIDs kept their 7-day uptime on dgx1/2/3/8 and
dgx1:30000/v1/models still answered afterwards.umount while it had zero open files. Then the real finding:
dgx1 cannot mount NFSv4.1 at all — a fresh vers=4.1 mount hangs with one
clean client and with nconnect=1, while dgx2/3/4/8 mount 4.1 instantly. Its
fstab is pinned to vers=4.0 (with the new timeouts). Do not "fix" it back to 4.1./tmp/flare-before-gx10-dgxN.txt and copied to /mnt/nas/ai/flare/. Includes the
exact rank launch cmdline (recovery recipe while the orbit controller is unpaired).deployments/ reads in
2–3 s; many-small-file deployments would load ~25× slower. v4.0 vs v4.1: no
measurable difference.Objective file updated: added ## Milestones, ## Round plan, ## Checks (the
check passes on all five powered nodes: fstab=1 live=1), rewrote Phase 1 (the
original blanket sed would have silently forced 4.1 onto dgx1 and hung it; and
nofail never shows up in live options, so verify with timeo=100), and filled in
§6.3.
R2 (Q1): Phase 2 first half — rsync -aH ~/.cache/flare/packages/
/mnt/nas/ai/flare/store/ from dgx1 (rank 0, the node holding the 477 GiB). At the
measured ~660 MiB/s write that is ~12 minutes, so run it in the foreground-ish
with --info=progress2 and then verify with
flare models list --model-cache /mnt/nas/ai/flare/store. Do not delete the
local copy in the same round — that is R3, gated on the models list result.
Watch out for:
echo 3 > /proc/sys/vm/drop_caches on a serving node — the 260 GiB of
weights are served from page cache. Use dd iflag=direct for any storage test.## Checks
command reports them as off).Goal quadrant: impact × effort — delete 783 GiB of dead data and find out whether the fleet is even on the current model build. Both landed.
The operator rejected migrating the old cache (“*Syncing those old flare cache to store does not make much sense honestly*”) and was right on the facts, harder than expected:
checkpoint-expanded-scales/gb10-tp4-202611021936
(261 GiB × 3 nodes) with nothing holding it open — verified per node via
/proc/<rank>/fd, /proc/<rank>/maps and lsof +D before deleting. Freed
783 GiB: dgx1 98 % full → 283 G free, dgx2 → 467 G, dgx3 → 470 G. Rank PIDs
untouched (7-day uptime), dgx1:30000/v1/models still answering.package_cache had silently drifted to an empty ~/flare-pkgs/gb10
skeleton: flare models list said “No local model versions found”, while the live
rank used ~/.cache/flare. Fix that before anything points at the NAS.The real finding: the fleet serves a superseded package from a repo that no longer
exists. The manifest inside model.safetensors names
…/DeepSeek-V4.1-Flash-1017-FLARE2 → 404 with a valid token (401 unauthenticated is
HF masking a missing repo). Current source: …/DeepSeek-V4.1-Flash-1017-FLARE, revision
= branch gb10, last commit 2026-11-05 “Publish the GB10 TP4 package with
vocabulary-parallel sampling and an FP8 Markov head”, 187 files / 474.7 GiB, and it
requires flare-inference ≥ 0.1.0~git20261105002735.0009b89d-1.
Consequences recorded in the plan (Phase 2R, ## Runtime parameters, §6 items 4–7):
0), and it
predates the package's minimum runtime → functionally unusable for the current
package, however fast. Prod = release build, everywhere. Installed+identical already
(03c8e463, Installed == Candidate on dgx1/2/3/4/8), but dgx1/2/3/8 execute
deleted inodes — only dgx4 truly runs the release; the restart window fixes that.--mtp-adaptive-verification-alpha 0.75 added to
both dsv41-flash-1017-4x-flare-gb10* recipes. --transfer-streams 1 (RoCE poison
workaround) and --served-model-name stay. --memory-limit / --kv-cache-limit:
upstream says keep the memory default, so nothing.flare download … --revision gb10 into the NAS store (474.7 GiB), verify
with flare models list --model-cache, then a window to cut over and delete the
superseded 477 G on dgx1 + 261 G dirs on the ranks.Skill updated (~/.agents/skills/flare): rule 1 now checks readlink /proc/<pid>/exe
for (deleted) — hash equality does not prove a node runs the installed build; rule 2
reversed to “prod runs the latest release build”, with the package-gates-runtime reason;
rule 5 adds “keep the flag set aligned with the package README, deviate only where
documented”.