Done and verified.
rw,hard,nofail,x-systemd.mount-timeout=30,…,timeo=100,retrans=3,… with a backup
/etc/fstab.bak-<date>-<time> on each node. Verified live (timeo=100 in
findmnt), sudo mount -a returns 0 without remounting.
Serving untouched: rank PIDs kept their 7-day uptime on dgx1/2/3/8 and
dgx1:30000/v1/models still answered afterwards.umount while it had zero open files. Then the real finding:
dgx1 cannot mount NFSv4.1 at all — a fresh vers=4.1 mount hangs with one
clean client and with nconnect=1, while dgx2/3/4/8 mount 4.1 instantly. Its
fstab is pinned to vers=4.0 (with the new timeouts). Do not "fix" it back to 4.1./tmp/flare-before-gx10-dgxN.txt and copied to /mnt/nas/ai/flare/. Includes the
exact rank launch cmdline (recovery recipe while the orbit controller is unpaired).deployments/ reads in
2–3 s; many-small-file deployments would load ~25× slower. v4.0 vs v4.1: no
measurable difference.Objective file updated: added ## Milestones, ## Round plan, ## Checks (the
check passes on all five powered nodes: fstab=1 live=1), rewrote Phase 1 (the
original blanket sed would have silently forced 4.1 onto dgx1 and hung it; and
nofail never shows up in live options, so verify with timeo=100), and filled in
§6.3.
R2 (Q1): Phase 2 first half — rsync -aH ~/.cache/flare/packages/
/mnt/nas/ai/flare/store/ from dgx1 (rank 0, the node holding the 477 GiB). At the
measured ~660 MiB/s write that is ~12 minutes, so run it in the foreground-ish
with --info=progress2 and then verify with
flare models list --model-cache /mnt/nas/ai/flare/store. Do not delete the
local copy in the same round — that is R3, gated on the models list result.
Watch out for:
echo 3 > /proc/sys/vm/drop_caches on a serving node — the 260 GiB of
weights are served from page cache. Use dd iflag=direct for any storage test.## Checks
command reports them as off).