RALPH_PROGRESS.md 2.8 KB

Progress

Round 1 (2026-11-11) — Phase 0.2–0.4 + Phase 1 on dgx1/2/3/4/8

Done and verified.

  • Phase 1 applied on dgx1, dgx2, dgx3, dgx4, dgx8: NAS entry rewritten to rw,hard,nofail,x-systemd.mount-timeout=30,…,timeo=100,retrans=3,… with a backup /etc/fstab.bak-<date>-<time> on each node. Verified live (timeo=100 in findmnt), sudo mount -a returns 0 without remounting. Serving untouched: rank PIDs kept their 7-day uptime on dgx1/2/3/8 and dgx1:30000/v1/models still answered afterwards.
  • dgx8 NAS mount restored — it was in fstab but had never been mounted.
  • dgx1: no reboot was needed. The two stale NFS client IDs were cleared with a plain (non-lazy) umount while it had zero open files. Then the real finding: dgx1 cannot mount NFSv4.1 at all — a fresh vers=4.1 mount hangs with one clean client and with nconnect=1, while dgx2/3/4/8 mount 4.1 instantly. Its fstab is pinned to vers=4.0 (with the new timeouts). Do not "fix" it back to 4.1.
  • Phase 0.3 state capture for dgx1/2/3/4/8: on each node at /tmp/flare-before-gx10-dgxN.txt and copied to /mnt/nas/ai/flare/. Includes the exact rank launch cmdline (recovery recipe while the orbit controller is unpaired).
  • Open item 6.3 measured: the NAS is 10 GbE, not 1 GbE — 659–822 MiB/s at 1 MiB reads, but only ~30 MiB/s at 8 KiB. So a 1.8 GiB deployments/ reads in 2–3 s; many-small-file deployments would load ~25× slower. v4.0 vs v4.1: no measurable difference.

Objective file updated: added ## Milestones, ## Round plan, ## Checks (the check passes on all five powered nodes: fstab=1 live=1), rewrote Phase 1 (the original blanket sed would have silently forced 4.1 onto dgx1 and hung it; and nofail never shows up in live options, so verify with timeo=100), and filled in §6.3.

Next round

R2 (Q1): Phase 2 first half — rsync -aH ~/.cache/flare/packages/ /mnt/nas/ai/flare/store/ from dgx1 (rank 0, the node holding the 477 GiB). At the measured ~660 MiB/s write that is ~12 minutes, so run it in the foreground-ish with --info=progress2 and then verify with flare models list --model-cache /mnt/nas/ai/flare/store. Do not delete the local copy in the same round — that is R3, gated on the models list result.

Watch out for:

  • Never echo 3 > /proc/sys/vm/drop_caches on a serving node — the 260 GiB of weights are served from page cache. Use dd iflag=direct for any storage test.
  • dgx5/6/7 are off; they still need Phase 1 when they come up (the ## Checks command reports them as off).
  • R4 (PR #184 rebuild) needs dgx5 powered on and must precede any agent restart.

Loop log

  • round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…