RALPH_PROGRESS.md 6.1 KB

Progress

Round 1 (2026-11-11) — Phase 0.2–0.4 + Phase 1 on dgx1/2/3/4/8

Done and verified.

  • Phase 1 applied on dgx1, dgx2, dgx3, dgx4, dgx8: NAS entry rewritten to rw,hard,nofail,x-systemd.mount-timeout=30,…,timeo=100,retrans=3,… with a backup /etc/fstab.bak-<date>-<time> on each node. Verified live (timeo=100 in findmnt), sudo mount -a returns 0 without remounting. Serving untouched: rank PIDs kept their 7-day uptime on dgx1/2/3/8 and dgx1:30000/v1/models still answered afterwards.
  • dgx8 NAS mount restored — it was in fstab but had never been mounted.
  • dgx1: no reboot was needed. The two stale NFS client IDs were cleared with a plain (non-lazy) umount while it had zero open files. Then the real finding: dgx1 cannot mount NFSv4.1 at all — a fresh vers=4.1 mount hangs with one clean client and with nconnect=1, while dgx2/3/4/8 mount 4.1 instantly. Its fstab is pinned to vers=4.0 (with the new timeouts). Do not "fix" it back to 4.1.
  • Phase 0.3 state capture for dgx1/2/3/4/8: on each node at /tmp/flare-before-gx10-dgxN.txt and copied to /mnt/nas/ai/flare/. Includes the exact rank launch cmdline (recovery recipe while the orbit controller is unpaired).
  • Open item 6.3 measured: the NAS is 10 GbE, not 1 GbE — 659–822 MiB/s at 1 MiB reads, but only ~30 MiB/s at 8 KiB. So a 1.8 GiB deployments/ reads in 2–3 s; many-small-file deployments would load ~25× slower. v4.0 vs v4.1: no measurable difference.

Objective file updated: added ## Milestones, ## Round plan, ## Checks (the check passes on all five powered nodes: fstab=1 live=1), rewrote Phase 1 (the original blanket sed would have silently forced 4.1 onto dgx1 and hung it; and nofail never shows up in live options, so verify with timeo=100), and filled in §6.3.

Next round

R2 (Q1): Phase 2 first half — rsync -aH ~/.cache/flare/packages/ /mnt/nas/ai/flare/store/ from dgx1 (rank 0, the node holding the 477 GiB). At the measured ~660 MiB/s write that is ~12 minutes, so run it in the foreground-ish with --info=progress2 and then verify with flare models list --model-cache /mnt/nas/ai/flare/store. Do not delete the local copy in the same round — that is R3, gated on the models list result.

Watch out for:

  • Never echo 3 > /proc/sys/vm/drop_caches on a serving node — the 260 GiB of weights are served from page cache. Use dd iflag=direct for any storage test.
  • dgx5/6/7 are off; they still need Phase 1 when they come up (the ## Checks command reports them as off).
  • R4 (PR #184 rebuild) needs dgx5 powered on and must precede any agent restart.

Loop log

  • round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…

2026-11-11 — R2 (flare-storage-nas): purge instead of migrate; what we serve is superseded

Goal quadrant: impact × effort — delete 783 GiB of dead data and find out whether the fleet is even on the current model build. Both landed.

The operator rejected migrating the old cache (“*Syncing those old flare cache to store does not make much sense honestly*”) and was right on the facts, harder than expected:

  • Every node held the Nov-2 build checkpoint-expanded-scales/gb10-tp4-202611021936 (261 GiB × 3 nodes) with nothing holding it open — verified per node via /proc/<rank>/fd, /proc/<rank>/maps and lsof +D before deleting. Freed 783 GiB: dgx1 98 % full → 283 G free, dgx2 → 467 G, dgx3 → 470 G. Rank PIDs untouched (7-day uptime), dgx1:30000/v1/models still answering.
  • dgx1's root is 915 GiB, not 1.8 TB — it was 22 G from full. Plans sized on dgx4's disk miss this.
  • dgx1's package_cache had silently drifted to an empty ~/flare-pkgs/gb10 skeleton: flare models list said “No local model versions found”, while the live rank used ~/.cache/flare. Fix that before anything points at the NAS.
  • The started rsync (105 GiB in) was killed and the partial NAS copy deleted; Phase 2 is marked SUPERSEDED, do not run.

The real finding: the fleet serves a superseded package from a repo that no longer exists. The manifest inside model.safetensors names …/DeepSeek-V4.1-Flash-1017-FLARE2 → 404 with a valid token (401 unauthenticated is HF masking a missing repo). Current source: …/DeepSeek-V4.1-Flash-1017-FLARE, revision = branch gb10, last commit 2026-11-05 “Publish the GB10 TP4 package with vocabulary-parallel sampling and an FP8 Markov head”, 187 files / 474.7 GiB, and it requires flare-inference ≥ 0.1.0~git20261105002735.0009b89d-1.

Consequences recorded in the plan (Phase 2R, ## Runtime parameters, §6 items 4–7):

  • PR #184 rebuild dropped. It is not in the release (marker string → 0), and it predates the package's minimum runtime → functionally unusable for the current package, however fast. Prod = release build, everywhere. Installed+identical already (03c8e463, Installed == Candidate on dgx1/2/3/4/8), but dgx1/2/3/8 execute deleted inodes — only dgx4 truly runs the release; the restart window fixes that.
  • Launch parameters taken from the package's own README, port stays 30000 (operator). Only delta in practice: --mtp-adaptive-verification-alpha 0.75 added to both dsv41-flash-1017-4x-flare-gb10* recipes. --transfer-streams 1 (RoCE poison workaround) and --served-model-name stay. --memory-limit / --kv-cache-limit: upstream says keep the memory default, so nothing.
  • Next (R3): flare download … --revision gb10 into the NAS store (474.7 GiB), verify with flare models list --model-cache, then a window to cut over and delete the superseded 477 G on dgx1 + 261 G dirs on the ranks.

Skill updated (~/.agents/skills/flare): rule 1 now checks readlink /proc/<pid>/exe for (deleted) — hash equality does not prove a node runs the installed build; rule 2 reversed to “prod runs the latest release build”, with the package-gates-runtime reason; rule 5 adds “keep the flag set aligned with the package README, deviate only where documented”.