Преглед изворни кода

ops(nas): phase 1 applied on dgx1-4 + dgx8, with a per-node protocol script

NAS entries now carry nofail, x-systemd.mount-timeout=30, timeo=100, retrans=3
on every powered node; verified live and 'mount -a' is a no-op. Serving was not
disturbed: rank PIDs kept their 7-day uptime and dgx1:30000/v1/models kept
answering throughout.

dgx8 had the NAS in fstab but had never mounted it - fixed. dgx1's two stale NFS
client IDs were cleared with a plain umount at zero open files, so no reboot was
needed; the real finding is that dgx1 cannot mount NFSv4.1 at all (a fresh 4.1
mount hangs with one clean client and with nconnect=1, while dgx2/3/4/8 mount it
instantly), so its fstab is pinned to 4.0.

The original phase 1 instruction was a blanket sed over the nas.lan:/sna line,
which would have forced 4.1 onto dgx1 and hung it; the protocol lives in that same
line. Replaced by host-scripts/set-nas-mount-opts.sh (rewrites by mount-point
field, takes the version per node; proven idempotent on dgx4). Also: nofail never
appears in live options, so the check verifies timeo=100.

Measured the open throughput question: the NAS is 10 GbE, 659-822 MiB/s at 1 MiB
but ~30 MiB/s at 8 KiB, so a 1.8 GiB deployments/ dir reads in 2-3 s while a
many-small-file one would be ~25x slower. v4.0 vs v4.1 made no difference.

Plan file gains Milestones / Round plan / Checks; phase 0.3 state captures are on
each node and in /mnt/nas/ai/flare/ (hostnames are gx10-dgxN, not the aliases).
Alexander Korolev пре 1 недеља
родитељ
комит
4ace161940

+ 54 - 0
host-scripts/set-nas-mount-opts.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+# Make the NAS mount boot-proof on one node, keeping the protocol that works there.
+#
+# Used on dgx1/2/3/4/8 on 2026-11-11 (phase 1 of flare-storage-nas.md).
+#
+#   ./set-nas-mount-opts.sh <nfs-vers> [remount yes|no]
+#   ./set-nas-mount-opts.sh 4.1        # dgx2, dgx3, dgx4, dgx8
+#   ./set-nas-mount-opts.sh 4.0 no     # dgx1 (4.1 hangs there; live mount was fine)
+#
+# Why a script and not a sed over the nas.lan:/sna line: the protocol version lives
+# in that same line, so a blanket rewrite silently forces 4.1 onto nodes that cannot
+# mount it (dgx1 hangs on a fresh 4.1 mount, with one clean client and nconnect=1).
+# The entry is rewritten by its mount-point field instead.
+#
+# Note: `nofail` never appears in live mount options (it is an fstab/systemd
+# option), so verify the applied change with `timeo=100`, not with `nofail`.
+set -e
+export LC_ALL=C
+
+VERS="${1:?usage: $0 <nfs-vers> [remount yes|no]}"
+REMOUNT="${2:-yes}"
+
+sudo cp -a /etc/fstab "/etc/fstab.bak-$(date +%F-%H%M)"
+
+sudo python3 - "$VERS" <<'PY'
+import sys
+vers = sys.argv[1]
+path, out, hit = "/etc/fstab", [], False
+for line in open(path).read().splitlines(True):
+    f = line.split()
+    if len(f) >= 3 and f[1] == "/mnt/nas":
+        opts = ("rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,"
+                "timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=" + vers)
+        out.append("nas.lan:/sna\t/mnt/nas\tnfs4\t%s\t0\t0\n" % opts)
+        hit = True
+    else:
+        out.append(line)
+assert hit, "no /mnt/nas entry in fstab"
+open(path, "w").write("".join(out))
+PY
+
+grep "/mnt/nas" /etc/fstab
+sudo systemctl daemon-reload
+
+if [ "$REMOUNT" = yes ]; then
+  # A plain umount (never -l): it fails safely when something holds the mount, and
+  # it is the only thing that actually clears stale NFS client IDs.
+  if sudo umount /mnt/nas 2>/dev/null; then echo "umount ok"; else echo "umount skipped (busy?)"; fi
+  timeout 40 sudo mount /mnt/nas && echo "mount ok"
+fi
+
+findmnt -no OPTIONS /mnt/nas | grep -qo 'timeo=100' \
+  && echo "VERIFIED timeo=100 live" \
+  || echo "NOT live yet (fstab only; takes effect on next mount)"

+ 48 - 0
specs/tech-architecture/RALPH_PROGRESS.md

@@ -0,0 +1,48 @@
+# Progress
+
+## Round 1 (2026-11-11) — Phase 0.2–0.4 + Phase 1 on dgx1/2/3/4/8
+
+**Done and verified.**
+
+- **Phase 1 applied on dgx1, dgx2, dgx3, dgx4, dgx8**: NAS entry rewritten to
+  `rw,hard,nofail,x-systemd.mount-timeout=30,…,timeo=100,retrans=3,…` with a backup
+  `/etc/fstab.bak-<date>-<time>` on each node. Verified live (`timeo=100` in
+  `findmnt`), `sudo mount -a` returns 0 without remounting.
+  **Serving untouched**: rank PIDs kept their 7-day uptime on dgx1/2/3/8 and
+  `dgx1:30000/v1/models` still answered afterwards.
+- **dgx8 NAS mount restored** — it was in fstab but had never been mounted.
+- **dgx1: no reboot was needed.** The two stale NFS client IDs were cleared with a
+  plain (non-lazy) `umount` while it had zero open files. Then the real finding:
+  **dgx1 cannot mount NFSv4.1 at all** — a fresh `vers=4.1` mount hangs with one
+  clean client and with `nconnect=1`, while dgx2/3/4/8 mount 4.1 instantly. Its
+  fstab is pinned to `vers=4.0` (with the new timeouts). Do not "fix" it back to 4.1.
+- **Phase 0.3 state capture** for dgx1/2/3/4/8: on each node at
+  `/tmp/flare-before-gx10-dgxN.txt` and copied to `/mnt/nas/ai/flare/`. Includes the
+  exact rank launch cmdline (recovery recipe while the orbit controller is unpaired).
+- **Open item 6.3 measured**: the NAS is **10 GbE**, not 1 GbE — 659–822 MiB/s at
+  1 MiB reads, but only **~30 MiB/s at 8 KiB**. So a 1.8 GiB `deployments/` reads in
+  2–3 s; many-small-file deployments would load ~25× slower. v4.0 vs v4.1: no
+  measurable difference.
+
+Objective file updated: added `## Milestones`, `## Round plan`, `## Checks` (the
+check passes on all five powered nodes: `fstab=1 live=1`), rewrote Phase 1 (the
+original blanket `sed` would have silently forced 4.1 onto dgx1 and hung it; and
+`nofail` never shows up in live options, so verify with `timeo=100`), and filled in
+§6.3.
+
+## Next round
+
+**R2 (Q1): Phase 2 first half** — `rsync -aH ~/.cache/flare/packages/
+/mnt/nas/ai/flare/store/` from dgx1 (rank 0, the node holding the 477 GiB). At the
+measured ~660 MiB/s write that is **~12 minutes**, so run it in the foreground-ish
+with `--info=progress2` and then verify with
+`flare models list --model-cache /mnt/nas/ai/flare/store`. Do **not** delete the
+local copy in the same round — that is R3, gated on the `models list` result.
+
+Watch out for:
+
+- **Never `echo 3 > /proc/sys/vm/drop_caches` on a serving node** — the 260 GiB of
+  weights are served from page cache. Use `dd iflag=direct` for any storage test.
+- dgx5/6/7 are off; they still need Phase 1 when they come up (the `## Checks`
+  command reports them as `off`).
+- R4 (PR #184 rebuild) needs dgx5 powered on and must precede any agent restart.

+ 111 - 14
specs/tech-architecture/flare-storage-nas.md

@@ -5,6 +5,47 @@ NVMe** (260 GiB for `DeepSeek-V4.1-Flash` tp4) and **all other model artefacts l
 on the NAS**. This document is written to be executed by someone with no context
 beyond it. Every command is copy-pasteable and every phase has a gate.
 
+## Milestones
+
+- [x] NAS mount made boot-proof on every participating node (`nofail`,
+      `timeo=100`, `x-systemd.mount-timeout=30`) — verified live, serving untouched
+- [x] dgx8's missing NAS mount restored (it was in fstab but never mounted)
+- [x] dgx1's stale NFS state cleared without a reboot; node pinned to the protocol
+      that works there (4.0) — see the correction in Phase 1
+- [x] Pre-migration state of dgx1/2/3/4/8 captured locally and on the NAS
+- [x] NAS throughput measured (was open item 6.3)
+- [ ] PR #184 binary rebuilt and staged (Phase 0.1) — needs dgx5 powered on
+- [ ] Package store (477 GiB) on the NAS, local copy removed, config pointed at it (Phase 2)
+- [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
+- [ ] Sparks rolled out (Phase 4, needs a window)
+- [ ] prod4 rolled out (Phase 5, needs a window)
+- [ ] Verification matrix run and rollback drilled (Phase 6)
+
+## Round plan
+
+- [x] **R1 (Q1)** — Phase 0.2–0.4 + Phase 1 on dgx1/2/3/4/8
+- [ ] **R2 (Q1)** — Phase 2 first half: `rsync -aH` the 477 GiB store to the NAS in
+      the background, then verify with `flare models list --model-cache`
+- [ ] **R3 (Q1)** — Phase 2 second half: point `[paths] package_cache` at the NAS,
+      remove the local store, confirm the launched rank cmdline (open item 6.1)
+- [ ] **R4 (Q1)** — PR #184 rebuild and staging (needs dgx5 on; prerequisite for any restart)
+- [ ] **R5 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4
+- [ ] **R6 (Q2)** — Phase 4 sparks rollout (window; ranks stop)
+- [ ] **R7 (Q2)** — Phase 5 prod4 rollout (window; ranks stop)
+- [ ] **R8 (Q3)** — Phase 6 verification matrix and rollback drill
+
+## Checks
+
+Run from the controller; every node must report `fstab=1 live=1`:
+
+```bash
+for h in dgx1 dgx2 dgx3 dgx4 dgx5 dgx6 dgx7 dgx8; do printf "%-6s " $h
+  ssh -o BatchMode=yes -o ConnectTimeout=5 $h.lan 'printf "%s fstab=%s live=%s ranks=%s\n" \
+    "$(hostname -s)" "$(grep /mnt/nas /etc/fstab | grep -c nofail)" \
+    "$(findmnt -no OPTIONS /mnt/nas | grep -c "timeo=100")" "$(pgrep -fc "[f]lare run")"' \
+  2>/dev/null || echo "off"; done
+```
+
 ## 1. Measured baseline (2026-11-10, read-only)
 
 | node | role | `models/` | `packages/` | rank cmdline cache |
@@ -92,15 +133,27 @@ previous phase failed.
    ```bash
    ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
    ```
-3. Record the current state of each node for rollback:
+3. Record the current state of each node for rollback — **done 2026-11-11**, files
+   are on each node and in `/mnt/nas/ai/flare/` (`flare-before-gx10-dgxN.txt`).
+   Hostnames are `gx10-dgxN`, **not** the `dgxN.lan` aliases — build filenames from
+   `hostname -s` and do not assume the alias resolves to a filename:
    ```bash
-   du -sh ~/.cache/flare/* | sort -h > /tmp/flare-before-$(hostname).txt
+   du -sh ~/.cache/flare/* | sort -h > /tmp/flare-before-$(hostname -s).txt
    findmnt -no TARGET,SOURCE,OPTIONS /mnt/nas
-   pgrep -af "[f]lare run" | head -1
+   awk 'NR>1' /proc/fs/nfsfs/servers | sort -u | wc -l    # NFS client IDs
+   tr '\0' ' ' < /proc/$(pgrep -f "[f]lare run" | head -1)/cmdline   # exact launch command
    ```
-4. **dgx8 currently has the NAS in fstab but not mounted.** Fix that first
-   (`sudo mount /mnt/nas`), and make it non-blocking for boot by adding `nofail`
-   and a timeout to the fstab line (see phase 1).
+   The captured rank cmdline is the recovery recipe if a node has to be restarted
+   outside `orbit` (the controller is currently unpaired), e.g. rank 0 on dgx1:
+   `flare run ~/.cache/flare/packages/7cf40bec…-35842073/model.safetensors
+   --package-cache ~/.cache/flare --rank 0 --rendezvous 0.0.0.0:29800
+   --max-concurrency 32 --mtp-draft-tokens 5 --mtp-adaptive-verification
+   --embedding-table-storage disk --embedding-table-cache-size 1073741824
+   --embedding-table-resident-scales --transfer-streams 1 --host 0.0.0.0 --port 30000
+   --served-model-name deepseek-v4.1-flash --publisher-keys … --dangerously-approve-key …
+   --dangerously-accept-publisher local-inference-lab --gpus 0`
+4. ~~dgx8 has the NAS in fstab but not mounted~~ **fixed 2026-11-11** — mounted at
+   `vers=4.1` and made non-blocking with the phase 1 options.
 
 Gate: binary rebuilt and staged, NAS mounted on every participating node.
 
@@ -116,21 +169,54 @@ Two problems for this plan: a NAS hiccup blocks boot, and `hard` + long `timeo`
 means a stale mount hangs reads (this NAS has already produced a
 `server not responding` outage that needed reboots).
 
-Change to:
+Applied everywhere, **with one per-node exception** (see below):
 
 ```
 nas.lan:/sna  /mnt/nas  nfs4  rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0
 ```
 
+Do **not** use a blanket `sed` over the `nas.lan:/sna` line — it silently rewrites
+the protocol version too. Rewrite the entry by its mount-point field and pass the
+protocol per node — use **`host-scripts/set-nas-mount-opts.sh <vers> [remount]`**
+(proven idempotent on dgx4 on 2026-11-11; it backs up fstab, rewrites the entry, and
+plain-`umount`s before mounting). For reference, it is equivalent to:
+
 ```bash
-sudo cp /etc/fstab /etc/fstab.bak-$(date +%F)
-sudo sed -i 's|^\(nas.lan:/sna.*\)$|# replaced by the storage plan\nnas.lan:/sna  /mnt/nas  nfs4  rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0|' /etc/fstab
+sudo cp -a /etc/fstab /etc/fstab.bak-$(date +%F-%H%M)
+sudo python3 - "$VERS" <<'PY'
+import sys
+vers = sys.argv[1]
+path, out, hit = "/etc/fstab", [], False
+for line in open(path).read().splitlines(True):
+    f = line.split()
+    if len(f) >= 3 and f[1] == "/mnt/nas":
+        opts = ("rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,"
+                "timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=" + vers)
+        out.append("nas.lan:/sna\t/mnt/nas\tnfs4\t%s\t0\t0\n" % opts); hit = True
+    else:
+        out.append(line)
+assert hit, "no /mnt/nas entry in fstab"
+open(path, "w").write("".join(out))
+PY
 sudo systemctl daemon-reload
-sudo umount /mnt/nas 2>/dev/null; sudo mount /mnt/nas
-findmnt -no TARGET,OPTIONS /mnt/nas | grep -o 'nofail'
+sudo umount /mnt/nas && sudo mount /mnt/nas     # umount fails safely if something holds it
+findmnt -no OPTIONS /mnt/nas | grep -o 'timeo=100'
 ```
 
-Gate: mount present, `nofail` in effect, a `sudo mount -a` is a no-op.
+`nofail` is an fstab-only option: it never appears in live mount options, so verify
+the *live* change with `timeo=100`, not with `nofail`.
+
+**dgx1 exception — applied and verified 2026-11-11.** dgx1 cannot mount NFSv4.1 at
+all: a fresh `vers=4.1` mount hangs even with a single clean client and with
+`nconnect=1`, while dgx2/3/4/8 mount 4.1 instantly. Its fstab is therefore pinned
+to `vers=4.0` with the same other options. Its two stale client IDs were cleared
+without a reboot (`umount`, not `umount -l`, with zero open files — that is the
+procedure; a reboot was not needed).
+
+Gate: mount present, `nofail` in effect, a `sudo mount -a` is a no-op. **Passed on
+dgx1/2/3/4/8 (2026-11-11):** live options show `timeo=100`, `mount -a` returns 0
+without remounting, all rank PIDs kept their 7-day uptime and the API on
+`dgx1:30000` still answered. dgx5–7 are off and still need this.
 
 ### Phase 2 — package store (the 477 GiB) to NAS
 
@@ -285,6 +371,17 @@ flare orbit run <group> --cluster sparks
 1. Whether the agent's `--package-cache` path can be redirected by config or needs
    a launcher `extra_flags` change (phase 2 note).
 2. Whether FLARE tolerates the bind mount across a real stage (phase 3 gate).
-3. Real NAS read throughput for `deployments/` at load — 1.8 GiB over NFS at
-   1 GbE is ~15 s per rank; measure before rolling to prod, and keep `weights/`
+3. ~~Real NAS read throughput~~ **measured 2026-11-11** (`dd iflag=direct`, 2 GiB,
+   so no page-cache lying): the NAS is **10 GbE, not 1 GbE**.
+
+   | node | write 2 GiB | read 1 MiB | read 8 KiB |
+   |---|---|---|---|
+   | dgx8 (v4.1) | 659 MiB/s | 731 MiB/s | 28 MiB/s |
+   | dgx1 (v4.0) | 663 MiB/s | 822 MiB/s | 32 MiB/s |
+
+   So a 1.8 GiB `deployments/` directory reads in **2–3 s**, and the protocol
+   choice costs nothing. The real constraint is small-file/short-read work: at
+   8 KiB the same share runs at ~30 MiB/s, so a `deployments/` split into many
+   small files would load ~25× slower than the bulk figure suggests — count the
+   files in a rank deployment before trusting the 2–3 s number, and keep `weights/`
    local regardless.