|
|
@@ -5,6 +5,47 @@ NVMe** (260 GiB for `DeepSeek-V4.1-Flash` tp4) and **all other model artefacts l
|
|
|
on the NAS**. This document is written to be executed by someone with no context
|
|
|
beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
|
|
|
+## Milestones
|
|
|
+
|
|
|
+- [x] NAS mount made boot-proof on every participating node (`nofail`,
|
|
|
+ `timeo=100`, `x-systemd.mount-timeout=30`) — verified live, serving untouched
|
|
|
+- [x] dgx8's missing NAS mount restored (it was in fstab but never mounted)
|
|
|
+- [x] dgx1's stale NFS state cleared without a reboot; node pinned to the protocol
|
|
|
+ that works there (4.0) — see the correction in Phase 1
|
|
|
+- [x] Pre-migration state of dgx1/2/3/4/8 captured locally and on the NAS
|
|
|
+- [x] NAS throughput measured (was open item 6.3)
|
|
|
+- [ ] PR #184 binary rebuilt and staged (Phase 0.1) — needs dgx5 powered on
|
|
|
+- [ ] Package store (477 GiB) on the NAS, local copy removed, config pointed at it (Phase 2)
|
|
|
+- [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
|
|
|
+- [ ] Sparks rolled out (Phase 4, needs a window)
|
|
|
+- [ ] prod4 rolled out (Phase 5, needs a window)
|
|
|
+- [ ] Verification matrix run and rollback drilled (Phase 6)
|
|
|
+
|
|
|
+## Round plan
|
|
|
+
|
|
|
+- [x] **R1 (Q1)** — Phase 0.2–0.4 + Phase 1 on dgx1/2/3/4/8
|
|
|
+- [ ] **R2 (Q1)** — Phase 2 first half: `rsync -aH` the 477 GiB store to the NAS in
|
|
|
+ the background, then verify with `flare models list --model-cache`
|
|
|
+- [ ] **R3 (Q1)** — Phase 2 second half: point `[paths] package_cache` at the NAS,
|
|
|
+ remove the local store, confirm the launched rank cmdline (open item 6.1)
|
|
|
+- [ ] **R4 (Q1)** — PR #184 rebuild and staging (needs dgx5 on; prerequisite for any restart)
|
|
|
+- [ ] **R5 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4
|
|
|
+- [ ] **R6 (Q2)** — Phase 4 sparks rollout (window; ranks stop)
|
|
|
+- [ ] **R7 (Q2)** — Phase 5 prod4 rollout (window; ranks stop)
|
|
|
+- [ ] **R8 (Q3)** — Phase 6 verification matrix and rollback drill
|
|
|
+
|
|
|
+## Checks
|
|
|
+
|
|
|
+Run from the controller; every node must report `fstab=1 live=1`:
|
|
|
+
|
|
|
+```bash
|
|
|
+for h in dgx1 dgx2 dgx3 dgx4 dgx5 dgx6 dgx7 dgx8; do printf "%-6s " $h
|
|
|
+ ssh -o BatchMode=yes -o ConnectTimeout=5 $h.lan 'printf "%s fstab=%s live=%s ranks=%s\n" \
|
|
|
+ "$(hostname -s)" "$(grep /mnt/nas /etc/fstab | grep -c nofail)" \
|
|
|
+ "$(findmnt -no OPTIONS /mnt/nas | grep -c "timeo=100")" "$(pgrep -fc "[f]lare run")"' \
|
|
|
+ 2>/dev/null || echo "off"; done
|
|
|
+```
|
|
|
+
|
|
|
## 1. Measured baseline (2026-11-10, read-only)
|
|
|
|
|
|
| node | role | `models/` | `packages/` | rank cmdline cache |
|
|
|
@@ -92,15 +133,27 @@ previous phase failed.
|
|
|
```bash
|
|
|
ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
|
|
|
```
|
|
|
-3. Record the current state of each node for rollback:
|
|
|
+3. Record the current state of each node for rollback — **done 2026-11-11**, files
|
|
|
+ are on each node and in `/mnt/nas/ai/flare/` (`flare-before-gx10-dgxN.txt`).
|
|
|
+ Hostnames are `gx10-dgxN`, **not** the `dgxN.lan` aliases — build filenames from
|
|
|
+ `hostname -s` and do not assume the alias resolves to a filename:
|
|
|
```bash
|
|
|
- du -sh ~/.cache/flare/* | sort -h > /tmp/flare-before-$(hostname).txt
|
|
|
+ du -sh ~/.cache/flare/* | sort -h > /tmp/flare-before-$(hostname -s).txt
|
|
|
findmnt -no TARGET,SOURCE,OPTIONS /mnt/nas
|
|
|
- pgrep -af "[f]lare run" | head -1
|
|
|
+ awk 'NR>1' /proc/fs/nfsfs/servers | sort -u | wc -l # NFS client IDs
|
|
|
+ tr '\0' ' ' < /proc/$(pgrep -f "[f]lare run" | head -1)/cmdline # exact launch command
|
|
|
```
|
|
|
-4. **dgx8 currently has the NAS in fstab but not mounted.** Fix that first
|
|
|
- (`sudo mount /mnt/nas`), and make it non-blocking for boot by adding `nofail`
|
|
|
- and a timeout to the fstab line (see phase 1).
|
|
|
+ The captured rank cmdline is the recovery recipe if a node has to be restarted
|
|
|
+ outside `orbit` (the controller is currently unpaired), e.g. rank 0 on dgx1:
|
|
|
+ `flare run ~/.cache/flare/packages/7cf40bec…-35842073/model.safetensors
|
|
|
+ --package-cache ~/.cache/flare --rank 0 --rendezvous 0.0.0.0:29800
|
|
|
+ --max-concurrency 32 --mtp-draft-tokens 5 --mtp-adaptive-verification
|
|
|
+ --embedding-table-storage disk --embedding-table-cache-size 1073741824
|
|
|
+ --embedding-table-resident-scales --transfer-streams 1 --host 0.0.0.0 --port 30000
|
|
|
+ --served-model-name deepseek-v4.1-flash --publisher-keys … --dangerously-approve-key …
|
|
|
+ --dangerously-accept-publisher local-inference-lab --gpus 0`
|
|
|
+4. ~~dgx8 has the NAS in fstab but not mounted~~ **fixed 2026-11-11** — mounted at
|
|
|
+ `vers=4.1` and made non-blocking with the phase 1 options.
|
|
|
|
|
|
Gate: binary rebuilt and staged, NAS mounted on every participating node.
|
|
|
|
|
|
@@ -116,21 +169,54 @@ Two problems for this plan: a NAS hiccup blocks boot, and `hard` + long `timeo`
|
|
|
means a stale mount hangs reads (this NAS has already produced a
|
|
|
`server not responding` outage that needed reboots).
|
|
|
|
|
|
-Change to:
|
|
|
+Applied everywhere, **with one per-node exception** (see below):
|
|
|
|
|
|
```
|
|
|
nas.lan:/sna /mnt/nas nfs4 rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0
|
|
|
```
|
|
|
|
|
|
+Do **not** use a blanket `sed` over the `nas.lan:/sna` line — it silently rewrites
|
|
|
+the protocol version too. Rewrite the entry by its mount-point field and pass the
|
|
|
+protocol per node — use **`host-scripts/set-nas-mount-opts.sh <vers> [remount]`**
|
|
|
+(proven idempotent on dgx4 on 2026-11-11; it backs up fstab, rewrites the entry, and
|
|
|
+plain-`umount`s before mounting). For reference, it is equivalent to:
|
|
|
+
|
|
|
```bash
|
|
|
-sudo cp /etc/fstab /etc/fstab.bak-$(date +%F)
|
|
|
-sudo sed -i 's|^\(nas.lan:/sna.*\)$|# replaced by the storage plan\nnas.lan:/sna /mnt/nas nfs4 rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0|' /etc/fstab
|
|
|
+sudo cp -a /etc/fstab /etc/fstab.bak-$(date +%F-%H%M)
|
|
|
+sudo python3 - "$VERS" <<'PY'
|
|
|
+import sys
|
|
|
+vers = sys.argv[1]
|
|
|
+path, out, hit = "/etc/fstab", [], False
|
|
|
+for line in open(path).read().splitlines(True):
|
|
|
+ f = line.split()
|
|
|
+ if len(f) >= 3 and f[1] == "/mnt/nas":
|
|
|
+ opts = ("rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,"
|
|
|
+ "timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=" + vers)
|
|
|
+ out.append("nas.lan:/sna\t/mnt/nas\tnfs4\t%s\t0\t0\n" % opts); hit = True
|
|
|
+ else:
|
|
|
+ out.append(line)
|
|
|
+assert hit, "no /mnt/nas entry in fstab"
|
|
|
+open(path, "w").write("".join(out))
|
|
|
+PY
|
|
|
sudo systemctl daemon-reload
|
|
|
-sudo umount /mnt/nas 2>/dev/null; sudo mount /mnt/nas
|
|
|
-findmnt -no TARGET,OPTIONS /mnt/nas | grep -o 'nofail'
|
|
|
+sudo umount /mnt/nas && sudo mount /mnt/nas # umount fails safely if something holds it
|
|
|
+findmnt -no OPTIONS /mnt/nas | grep -o 'timeo=100'
|
|
|
```
|
|
|
|
|
|
-Gate: mount present, `nofail` in effect, a `sudo mount -a` is a no-op.
|
|
|
+`nofail` is an fstab-only option: it never appears in live mount options, so verify
|
|
|
+the *live* change with `timeo=100`, not with `nofail`.
|
|
|
+
|
|
|
+**dgx1 exception — applied and verified 2026-11-11.** dgx1 cannot mount NFSv4.1 at
|
|
|
+all: a fresh `vers=4.1` mount hangs even with a single clean client and with
|
|
|
+`nconnect=1`, while dgx2/3/4/8 mount 4.1 instantly. Its fstab is therefore pinned
|
|
|
+to `vers=4.0` with the same other options. Its two stale client IDs were cleared
|
|
|
+without a reboot (`umount`, not `umount -l`, with zero open files — that is the
|
|
|
+procedure; a reboot was not needed).
|
|
|
+
|
|
|
+Gate: mount present, `nofail` in effect, a `sudo mount -a` is a no-op. **Passed on
|
|
|
+dgx1/2/3/4/8 (2026-11-11):** live options show `timeo=100`, `mount -a` returns 0
|
|
|
+without remounting, all rank PIDs kept their 7-day uptime and the API on
|
|
|
+`dgx1:30000` still answered. dgx5–7 are off and still need this.
|
|
|
|
|
|
### Phase 2 — package store (the 477 GiB) to NAS
|
|
|
|
|
|
@@ -285,6 +371,17 @@ flare orbit run <group> --cluster sparks
|
|
|
1. Whether the agent's `--package-cache` path can be redirected by config or needs
|
|
|
a launcher `extra_flags` change (phase 2 note).
|
|
|
2. Whether FLARE tolerates the bind mount across a real stage (phase 3 gate).
|
|
|
-3. Real NAS read throughput for `deployments/` at load — 1.8 GiB over NFS at
|
|
|
- 1 GbE is ~15 s per rank; measure before rolling to prod, and keep `weights/`
|
|
|
+3. ~~Real NAS read throughput~~ **measured 2026-11-11** (`dd iflag=direct`, 2 GiB,
|
|
|
+ so no page-cache lying): the NAS is **10 GbE, not 1 GbE**.
|
|
|
+
|
|
|
+ | node | write 2 GiB | read 1 MiB | read 8 KiB |
|
|
|
+ |---|---|---|---|
|
|
|
+ | dgx8 (v4.1) | 659 MiB/s | 731 MiB/s | 28 MiB/s |
|
|
|
+ | dgx1 (v4.0) | 663 MiB/s | 822 MiB/s | 32 MiB/s |
|
|
|
+
|
|
|
+ So a 1.8 GiB `deployments/` directory reads in **2–3 s**, and the protocol
|
|
|
+ choice costs nothing. The real constraint is small-file/short-read work: at
|
|
|
+ 8 KiB the same share runs at ~30 MiB/s, so a `deployments/` split into many
|
|
|
+ small files would load ~25× slower than the bulk figure suggests — count the
|
|
|
+ files in a rank deployment before trusting the 2–3 s number, and keep `weights/`
|
|
|
local regardless.
|