|
|
@@ -31,7 +31,15 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
signatures verified, visible from dgx1/dgx4 over NFS (Phase 2R)
|
|
|
- [ ] That core **deployed to the ranks, and the superseded build deleted**
|
|
|
(needs the window; dgx1's 477 G and the ranks' 261 G dirs are still the live data)
|
|
|
-- [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
|
|
|
+- [x] **Bind-mount layout validated on idle dgx4 (Phase 3 gate)** — a rank served from
|
|
|
+ a NAS-backed `deployments/` (TP1 test bed), negative control failed correctly, no
|
|
|
+ writes/rename by FLARE, fstab bind survives reboot with `Requires=mnt-nas.mount`.
|
|
|
+ Remaining for the layout: apply it to the TP4 rank slices in the window (R7/R8).
|
|
|
+- [ ] **Open question worth a pre-window check:** every rank startup pulled ~2.6 GB from
|
|
|
+ the NAS *even when `deployments/` was local* (see the measurement caveats). Since
|
|
|
+ R4 pointed `package_cache` at the NAS store, confirm on an idle node that a rank
|
|
|
+ still starts when `/mnt/nas` is unavailable — if it does not, the NAS is on the
|
|
|
+ rank restart path and that must be documented as an availability dependency.
|
|
|
- [ ] Sparks rolled out (Phase 4, needs a window)
|
|
|
- [ ] prod4 rolled out (Phase 5, needs a window)
|
|
|
- [ ] Verification matrix run and rollback drilled (Phase 6)
|
|
|
@@ -69,10 +77,16 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
five stage attempts. Also found: the plan's Phase 3 procedure was **invalid**
|
|
|
(a standalone model-dir copy is rejected in v0.1.0) and there is a **data-loss
|
|
|
trap** in Phase 4 cleanup — see “Phase 3 gate results”. No data was lost.
|
|
|
-- [ ] **R6b (Q2)** — part 2 needs a *completed* TP4 stage (peers, so it belongs to the
|
|
|
- window): do deployment reads really traverse the bind, and does FLARE ever
|
|
|
- replace `deployments/` by rename or hardlink into it (both would break a
|
|
|
- cross-device bind)? If either happens → **option D**.
|
|
|
+- [x] **R6b (Q1)** — **closed without a window**, on a single-GPU core (see “The
|
|
|
+ single-GPU test bed”): a rank **served chat completions with `deployments/`
|
|
|
+ bind-mounted from the NAS**, and the negative control (bind an empty NAS dir →
|
|
|
+ rank refuses to start) proves the data comes *through* the bind. Across a complete
|
|
|
+ load+serve cycle FLARE wrote **nothing** into the bind (0 files newer than rank
|
|
|
+ start) and never replaced it (same inode, still mounted after serving and after
|
|
|
+ reboot) ⇒ **option D is not needed**. Also validated the **fstab-persisted bind
|
|
|
+ across a reboot of dgx4**: mounted at boot, its unit provably
|
|
|
+ `Requires=mnt-nas.mount`, and a rank came up from it **22 s after boot**.
|
|
|
+ Steady-state inference touches the NAS **0 MiB**.
|
|
|
- [ ] **R7 (Q2)** — Phase 4 sparks rollout (window; ranks stop)
|
|
|
- [ ] **R8 (Q2)** — Phase 5 prod4 rollout (window; ranks stop)
|
|
|
- [ ] **R9 (Q3)** — Phase 6 verification matrix and rollback drill
|
|
|
@@ -739,14 +753,19 @@ can get). The bind was unchanged before and after: same inode, same mount,
|
|
|
(`474.66 GiB`, `181 hardlinked objects`) while `blobs/` and `weights/` were bind mounts
|
|
|
onto a **different device** (st_dev 80 = NFS vs 66306 = local NVMe).
|
|
|
|
|
|
-**The plan's Phase 3 step 1–2 procedure was wrong.** Copying `model.safetensors` +
|
|
|
-`deployments` out of a store and pointing `run`/`inspect` at the copy **fails in v0.1.0**
|
|
|
-at `Verify core` with `open core directory: No such file or directory (os error 2)`: the
|
|
|
-envelope resolves its core **within the store that contains it**, so the model path must
|
|
|
-sit inside a coherent store root (the `blobs/ refs/ sources/ origins/ staging/ locks/`
|
|
|
-scaffolding alongside `models/`). The store has **no metadata files at all** — it is
|
|
|
-purely structural, which is why copying only the model dir loses the context. Working
|
|
|
-test shape (verified):
|
|
|
+> **Correction to what R5 concluded here (2026-11-12, R6).** R5 blamed a missing
|
|
|
+> store-root scaffolding for the `open core directory: No such file or directory`
|
|
|
+> failure. That was a **misattribution**. The real cause, proved by the negative control
|
|
|
+> below: that error is what v0.1.0 reports when **`deployments/<deployment-id>/rank-N`
|
|
|
+> is missing or one level off**. R5's failing binds pointed at
|
|
|
+> `deployments/<id>` *as* `deployments/`, so the expected `rank-N` level vanished. Bind a
|
|
|
+> directory that contains `<id>/rank-N` and the very same tree verifies and serves. A
|
|
|
+> store does indeed have no metadata files (purely structural), and the model dir does
|
|
|
+> have to live inside its store — but do **not** go copying `blobs/ refs/ staging/`
|
|
|
+> around on the strength of the old wording; that was never what the error meant.
|
|
|
+
|
|
|
+The plan's original Phase 3 step 1–2 procedure was still wrong in another way: a copied
|
|
|
+`model.safetensors` alone is not a core. Working test shape (verified):
|
|
|
|
|
|
```bash
|
|
|
S=/mnt/nas/ai/flare/store; R2=$HOME/flare-bindtest # any local store root
|
|
|
@@ -799,6 +818,59 @@ completed stage turns out to rename over `deployments/` or to hardlink into it
|
|
|
(cross-device hardlinks fail with EXDEV, as just demonstrated on NFS), take **option D**
|
|
|
without arguing: leave `deployments/` local (1.3 GiB) and move only the package store.
|
|
|
|
|
|
+### The single-GPU test bed — how to close any layout question in 6 minutes, no window
|
|
|
+
|
|
|
+Every “does FLARE tolerate X” question above was answered on **one idle node** with a
|
|
|
+25 GB single-GPU core instead of waiting for the TP4 group. Keep this recipe — it is the
|
|
|
+cheapest way to test a storage or launch change:
|
|
|
+
|
|
|
+```bash
|
|
|
+# local-inference-lab/Qwen3.8-27B-FLARE, branch gb10 → 25.5 GB, 317 s at ~80 MiB/s
|
|
|
+R=$HOME/teststore; mkdir -p $R
|
|
|
+flare download local-inference-lab/Qwen3.8-27B-FLARE --tensor-parallel 1 \
|
|
|
+ --model-cache $R --publisher-keys ~/.cache/flare/publishers/937c0143*.json \
|
|
|
+ --token-file ~/.config/hf/token
|
|
|
+# → model Qwen3.8-27B-NVIDIA-NVFP4-QAD-MXFP8 · gb10-tp1-<ts> · payload v2
|
|
|
+# 23.77 GiB · 69 hardlinked objects · Deployment gb10-tp1 · TP 1 · ready
|
|
|
+# context/prefill 262,144 / 4,096 · weights 22 G · deployments 2.5 G · envelope 18 M
|
|
|
+flare run $R/models/<model>/<version>/model.safetensors --package-cache $R --gpus 0 \
|
|
|
+ --host 127.0.0.1 --port 30080 --publisher-keys ~/.cache/flare/publishers/937c0143*.json
|
|
|
+curl -s http://127.0.0.1:30080/v1/models # 200 in ~20 s; chat completions work
|
|
|
+```
|
|
|
+
|
|
|
+Gotchas measured on the way: **`--rank` and `--rendezvous` are rejected for a single-host
|
|
|
+core** (`--rendezvous applies only to a multi-host core`), the deployment id here is
|
|
|
+human-readable (`gb10-tp1`) instead of a hash, this model streams into
|
|
|
+`reasoning_content` with `content: null`, a 262k context makes the rank reserve ~109 GiB
|
|
|
+of RAM (`MemAvailable 117 → 8 GiB`), and after `pkill` of a rank allow **~10 s** for the
|
|
|
+port to free before restarting (unmapping 109 GiB is not instant — restarting after 3 s
|
|
|
+produced silent failures here).
|
|
|
+
|
|
|
+**Proven with it** (`deployments/` moved to `nodes/<node>/<model>/<version>/deployments`
|
|
|
+on the NAS and bind-mounted over the model directory's `deployments/`):
|
|
|
+
|
|
|
+- rank serves: `api=200`, `20 s` (warm) / `21 s` (cold, after `drop_caches` on the idle node);
|
|
|
+- negative control — bind a NAS dir with **no** rank slice → `[FAILED] Verify core …
|
|
|
+ open core directory: No such file or directory`, i.e. no hidden fallback to disk;
|
|
|
+- **nothing written into the bind**: `find $D/deployments -newermt "$START"` → 0 files,
|
|
|
+ inode identical, mount intact after serving, after restart, and after reboot;
|
|
|
+- **steady-state inference pulls 0 MiB** over the network — the artefacts are read at
|
|
|
+ load, so the NAS cost of a bind is bounded by the rank slice (1.3 GiB for TP4,
|
|
|
+ measured 1.29 s at 1.1 GB/s) and happens once per start.
|
|
|
+
|
|
|
+**Boot persistence (fstab) — validated by rebooting idle dgx4**, the exact stanza Phase 4
|
|
|
+installs, and nothing else in `/etc/fstab` touched (backup `/etc/fstab.bak-20261112-fsbind`):
|
|
|
+
|
|
|
+```
|
|
|
+<mnt-nas>/ai/flare/nodes/<node>/<model>/<version>/deployments /home/kilork/.cache/flare/models/<model>/<version>/deployments none bind,nofail,x-systemd.mount-timeout=30,x-systemd.requires=/mnt/nas,x-systemd.requires-mounts-for=/mnt/nas 0 0
|
|
|
+```
|
|
|
+
|
|
|
+After reboot: `findmnt` shows the bind, `systemctl show <bind-unit> -p Requires` contains
|
|
|
+**`mnt-nas.mount`** (so the bind can never be mounted before the NFS root), and a rank
|
|
|
+started from it **22 s after boot** with the bind intact. Removing it again: delete the
|
|
|
+fstab line, `daemon-reload`, **unmount deepest-first, then** `rm -rf --one-file-system`
|
|
|
+(see the trap above).
|
|
|
+
|
|
|
### Phase 4 — roll out to sparks (window; ranks stop)
|
|
|
|
|
|
Repeat per node in rank order, one node at a time, verifying before moving on.
|