Преглед изворни кода

docs(flare-storage): R6b closed without a window on a 25 GB TP1 core; option D not needed; fstab bind survives reboot; R5's store-root claim corrected

Answered every open layout question on ONE idle node instead of waiting for the TP4
group, using local-inference-lab/Qwen3.8-27B-FLARE (branch gb10, 25.5 GB in 317 s) ->
Qwen3.8-27B-NVIDIA-NVFP4-QAD-MXFP8 gb10-tp1, TP 1 ready, 23.77 GiB. A rank SERVED from
a deployments/ bind-mounted off the NAS (api=200 in 20 s warm / 21 s cold after
drop_caches) and answered chat completions. Negative control: binding a NAS dir with
no rank slice fails with "open core directory: No such file or directory" - no hidden
fallback, so the data provably comes through the bind.

Across a full load+serve cycle FLARE wrote nothing into the bind (0 files newer than
rank start), never replaced it (same inode, still mounted after serving, restart and
reboot) => OPTION D IS NOT NEEDED. Steady-state inference pulls 0 MiB: the bind cost
is bounded by the rank slice and paid once at start.

fstab persistence validated by rebooting idle dgx4 with the exact Phase 4 stanza:
bind present at boot, its unit provably Requires=mnt-nas.mount, rank up 22 s after
boot. Cleanup done the taught way: fstab line, daemon-reload, unmount deepest-first,
then rm -rf --one-file-system; 0 stray binds left, store intact.

Correction to R5: the "open core directory" error is NOT about missing store-root
scaffolding - it is what v0.1.0 reports when deployments/<id>/rank-N is missing or one
level off (R5's bind put <id> where deployments/ should be). Recorded, plus the new
open question: every rank startup pulled ~2.6 GB from the NAS even with deployments
local, so a rank-restart-without-NAS check is now a pre-window item.
Alexander Korolev пре 1 недеља
родитељ
комит
1cf8f3c54b

+ 8 - 3
specs/tech-architecture/.rounds-state.flare-storage-nas.json

@@ -1,12 +1,12 @@
 {
   "objective": "/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md",
-  "round": 5,
+  "round": 6,
   "maxRounds": 12,
   "turnsBudget": 60,
   "progressEntries": 12,
   "verifyTimeout": 120,
   "minutesBudget": null,
-  "lastObjectiveHash": "538560e0-44582",
+  "lastObjectiveHash": "9396e96d-54510",
   "attemptsBudget": 3,
   "attemptsUsed": 0,
   "lastCheck": null,
@@ -15,10 +15,15 @@
       "id": 12,
       "subject": "Deploy the current gb10 package from the NAS store",
       "firstSeen": 1
+    },
+    {
+      "id": 15,
+      "subject": "Prove deployment reads traverse the bind (R6b, in the window)",
+      "firstSeen": 5
     }
   ],
   "active": true,
   "finished": false,
   "lastError": null,
-  "updated": "2026-11-12T00:18:07.696Z"
+  "updated": "2026-11-12T00:54:59.662Z"
 }

+ 7 - 6
specs/tech-architecture/RALPH_PROGRESS.md

@@ -250,12 +250,6 @@ Next: **R6 (Phase 3 gate) on dgx4** — now unblocked, with real paths in the do
 the window (R7/R8) which is also what lands v0.1.0 as the *running* binary and frees
 the superseded 477 G.
 
-## Loop log
-- round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…
-- round 2 · -- · 74 turns · continue → continue · (no ROUND line)
-- round 3 · -- · 36 turns · continue → continue · outcome
-- round 4 · -- · 28 turns · continue → continue · outcome
-
 ## 2026-11-12 — R5 round (flare-storage-nas): the bind-mount design works, and I broke a core while testing it
 
 **Round goal was R6 (Phase 3 gate) on idle dgx4. Half of it is now proven, and the plan
@@ -360,3 +354,10 @@ stable** (every reference to `db20209d…` in the plan stayed valid). `flare mod
 wants `MODEL@VERSION`, not `MODEL/VERSION`. Per-node artefacts were rebuilt for the new
 version (5.1 G). NAS at 1.6 T/33 T. Serving untouched throughout: dgx1/2/3/8 ranks=1,
 dgx1 api=200.
+
+## Loop log
+- round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…
+- round 2 · -- · 74 turns · continue → continue · (no ROUND line)
+- round 3 · -- · 36 turns · continue → continue · outcome
+- round 4 · -- · 28 turns · continue → continue · outcome
+- round 5 · -- · 54 turns · continue → continue · validate the bind-mount model on idle dgx4 — Phase 3 gate…

+ 85 - 13
specs/tech-architecture/flare-storage-nas.md

@@ -31,7 +31,15 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       signatures verified, visible from dgx1/dgx4 over NFS (Phase 2R)
 - [ ] That core **deployed to the ranks, and the superseded build deleted**
       (needs the window; dgx1's 477 G and the ranks' 261 G dirs are still the live data)
-- [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
+- [x] **Bind-mount layout validated on idle dgx4 (Phase 3 gate)** — a rank served from
+      a NAS-backed `deployments/` (TP1 test bed), negative control failed correctly, no
+      writes/rename by FLARE, fstab bind survives reboot with `Requires=mnt-nas.mount`.
+      Remaining for the layout: apply it to the TP4 rank slices in the window (R7/R8).
+- [ ] **Open question worth a pre-window check:** every rank startup pulled ~2.6 GB from
+      the NAS *even when `deployments/` was local* (see the measurement caveats). Since
+      R4 pointed `package_cache` at the NAS store, confirm on an idle node that a rank
+      still starts when `/mnt/nas` is unavailable — if it does not, the NAS is on the
+      rank restart path and that must be documented as an availability dependency.
 - [ ] Sparks rolled out (Phase 4, needs a window)
 - [ ] prod4 rolled out (Phase 5, needs a window)
 - [ ] Verification matrix run and rollback drilled (Phase 6)
@@ -69,10 +77,16 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       five stage attempts. Also found: the plan's Phase 3 procedure was **invalid**
       (a standalone model-dir copy is rejected in v0.1.0) and there is a **data-loss
       trap** in Phase 4 cleanup — see “Phase 3 gate results”. No data was lost.
-- [ ] **R6b (Q2)** — part 2 needs a *completed* TP4 stage (peers, so it belongs to the
-      window): do deployment reads really traverse the bind, and does FLARE ever
-      replace `deployments/` by rename or hardlink into it (both would break a
-      cross-device bind)? If either happens → **option D**.
+- [x] **R6b (Q1)** — **closed without a window**, on a single-GPU core (see “The
+      single-GPU test bed”): a rank **served chat completions with `deployments/`
+      bind-mounted from the NAS**, and the negative control (bind an empty NAS dir →
+      rank refuses to start) proves the data comes *through* the bind. Across a complete
+      load+serve cycle FLARE wrote **nothing** into the bind (0 files newer than rank
+      start) and never replaced it (same inode, still mounted after serving and after
+      reboot) ⇒ **option D is not needed**. Also validated the **fstab-persisted bind
+      across a reboot of dgx4**: mounted at boot, its unit provably
+      `Requires=mnt-nas.mount`, and a rank came up from it **22 s after boot**.
+      Steady-state inference touches the NAS **0 MiB**.
 - [ ] **R7 (Q2)** — Phase 4 sparks rollout (window; ranks stop)
 - [ ] **R8 (Q2)** — Phase 5 prod4 rollout (window; ranks stop)
 - [ ] **R9 (Q3)** — Phase 6 verification matrix and rollback drill
@@ -739,14 +753,19 @@ can get). The bind was unchanged before and after: same inode, same mount,
 (`474.66 GiB`, `181 hardlinked objects`) while `blobs/` and `weights/` were bind mounts
 onto a **different device** (st_dev 80 = NFS vs 66306 = local NVMe).
 
-**The plan's Phase 3 step 1–2 procedure was wrong.** Copying `model.safetensors` +
-`deployments` out of a store and pointing `run`/`inspect` at the copy **fails in v0.1.0**
-at `Verify core` with `open core directory: No such file or directory (os error 2)`: the
-envelope resolves its core **within the store that contains it**, so the model path must
-sit inside a coherent store root (the `blobs/ refs/ sources/ origins/ staging/ locks/`
-scaffolding alongside `models/`). The store has **no metadata files at all** — it is
-purely structural, which is why copying only the model dir loses the context. Working
-test shape (verified):
+> **Correction to what R5 concluded here (2026-11-12, R6).** R5 blamed a missing
+> store-root scaffolding for the `open core directory: No such file or directory`
+> failure. That was a **misattribution**. The real cause, proved by the negative control
+> below: that error is what v0.1.0 reports when **`deployments/<deployment-id>/rank-N`
+> is missing or one level off**. R5's failing binds pointed at
+> `deployments/<id>` *as* `deployments/`, so the expected `rank-N` level vanished. Bind a
+> directory that contains `<id>/rank-N` and the very same tree verifies and serves. A
+> store does indeed have no metadata files (purely structural), and the model dir does
+> have to live inside its store — but do **not** go copying `blobs/ refs/ staging/`
+> around on the strength of the old wording; that was never what the error meant.
+
+The plan's original Phase 3 step 1–2 procedure was still wrong in another way: a copied
+`model.safetensors` alone is not a core. Working test shape (verified):
 
 ```bash
 S=/mnt/nas/ai/flare/store; R2=$HOME/flare-bindtest          # any local store root
@@ -799,6 +818,59 @@ completed stage turns out to rename over `deployments/` or to hardlink into it
 (cross-device hardlinks fail with EXDEV, as just demonstrated on NFS), take **option D**
 without arguing: leave `deployments/` local (1.3 GiB) and move only the package store.
 
+### The single-GPU test bed — how to close any layout question in 6 minutes, no window
+
+Every “does FLARE tolerate X” question above was answered on **one idle node** with a
+25 GB single-GPU core instead of waiting for the TP4 group. Keep this recipe — it is the
+cheapest way to test a storage or launch change:
+
+```bash
+# local-inference-lab/Qwen3.8-27B-FLARE, branch gb10 → 25.5 GB, 317 s at ~80 MiB/s
+R=$HOME/teststore; mkdir -p $R
+flare download local-inference-lab/Qwen3.8-27B-FLARE --tensor-parallel 1 \
+      --model-cache $R --publisher-keys ~/.cache/flare/publishers/937c0143*.json \
+      --token-file ~/.config/hf/token
+# → model Qwen3.8-27B-NVIDIA-NVFP4-QAD-MXFP8 · gb10-tp1-<ts> · payload v2
+#   23.77 GiB · 69 hardlinked objects · Deployment gb10-tp1 · TP 1 · ready
+#   context/prefill 262,144 / 4,096 · weights 22 G · deployments 2.5 G · envelope 18 M
+flare run $R/models/<model>/<version>/model.safetensors --package-cache $R --gpus 0 \
+      --host 127.0.0.1 --port 30080 --publisher-keys ~/.cache/flare/publishers/937c0143*.json
+curl -s http://127.0.0.1:30080/v1/models          # 200 in ~20 s; chat completions work
+```
+
+Gotchas measured on the way: **`--rank` and `--rendezvous` are rejected for a single-host
+core** (`--rendezvous applies only to a multi-host core`), the deployment id here is
+human-readable (`gb10-tp1`) instead of a hash, this model streams into
+`reasoning_content` with `content: null`, a 262k context makes the rank reserve ~109 GiB
+of RAM (`MemAvailable 117 → 8 GiB`), and after `pkill` of a rank allow **~10 s** for the
+port to free before restarting (unmapping 109 GiB is not instant — restarting after 3 s
+produced silent failures here).
+
+**Proven with it** (`deployments/` moved to `nodes/<node>/<model>/<version>/deployments`
+on the NAS and bind-mounted over the model directory's `deployments/`):
+
+- rank serves: `api=200`, `20 s` (warm) / `21 s` (cold, after `drop_caches` on the idle node);
+- negative control — bind a NAS dir with **no** rank slice → `[FAILED] Verify core …
+  open core directory: No such file or directory`, i.e. no hidden fallback to disk;
+- **nothing written into the bind**: `find $D/deployments -newermt "$START"` → 0 files,
+  inode identical, mount intact after serving, after restart, and after reboot;
+- **steady-state inference pulls 0 MiB** over the network — the artefacts are read at
+  load, so the NAS cost of a bind is bounded by the rank slice (1.3 GiB for TP4,
+  measured 1.29 s at 1.1 GB/s) and happens once per start.
+
+**Boot persistence (fstab) — validated by rebooting idle dgx4**, the exact stanza Phase 4
+installs, and nothing else in `/etc/fstab` touched (backup `/etc/fstab.bak-20261112-fsbind`):
+
+```
+<mnt-nas>/ai/flare/nodes/<node>/<model>/<version>/deployments  /home/kilork/.cache/flare/models/<model>/<version>/deployments  none  bind,nofail,x-systemd.mount-timeout=30,x-systemd.requires=/mnt/nas,x-systemd.requires-mounts-for=/mnt/nas  0 0
+```
+
+After reboot: `findmnt` shows the bind, `systemctl show <bind-unit> -p Requires` contains
+**`mnt-nas.mount`** (so the bind can never be mounted before the NFS root), and a rank
+started from it **22 s after boot** with the bind intact. Removing it again: delete the
+fstab line, `daemon-reload`, **unmount deepest-first, then** `rm -rf --one-file-system`
+(see the trap above).
+
 ### Phase 4 — roll out to sparks (window; ranks stop)
 
 Repeat per node in rank order, one node at a time, verifying before moving on.

+ 1 - 0
specs/tech-architecture/rounds-history.flare-storage-nas.jsonl

@@ -2,3 +2,4 @@
 {"timestamp":"2026-11-11T22:03:33.500Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":2,"goal":null,"quadrant":null,"turns":74,"marker":"continue","decision":"continue","reason":null,"check":null}
 {"timestamp":"2026-11-11T22:12:57.404Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":3,"goal":"outcome","quadrant":null,"turns":36,"marker":"continue","decision":"continue","reason":null,"check":null}
 {"timestamp":"2026-11-12T00:18:07.693Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":4,"goal":"outcome","quadrant":null,"turns":28,"marker":"continue","decision":"continue","reason":null,"check":null}
+{"timestamp":"2026-11-12T00:54:59.659Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":5,"goal":"validate the bind-mount model on idle dgx4 — Phase 3 gate (Q2). Outcome: **half the gate passed with measurements, half is provably window-only — and I broke and fully repaired a core while learning it.","quadrant":null,"turns":54,"marker":"continue","decision":"continue","reason":null,"check":null}