Jelajahi Sumber

docs(progress): R6 round acts 2-3 - full §2 layout validated (envelope file bind), package_cache is a CLI fallback only, rank starts with the NAS unmounted, fleet config drift recorded, orbit push/stage flagged for the window

Alexander Korolev 1 Minggu lalu
induk
melakukan
089fdc5f86
1 mengubah file dengan 38 tambahan dan 0 penghapusan
  1. 38 0
      specs/tech-architecture/RALPH_PROGRESS.md

+ 38 - 0
specs/tech-architecture/RALPH_PROGRESS.md

@@ -410,3 +410,41 @@ deepest-first**, then `rm -rf --one-file-system`: 0 stray binds, `teststore` gon
 `nodes/` holds only the real `gx10-dgx4` artefacts, store still
 `1 model · 1 version · gb10-tp4-202611120031`. dgx4 back at 215 G used / 1.5 T free.
 Serving untouched throughout, including dgx4's reboot.
+
+### Third act: the §2 layout is validated in full, and the rank/NAS dependency question is closed
+
+**The complete target layout now has a passing test.** Beyond the directory bind,
+`model.safetensors` itself was bind-mounted **as a file** (mode 0444) from the NAS over
+the local copy: the rank resolved it, `Start server … 0.8 s`, answered a chat completion,
+and **the file bind survived with the local original byte-for-byte untouched**.
+Real `weights/` + envelope file bind + `deployments/` directory bind = exactly what
+Phase 4 installs, and nothing in FLARE renamed or wrote to either bind.
+
+**The availability risk R4 created is closed by measurement, not argument.** The installed
+binary's own help settles the semantics: `package_cache` is the **“fallback prepared-model
+cache root when `--model-cache` is absent”** — a CLI default, not the rank's setting.
+Confirmed from the fleet: dgx2/dgx3 carry `~/flare-pkgs/gb10` in config while their own
+captured rank command lines pass `--package-cache ~/.cache/flare`, so the group spec/agent
+sets it explicitly. I reverted the NAS value on dgx1 + dgx4 (backups
+`config.toml.bak-20261112-rank-store`; dgx1 `api=200` throughout) and recorded that
+dgx2/dgx3/dgx8 hold three *different* drifted values to normalise in the window. Then the
+decisive test: **with `/mnt/nas` fully unmounted on dgx4, the rank started and answered a
+completion.** The NAS is not on the rank-restart path — while the bind *is* required at
+start (missing bind → no `rank-N` → refusal), which is precisely what makes
+`nofail,x-systemd.requires-mounts-for=/mnt/nas` load-bearing in fstab.
+
+Two finds for the window. FLARE ships the distribution tooling the plan hand-rolls —
+`orbit push --to … --streams N` (parallel TCP **over the fabric**), `orbit stage` (stage
+without starting ranks), `orbit status` / `cache list` (each node's cache) — **currently
+blocked** because re-pairing the controller needs no group running; if it works it removes
+Phase 4's rsync steps and their `EXDEV`/hardlink traps. And `flare models list` loudly
+labels corrupt entries (`[UNREADABLE] … has no primary`), which is how I found and removed
+the last R5 debris (12 K, 0 files, `findmnt` verified 0 binds first, `rm -rf
+--one-file-system`).
+
+Left tidy and reproducible: 0 stray binds, dgx4 at 238 G used / 1.5 T free, NAS 1.6 T/33 T,
+`nodes/` holding only the real `gx10-dgx4` artefacts, and a **24 G TP1 core kept at
+`~/.teststore2` on dgx4** as a standing test bed, so layout or launch questions in later
+rounds cost seconds instead of a 5-minute download. Serving never disturbed: dgx1
+`api=200`, dgx1/2/3/8 all `ranks=1`.
+