Kaynağa Gözat

docs(flare-storage): ranks have no NAS dependency (proved with the NAS unmounted); package_cache is only a CLI fallback; fleet drift recorded

The installed binary's own help says it: package_cache is the "Fallback prepared-model
cache root when --model-cache is absent [default: paths.package_cache...]". So it is
the CLI's fallback, not the rank's cache - confirmed by dgx2/dgx3 carrying
~/flare-pkgs/gb10 in config while their captured rank command lines pass
--package-cache ~/.cache/flare (the group spec/agent sets it explicitly).

R4's NAS value is therefore reverted to ~/.cache/flare on dgx1 and dgx4 (backups
config.toml.bak-20261112-rank-store); dgx1 stayed api=200. Fleet drift table recorded:
dgx2/dgx3 on a skeleton path, dgx8 on the old default - normalise inside the window.

Availability measured, not assumed: with /mnt/nas fully unmounted on dgx4 (0 holders,
clean umount) the rank started and answered a chat completion, so the NAS is not on the
rank-restart path while weights stay local. The bind itself does need the NAS at start
(missing bind = no rank-N = rank refuses), which is what makes
nofail,x-systemd.requires-mounts-for=/mnt/nas load-bearing.

Also from the help: FLARE ships orbit push --to/--streams (fabric TCP), orbit stage
(stage without starting ranks), orbit status and cache list - candidates to replace the
plan's hand-rolled rsync steps and their EXDEV traps. Try on dgx4 with the TP1 core.
R5 debris in dgx4's default store was flagged by FLARE as [UNREADABLE] ... has no
primary; verified 0 binds, then removed with rm -rf --one-file-system.
Alexander Korolev 1 hafta önce
ebeveyn
işleme
64fb5a2457
1 değiştirilmiş dosya ile 62 ekleme ve 5 silme
  1. 62 5
      specs/tech-architecture/flare-storage-nas.md

+ 62 - 5
specs/tech-architecture/flare-storage-nas.md

@@ -35,11 +35,13 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       a NAS-backed `deployments/` (TP1 test bed), negative control failed correctly, no
       writes/rename by FLARE, fstab bind survives reboot with `Requires=mnt-nas.mount`.
       Remaining for the layout: apply it to the TP4 rank slices in the window (R7/R8).
-- [ ] **Open question worth a pre-window check:** every rank startup pulled ~2.6 GB from
-      the NAS *even when `deployments/` was local* (see the measurement caveats). Since
-      R4 pointed `package_cache` at the NAS store, confirm on an idle node that a rank
-      still starts when `/mnt/nas` is unavailable — if it does not, the NAS is on the
-      rank restart path and that must be documented as an availability dependency.
+- [x] **Ranks do not depend on the NAS — proven, and the `package_cache` drift fixed.**
+      A rank started **and answered a chat completion with `/mnt/nas` unmounted**, as long
+      as its model lives in a local store, so the storage design has no NAS dependency on
+      the rank-restart path. `paths.package_cache` turned out to be a **CLI fallback**
+      (see §“What the binary says about cache roots”), not the rank's setting, so R4's
+      NAS value was reverted to `/home/kilork/.cache/flare` on dgx1 + dgx4
+      (backups `config.toml.bak-20261112-rank-store`); dgx1 stayed `api=200` throughout
 - [ ] Sparks rolled out (Phase 4, needs a window)
 - [ ] prod4 rolled out (Phase 5, needs a window)
 - [ ] Verification matrix run and rollback drilled (Phase 6)
@@ -871,6 +873,61 @@ started from it **22 s after boot** with the bind intact. Removing it again: del
 fstab line, `daemon-reload`, **unmount deepest-first, then** `rm -rf --one-file-system`
 (see the trap above).
 
+## 5. What the binary says about cache roots (read before editing any config)
+
+From `flare run --help` in the installed build (not inference):
+
+> `package_cache PACKAGE_CACHE` — **Fallback** prepared-model cache root when
+> `--model-cache` is absent [default: `paths.package_cache`, else `~/.cache/flare/packages`]
+
+So `paths.package_cache` is the **CLI's fallback**, not the rank's cache. Proof from the
+fleet: dgx2/dgx3 config says `package_cache = "/home/kilork/flare-pkgs/gb10"`, yet their
+captured rank command lines both carry `--package-cache /home/kilork/.cache/flare` — the
+group spec / agent passes it **explicitly**. Measured drift across participating nodes
+(2026-11-12), all three values different:
+
+| node | `paths.package_cache` | rank actually used |
+|---|---|---|
+| dgx1 | `~/.cache/flare` (fixed 2026-11-12) | `~/.cache/flare` |
+| dgx2, dgx3 | `~/flare-pkgs/gb10` ← skeleton, drift | `~/.cache/flare` |
+| dgx4 | `~/.cache/flare` | — (no ranks) |
+| dgx8 | `~/.cache/flare/packages` ← old default | `~/.cache/flare` |
+
+**Normalise dgx2/dgx3/dgx8 to `~/.cache/flare` inside the window** (CLI-only edit, but
+serving nodes are not edited outside a window by rule). And never point this key at the
+NAS: it is the fallback, and a fallback that resolves off a network share is a rank that
+might one day try to load 474 G over NFS.
+
+**Availability, measured rather than assumed:** with `/mnt/nas` fully unmounted on dgx4
+(`lsof +D /mnt/nas` → 0 holders, clean `umount`), the TP1 rank started and returned a
+chat completion. The NAS is therefore **not** on the rank-restart path as long as each
+rank's model lives in a local store — which is exactly what this plan guarantees
+(`weights/` local, small artefacts bind-mounted). The bind itself *does* depend on the
+NAS at start (a missing bind means no `rank-N` → the rank refuses to start), so
+`nofail,x-systemd.requires-mounts-for=/mnt/nas` in fstab is load-bearing, not decoration.
+
+Broken entries are labelled, not silently picked: `flare models list` reported
+`gb10-tp4-202611112207 [UNREADABLE] … cache entry … has no primary` for the R5 debris
+(12 K, 0 files). Verified `findmnt` showed 0 binds under it, then removed it with
+`rm -rf --one-file-system`; dgx1/2/3/8 report no such entries.
+
+### FLARE already ships the distribution tools the plan hand-rolls
+
+Also straight from the help text — worth trying **instead of rsync** during the window:
+
+- `orbit push <path|NODE:/abs/path> --to NODE[,…]` — “one per rank in rank order, or any
+  number to receive the complete package”, with `--streams N` “parallel TCP streams per
+  destination file **over the fabric**”, and `--roce-ring` / `--roce-chain` placement.
+- `orbit stage` — “Stage package data on nodes **without starting ranks**”; `orbit ls`,
+  `orbit status` (“each node's capacity, groups, and package cache”), and `cache list
+  [MODEL] [--json]` (“show what every node's package cache holds”).
+- `flare run PACKAGE` accepts a local path, `MODEL[@VERSION]`, or a HF repo — **“local
+  models resolve first”**, and `--verify-weights` exists to re-check stored ranges.
+
+If `orbit push` writes rank slices directly into each node's local store, Phase 4's
+rsync steps (and their `EXDEV`/hardlink traps) disappear. Try it on dgx4 with the TP1
+core first.
+
 ### Phase 4 — roll out to sparks (window; ranks stop)
 
 Repeat per node in rank order, one node at a time, verifying before moving on.