|
|
@@ -35,11 +35,13 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
a NAS-backed `deployments/` (TP1 test bed), negative control failed correctly, no
|
|
|
writes/rename by FLARE, fstab bind survives reboot with `Requires=mnt-nas.mount`.
|
|
|
Remaining for the layout: apply it to the TP4 rank slices in the window (R7/R8).
|
|
|
-- [ ] **Open question worth a pre-window check:** every rank startup pulled ~2.6 GB from
|
|
|
- the NAS *even when `deployments/` was local* (see the measurement caveats). Since
|
|
|
- R4 pointed `package_cache` at the NAS store, confirm on an idle node that a rank
|
|
|
- still starts when `/mnt/nas` is unavailable — if it does not, the NAS is on the
|
|
|
- rank restart path and that must be documented as an availability dependency.
|
|
|
+- [x] **Ranks do not depend on the NAS — proven, and the `package_cache` drift fixed.**
|
|
|
+ A rank started **and answered a chat completion with `/mnt/nas` unmounted**, as long
|
|
|
+ as its model lives in a local store, so the storage design has no NAS dependency on
|
|
|
+ the rank-restart path. `paths.package_cache` turned out to be a **CLI fallback**
|
|
|
+ (see §“What the binary says about cache roots”), not the rank's setting, so R4's
|
|
|
+ NAS value was reverted to `/home/kilork/.cache/flare` on dgx1 + dgx4
|
|
|
+ (backups `config.toml.bak-20261112-rank-store`); dgx1 stayed `api=200` throughout
|
|
|
- [ ] Sparks rolled out (Phase 4, needs a window)
|
|
|
- [ ] prod4 rolled out (Phase 5, needs a window)
|
|
|
- [ ] Verification matrix run and rollback drilled (Phase 6)
|
|
|
@@ -871,6 +873,61 @@ started from it **22 s after boot** with the bind intact. Removing it again: del
|
|
|
fstab line, `daemon-reload`, **unmount deepest-first, then** `rm -rf --one-file-system`
|
|
|
(see the trap above).
|
|
|
|
|
|
+## 5. What the binary says about cache roots (read before editing any config)
|
|
|
+
|
|
|
+From `flare run --help` in the installed build (not inference):
|
|
|
+
|
|
|
+> `package_cache PACKAGE_CACHE` — **Fallback** prepared-model cache root when
|
|
|
+> `--model-cache` is absent [default: `paths.package_cache`, else `~/.cache/flare/packages`]
|
|
|
+
|
|
|
+So `paths.package_cache` is the **CLI's fallback**, not the rank's cache. Proof from the
|
|
|
+fleet: dgx2/dgx3 config says `package_cache = "/home/kilork/flare-pkgs/gb10"`, yet their
|
|
|
+captured rank command lines both carry `--package-cache /home/kilork/.cache/flare` — the
|
|
|
+group spec / agent passes it **explicitly**. Measured drift across participating nodes
|
|
|
+(2026-11-12), all three values different:
|
|
|
+
|
|
|
+| node | `paths.package_cache` | rank actually used |
|
|
|
+|---|---|---|
|
|
|
+| dgx1 | `~/.cache/flare` (fixed 2026-11-12) | `~/.cache/flare` |
|
|
|
+| dgx2, dgx3 | `~/flare-pkgs/gb10` ← skeleton, drift | `~/.cache/flare` |
|
|
|
+| dgx4 | `~/.cache/flare` | — (no ranks) |
|
|
|
+| dgx8 | `~/.cache/flare/packages` ← old default | `~/.cache/flare` |
|
|
|
+
|
|
|
+**Normalise dgx2/dgx3/dgx8 to `~/.cache/flare` inside the window** (CLI-only edit, but
|
|
|
+serving nodes are not edited outside a window by rule). And never point this key at the
|
|
|
+NAS: it is the fallback, and a fallback that resolves off a network share is a rank that
|
|
|
+might one day try to load 474 G over NFS.
|
|
|
+
|
|
|
+**Availability, measured rather than assumed:** with `/mnt/nas` fully unmounted on dgx4
|
|
|
+(`lsof +D /mnt/nas` → 0 holders, clean `umount`), the TP1 rank started and returned a
|
|
|
+chat completion. The NAS is therefore **not** on the rank-restart path as long as each
|
|
|
+rank's model lives in a local store — which is exactly what this plan guarantees
|
|
|
+(`weights/` local, small artefacts bind-mounted). The bind itself *does* depend on the
|
|
|
+NAS at start (a missing bind means no `rank-N` → the rank refuses to start), so
|
|
|
+`nofail,x-systemd.requires-mounts-for=/mnt/nas` in fstab is load-bearing, not decoration.
|
|
|
+
|
|
|
+Broken entries are labelled, not silently picked: `flare models list` reported
|
|
|
+`gb10-tp4-202611112207 [UNREADABLE] … cache entry … has no primary` for the R5 debris
|
|
|
+(12 K, 0 files). Verified `findmnt` showed 0 binds under it, then removed it with
|
|
|
+`rm -rf --one-file-system`; dgx1/2/3/8 report no such entries.
|
|
|
+
|
|
|
+### FLARE already ships the distribution tools the plan hand-rolls
|
|
|
+
|
|
|
+Also straight from the help text — worth trying **instead of rsync** during the window:
|
|
|
+
|
|
|
+- `orbit push <path|NODE:/abs/path> --to NODE[,…]` — “one per rank in rank order, or any
|
|
|
+ number to receive the complete package”, with `--streams N` “parallel TCP streams per
|
|
|
+ destination file **over the fabric**”, and `--roce-ring` / `--roce-chain` placement.
|
|
|
+- `orbit stage` — “Stage package data on nodes **without starting ranks**”; `orbit ls`,
|
|
|
+ `orbit status` (“each node's capacity, groups, and package cache”), and `cache list
|
|
|
+ [MODEL] [--json]` (“show what every node's package cache holds”).
|
|
|
+- `flare run PACKAGE` accepts a local path, `MODEL[@VERSION]`, or a HF repo — **“local
|
|
|
+ models resolve first”**, and `--verify-weights` exists to re-check stored ranges.
|
|
|
+
|
|
|
+If `orbit push` writes rank slices directly into each node's local store, Phase 4's
|
|
|
+rsync steps (and their `EXDEV`/hardlink traps) disappear. Try it on dgx4 with the TP1
|
|
|
+core first.
|
|
|
+
|
|
|
### Phase 4 — roll out to sparks (window; ranks stop)
|
|
|
|
|
|
Repeat per node in rank order, one node at a time, verifying before moving on.
|