|
|
@@ -43,8 +43,12 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
all (§ Phase 0.1), the tagged v0.1.0 binary does it at 55–78 MiB/s ⇒ ETA ≈ 2 h.
|
|
|
Also done in this round: v0.1.0 **staged** on dgx1/2/3/4/8, and SGLang (Ripple
|
|
|
eval server) **stopped on dgx4** — 103 GiB → 4 GiB RAM used
|
|
|
-- [ ] **R4 (Q1)** — Phase 2R second half: fix dgx1's `package_cache` drift (it points
|
|
|
- at an empty `~/flare-pkgs/gb10`), confirm the launched rank cmdline (open item 6.1)
|
|
|
+- [x] **R4 (Q1)** — `package_cache` now `/mnt/nas/ai/flare/store` on dgx4 + dgx1
|
|
|
+ (backups kept; dgx1 rank untouched, `api=200`, same PID). Answered the config
|
|
|
+ question: v0.1.0 still honours `paths.package_cache`, the launcher never passes
|
|
|
+ the cache flag, and the flag renamed `--package-cache` → `--core-cache` — see the
|
|
|
+ new “v0.1.0 vocabulary and paths” section. Still open: confirm the *launched*
|
|
|
+ cmdline after the first restart (needs the window)
|
|
|
- [ ] **R5 (Q1)** — Phase 0.1 as rewritten: pre-flight that the packaged release is
|
|
|
the binary every node starts into at cutover (no PR-184 rebuild — dropped)
|
|
|
- [ ] **R6 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4
|
|
|
@@ -501,6 +505,71 @@ the superseded 477 G on dgx1 and the 261 G dirs everywhere, start on the new pac
|
|
|
bind-mount work on the old version dirs would be wasted: the paths contain the version
|
|
|
name, and every fstab entry would have to be rewritten.
|
|
|
|
|
|
+### v0.1.0 vocabulary and paths — read before running any command (2026-11-11, R4)
|
|
|
+
|
|
|
+The runtime we are moving to renamed the artefact. Measured against both binaries on
|
|
|
+dgx4 (`/usr/bin/flare` = Nov-5, `/usr/local/share/hyperflare/v0.1.0/flare`):
|
|
|
+
|
|
|
+| | Nov-5 build (installed) | v0.1.0 (staged) |
|
|
|
+|---|---|---|
|
|
|
+| artefact name | package | **core** (“Serve a local **core**”, “Download a prepared **core**”) |
|
|
|
+| serve/orbit flag | `--package-cache` | **`--core-cache`** (`run`, `orbit run`, `models list`) |
|
|
|
+| download flag | `--model-cache` | `--model-cache` (unchanged — the two flags coexist, do not “fix” one into the other) |
|
|
|
+| default store root | `~/.cache/flare/packages` | `~/.cache/flare/cores` |
|
|
|
+| config key | `paths.package_cache` | **still `paths.package_cache`** — verified: with `package_cache` set, `models list` reports that path under both binaries |
|
|
|
+| store root layout | `blobs/ refs/ staging/ locks/ sources/ origins/ models/` | same, plus `transport/` (the HF download cache) |
|
|
|
+
|
|
|
+Consequences that actually bite:
|
|
|
+
|
|
|
+1. **No automatic migration exists** — neither binary contains a
|
|
|
+ `migrat*`-with-core/package string. An upgraded node simply looks somewhere else
|
|
|
+ unless `paths.package_cache` says otherwise. Since the fleet pins
|
|
|
+ `package_cache` in `~/.config/flare/config.toml`, **that key is what keeps the
|
|
|
+ upgrade from orphaning 474 GiB** — do not let a node end up without it.
|
|
|
+2. **The launcher needs no change for this.** It never passes the cache flag
|
|
|
+ (`grep -rn "package-cache" src/` → only a comment in `flare_package.rs`); the agent
|
|
|
+ synthesises it. So the flag rename travels with the binary swap, not with our
|
|
|
+ config. Confirm in the window anyway:
|
|
|
+ `tr '\0' ' ' < /proc/$(pgrep -f "[f]lare run" | head -1)/cmdline`.
|
|
|
+3. **`flare models` is now a store manager** — `list, inspect, download, downloads,
|
|
|
+ select, remove, gc`. Use `models remove` / `models gc` for the superseded build
|
|
|
+ instead of `rm -rf`, and `models select` to pin the default version.
|
|
|
+ `models downloads` inspects the transport cache. **Open question for the
|
|
|
+ verification step:** whether `transport/` keeps a second copy of the payload (⇒
|
|
|
+ ~950 GiB on the NAS until `models gc`); check `du -sh /mnt/nas/ai/flare/store/*`
|
|
|
+ and `models downloads` once the acquisition lands. **Half-answered mid-flight**: at
|
|
|
+ 39 GiB downloaded, *all* of it sits in `transport/` (40 K in `locks`, 24 K in
|
|
|
+ `staging`, 8 K in `blobs`) — so materialisation into `models/` happens only after
|
|
|
+ the whole payload is cached, and `staging/trees/<pid>-<n>` accumulates one tree per
|
|
|
+ attempt (the killed Nov-5 attempt is still there as `1800521-0`).
|
|
|
+4. `flare trust` exists now (`revoke`), for publisher trust management.
|
|
|
+5. **Node clocks disagree by an hour** (dgx1 `23:14` while dgx4 `22:14` for the same
|
|
|
+ instant). Never compare timestamps across nodes; filename your backups with the
|
|
|
+ local node's clock and compare *mtime within a node* only.
|
|
|
+
|
|
|
+### R4 — CLI store pointed at the NAS store (done 2026-11-11)
|
|
|
+
|
|
|
+Applied on **dgx4** (the acquiring node) and **dgx1** (prod rank 0):
|
|
|
+
|
|
|
+```
|
|
|
+package_cache = "/mnt/nas/ai/flare/store" # NAS store (2026-11-11, R4); was /home/kilork/flare-pkgs/gb10 (empty skeleton)
|
|
|
+```
|
|
|
+
|
|
|
+Backups: `~/.config/flare/config.toml.bak-2026-11-11-2214-nas-store` (dgx4),
|
|
|
+`…-2314-nas-store` (dgx1). Verify: `flare models list` prints
|
|
|
+`Storage /mnt/nas/ai/flare/store`. **dgx1's rank was untouched** — same PID
|
|
|
+(533700), 7.5-day uptime, `api=200` — because the rank's cache path comes from the
|
|
|
+agent's explicit flag, not from this key.
|
|
|
+
|
|
|
+What the old value was doing: it deliberately equalled the recipe's `local_dir`
|
|
|
+(`/home/kilork/flare-pkgs/gb10`) so that the launcher's rsync mirror would see
|
|
|
+`src == dst` and skip copying, materialising the ~500 GB package once. In practice
|
|
|
+the mirror never filled up (the agent writes to `~/.cache/flare/packages`), so the
|
|
|
+key pointed at a 68 K empty skeleton and `flare models list` reported “No local model
|
|
|
+versions found” while 477 G sat elsewhere. With the store on the NAS, the
|
|
|
+`local_dir` mirror is obsolete — retire it in the Phase 5 window rather than now, so
|
|
|
+the launcher keeps a working path until the cutover.
|
|
|
+
|
|
|
### Phase 3 — validate the bind-mount model on an idle node (gate)
|
|
|
|
|
|
Do this **before** touching dgx1–dgx3.
|