Jelajahi Sumber

docs(flare-storage): v0.1.0 renames package->core; CLI store now on the NAS (R4)

Measured with both binaries side by side on dgx4 (installed Nov-5 vs staged v0.1.0):
"prepared package" -> prepared "core"; --package-cache -> --core-cache on run/orbit
run/models list; default store root ~/.cache/flare/packages -> ~/.cache/flare/cores;
flare download keeps --model-cache so both flags coexist. BUT paths.package_cache is
unchanged - with it set, both binaries report the configured path. Neither binary has
a migrate* string for core/package, so nothing moves automatically and that one config
key is what stops an upgrade orphaning hundreds of GB.

R4 applied: package_cache = /mnt/nas/ai/flare/store on dgx4 (acquiring node) and dgx1
(prod rank 0), replacing the value pointing at the 68 K empty ~/flare-pkgs/gb10
skeleton. Backups kept. Verified: models list reports the NAS path, and dgx1's rank is
untouched (same PID 533700, 7.5 d uptime, api=200) because the rank's cache path comes
from the agent's explicit flag, not this key. The old value was deliberate (it equalled
the recipe's local_dir so the rsync mirror saw src==dst) but the mirror never filled;
retire local_dir in the Phase 5 window, not now.

Launcher needs no change for the rename: grep -rn "package-cache" src/ finds only a
comment - the agent synthesises the flag, so it follows the binary swap. Launched
cmdline still to be confirmed in the window (open item 6.1).

Store mechanics: at 39 GiB downloaded every byte is in <store>/transport/ (models/,
blobs/, refs/ still empty) and staging/trees/<pid>-<n> keeps one tree per attempt, so
materialisation happens only after the full payload is cached - expect a ~2x peak and
use the new `flare models gc` (models also gained download/downloads/select/remove).
Also: node clocks disagree by an hour (dgx1 23:14 vs dgx4 22:14 same instant).
Alexander Korolev 1 Minggu lalu
induk
melakukan
5a1374505a

+ 5 - 10
specs/tech-architecture/.rounds-state.flare-storage-nas.json

@@ -1,29 +1,24 @@
 {
   "objective": "/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md",
-  "round": 3,
+  "round": 4,
   "maxRounds": 12,
   "turnsBudget": 60,
   "progressEntries": 12,
   "verifyTimeout": 120,
   "minutesBudget": null,
-  "lastObjectiveHash": "755fc9ef-33963",
+  "lastObjectiveHash": "c3766b67-37083",
   "attemptsBudget": 3,
-  "attemptsUsed": 1,
+  "attemptsUsed": 0,
   "lastCheck": null,
   "todoWatch": [
-    {
-      "id": 9,
-      "subject": "Compile table + conclusion",
-      "firstSeen": 1
-    },
     {
       "id": 12,
-      "subject": "Migrate flare model root to NAS in a maintenance window",
+      "subject": "Deploy the current gb10 package from the NAS store",
       "firstSeen": 1
     }
   ],
   "active": true,
   "finished": false,
   "lastError": null,
-  "updated": "2026-11-11T22:10:11.979Z"
+  "updated": "2026-11-11T22:12:57.406Z"
 }

+ 59 - 4
specs/tech-architecture/RALPH_PROGRESS.md

@@ -98,10 +98,6 @@ reversed to “prod runs the latest release build”, with the package-gates-run
 rule 5 adds “keep the flag set aligned with the package README, deviate only where
 documented”.
 
-## Loop log
-- round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…
-- round 2 · -- · 74 turns · continue → continue · (no ROUND line)
-
 ## 2026-11-11 — R3 (flare-storage-nas): the runtime is the blocker, not the disk
 
 **Goal quadrant: impact × effort — acquire the current `gb10` package onto the NAS
@@ -161,3 +157,62 @@ source); `references/build.md` gained “Distribution channels (2026-11-11)” w
 staging recipe; `references/troubleshooting.md` gained the acquisition section (verify
 hang, `--tensor-parallel 4`, publisher trust, token file, pkill self-match, rank-0-only
 API).
+
+## Loop log
+- round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…
+- round 2 · -- · 74 turns · continue → continue · (no ROUND line)
+- round 3 · -- · 36 turns · continue → continue · outcome
+
+## 2026-11-11 — R4 (flare-storage-nas): v0.1.0 renamed the artefact; the CLI store is on the NAS
+
+**Goal quadrant: impact × effort — close R3's verification and do R4 (the config drift)
+while the 475 GiB acquisition streams.** R4 is done and verified; R3's gate is still
+streaming (see below).
+
+**The runtime we are moving to speaks a different vocabulary, measured not assumed.**
+Side by side on dgx4 (`/usr/bin/flare` Nov-5 vs the staged v0.1.0): “prepared package”
+→ **prepared “core”**, `--package-cache` → **`--core-cache`** (on `run`, `orbit run`,
+`models list`), default store root `~/.cache/flare/packages` → `~/.cache/flare/cores`,
+and `flare download` keeps `--model-cache` (the two flags coexist — do not “fix” one
+into the other). But **`paths.package_cache` is unchanged**: with that key set, both
+binaries report the configured path from `models list`. And **nothing migrates
+automatically** — neither binary has a `migrat*` string for core/package. So that one
+config key is what prevents an upgrade from orphaning hundreds of GB.
+
+**R4 applied:** `package_cache = "/mnt/nas/ai/flare/store"` on **dgx4** (acquiring node)
+and **dgx1** (prod rank 0), replacing the value that pointed at
+`/home/kilork/flare-pkgs/gb10` — a 68 K empty skeleton. Backups
+`config.toml.bak-2026-11-11-2214-nas-store` (dgx4) / `…-2314-nas-store` (dgx1).
+Verified: `flare models list` → `Storage /mnt/nas/ai/flare/store`, and **dgx1's rank is
+untouched** — same PID 533700, 7.5-day uptime, `api=200` — because the rank's cache
+path comes from the agent's explicit flag, not this key. The old value was deliberate
+(it equalled the recipe's `local_dir` so the rsync mirror saw `src == dst`), but the
+mirror never filled, so the key was pointing at nothing; retire `local_dir` in the
+Phase 5 window, not now.
+
+**The launcher needs no change for the flag rename** — `grep -rn "package-cache" src/`
+finds only a comment; the agent synthesises the flag, so it travels with the binary
+swap. Still to confirm in the window: the launched cmdline itself (open item 6.1).
+
+**Store mechanics learned mid-flight**, which change the verification step: at 39 GiB
+downloaded, *all* bytes sit in `<store>/transport/` — `models/`, `blobs/`, `refs/` are
+still empty skeletons, and `staging/trees/<pid>-<n>` has one tree per attempt (the
+killed Nov-5 attempt is still there). So materialisation happens only after the whole
+payload is cached, expect a peak near 2× the payload, and `flare models gc` is the tool
+for it — v0.1.0's `flare models` gained `download · downloads · select · remove · gc`,
+i.e. real store management (`models remove`/`gc` beat `rm -rf`, `models select` pins a
+default version). `flare trust` also exists now.
+
+**Node clocks disagree by an hour** (dgx1 `23:14` vs dgx4 `22:14` at the same instant) —
+never compare timestamps across nodes.
+
+Acquisition state at last check: 38.1 GiB at 420 s, **93 MiB/s**, alive, one process,
+`--tensor-parallel 4`, target ≈ 23:32 UTC. R3's gate (`models list` showing the version)
+is the next round's first action if it does not land this round.
+
+Objective file: new “v0.1.0 vocabulary and paths” section (side-by-side table + the five
+consequences that bite), “R4 — CLI store pointed at the NAS store” (applied value,
+backup paths, verification, why the old value existed), R4 ticked. Skill
+`references/storage.md`: same vocabulary table, the no-migration warning, the agent
+synthesises-the-flag note, the `models` subcommands, and the transport-first layout with
+its 2× peak.

+ 71 - 2
specs/tech-architecture/flare-storage-nas.md

@@ -43,8 +43,12 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       all (§ Phase 0.1), the tagged v0.1.0 binary does it at 55–78 MiB/s ⇒ ETA ≈ 2 h.
       Also done in this round: v0.1.0 **staged** on dgx1/2/3/4/8, and SGLang (Ripple
       eval server) **stopped on dgx4** — 103 GiB → 4 GiB RAM used
-- [ ] **R4 (Q1)** — Phase 2R second half: fix dgx1's `package_cache` drift (it points
-      at an empty `~/flare-pkgs/gb10`), confirm the launched rank cmdline (open item 6.1)
+- [x] **R4 (Q1)** — `package_cache` now `/mnt/nas/ai/flare/store` on dgx4 + dgx1
+      (backups kept; dgx1 rank untouched, `api=200`, same PID). Answered the config
+      question: v0.1.0 still honours `paths.package_cache`, the launcher never passes
+      the cache flag, and the flag renamed `--package-cache` → `--core-cache` — see the
+      new “v0.1.0 vocabulary and paths” section. Still open: confirm the *launched*
+      cmdline after the first restart (needs the window)
 - [ ] **R5 (Q1)** — Phase 0.1 as rewritten: pre-flight that the packaged release is
       the binary every node starts into at cutover (no PR-184 rebuild — dropped)
 - [ ] **R6 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4
@@ -501,6 +505,71 @@ the superseded 477 G on dgx1 and the 261 G dirs everywhere, start on the new pac
 bind-mount work on the old version dirs would be wasted: the paths contain the version
 name, and every fstab entry would have to be rewritten.
 
+### v0.1.0 vocabulary and paths — read before running any command (2026-11-11, R4)
+
+The runtime we are moving to renamed the artefact. Measured against both binaries on
+dgx4 (`/usr/bin/flare` = Nov-5, `/usr/local/share/hyperflare/v0.1.0/flare`):
+
+| | Nov-5 build (installed) | v0.1.0 (staged) |
+|---|---|---|
+| artefact name | package | **core** (“Serve a local **core**”, “Download a prepared **core**”) |
+| serve/orbit flag | `--package-cache` | **`--core-cache`** (`run`, `orbit run`, `models list`) |
+| download flag | `--model-cache` | `--model-cache` (unchanged — the two flags coexist, do not “fix” one into the other) |
+| default store root | `~/.cache/flare/packages` | `~/.cache/flare/cores` |
+| config key | `paths.package_cache` | **still `paths.package_cache`** — verified: with `package_cache` set, `models list` reports that path under both binaries |
+| store root layout | `blobs/ refs/ staging/ locks/ sources/ origins/ models/` | same, plus `transport/` (the HF download cache) |
+
+Consequences that actually bite:
+
+1. **No automatic migration exists** — neither binary contains a
+   `migrat*`-with-core/package string. An upgraded node simply looks somewhere else
+   unless `paths.package_cache` says otherwise. Since the fleet pins
+   `package_cache` in `~/.config/flare/config.toml`, **that key is what keeps the
+   upgrade from orphaning 474 GiB** — do not let a node end up without it.
+2. **The launcher needs no change for this.** It never passes the cache flag
+   (`grep -rn "package-cache" src/` → only a comment in `flare_package.rs`); the agent
+   synthesises it. So the flag rename travels with the binary swap, not with our
+   config. Confirm in the window anyway:
+   `tr '\0' ' ' < /proc/$(pgrep -f "[f]lare run" | head -1)/cmdline`.
+3. **`flare models` is now a store manager** — `list, inspect, download, downloads,
+   select, remove, gc`. Use `models remove` / `models gc` for the superseded build
+   instead of `rm -rf`, and `models select` to pin the default version.
+   `models downloads` inspects the transport cache. **Open question for the
+   verification step:** whether `transport/` keeps a second copy of the payload (⇒
+   ~950 GiB on the NAS until `models gc`); check `du -sh /mnt/nas/ai/flare/store/*`
+   and `models downloads` once the acquisition lands. **Half-answered mid-flight**: at
+   39 GiB downloaded, *all* of it sits in `transport/` (40 K in `locks`, 24 K in
+   `staging`, 8 K in `blobs`) — so materialisation into `models/` happens only after
+   the whole payload is cached, and `staging/trees/<pid>-<n>` accumulates one tree per
+   attempt (the killed Nov-5 attempt is still there as `1800521-0`).
+4. `flare trust` exists now (`revoke`), for publisher trust management.
+5. **Node clocks disagree by an hour** (dgx1 `23:14` while dgx4 `22:14` for the same
+   instant). Never compare timestamps across nodes; filename your backups with the
+   local node's clock and compare *mtime within a node* only.
+
+### R4 — CLI store pointed at the NAS store (done 2026-11-11)
+
+Applied on **dgx4** (the acquiring node) and **dgx1** (prod rank 0):
+
+```
+package_cache = "/mnt/nas/ai/flare/store"   # NAS store (2026-11-11, R4); was /home/kilork/flare-pkgs/gb10 (empty skeleton)
+```
+
+Backups: `~/.config/flare/config.toml.bak-2026-11-11-2214-nas-store` (dgx4),
+`…-2314-nas-store` (dgx1). Verify: `flare models list` prints
+`Storage /mnt/nas/ai/flare/store`. **dgx1's rank was untouched** — same PID
+(533700), 7.5-day uptime, `api=200` — because the rank's cache path comes from the
+agent's explicit flag, not from this key.
+
+What the old value was doing: it deliberately equalled the recipe's `local_dir`
+(`/home/kilork/flare-pkgs/gb10`) so that the launcher's rsync mirror would see
+`src == dst` and skip copying, materialising the ~500 GB package once. In practice
+the mirror never filled up (the agent writes to `~/.cache/flare/packages`), so the
+key pointed at a 68 K empty skeleton and `flare models list` reported “No local model
+versions found” while 477 G sat elsewhere. With the store on the NAS, the
+`local_dir` mirror is obsolete — retire it in the Phase 5 window rather than now, so
+the launcher keeps a working path until the cutover.
+
 ### Phase 3 — validate the bind-mount model on an idle node (gate)
 
 Do this **before** touching dgx1–dgx3.

+ 1 - 0
specs/tech-architecture/rounds-history.flare-storage-nas.jsonl

@@ -1,2 +1,3 @@
 {"timestamp":"2026-11-11T21:49:43.957Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":1,"goal":"Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every powered node (quadrant Q1)","quadrant":null,"turns":27,"marker":"continue","decision":"continue","reason":null,"check":null}
 {"timestamp":"2026-11-11T22:03:33.500Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":2,"goal":null,"quadrant":null,"turns":74,"marker":"continue","decision":"continue","reason":null,"check":null}
+{"timestamp":"2026-11-11T22:12:57.404Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":3,"goal":"outcome","quadrant":null,"turns":36,"marker":"continue","decision":"continue","reason":null,"check":null}