Bladeren bron

docs(flare-storage): run the latest release build; take launch params from the package

Operator directives for the cutover, recorded where the next round will read them.

1. FLARE must be the latest release and properly updated on every node. Measured:
   Installed == Candidate == 0.1.0~git20261105002735.0009b89d-1 on dgx1/2/3/4/8 and
   /usr/bin/flare is one identical file (03c8e463) - but dgx1/2/3/8 EXECUTE deleted
   inodes (PR-184), so only dgx4 actually runs the release. readlink /proc/<pid>/exe
   showing "(deleted)" is the tell; hash equality with dpkg is not proof.
   => the PR #184 rebuild task is DROPPED. PR 184 is not in the release (marker
   strings → 0) and predates the current package's minimum runtime, so it is a hard
   functional regression for the new package, not a faster one.

2. Serve parameters come from the current gb10 package README, API port stays 30000
   (not their example's 8000). Diff vs the live rank cmdline: only
   --mtp-adaptive-verification-alpha 0.75 was missing -> added to both
   dsv41-flash-1017-4x-flare-gb10* recipes (upstream: "favor longer verification
   rounds, which raises decode throughput with four or more concurrent requests and
   coding throughput"). --transfer-streams 1 (RoCE poison workaround) and
   --served-model-name stay. --memory-limit / --kv-cache-limit: upstream says keep
   the memory default, and their budget (~11.4 GiB KV/GPU, ~13.1M tokens, 17-18 GiB
   free of 121) matches ours -> no change. Engram row cache stays 1 GiB (their
   default; our 48 GiB probe failed with "selected concurrency and verification
   width exceed available memory").

New "## Runtime parameters" section holds the flag-by-flag table; Phase 0.1 is
rewritten around running the packaged build; §6 gained items 4-7 (how to read a
package's identity and minimum runtime, the (deleted)-inode check, dgx1's 915 GiB
root vs dgx4's 1.8 TiB, and that this fleet's package store is flat - 0 hardlinks /
184 files, so rsync -H was never load-bearing).
Alexander Korolev 1 week geleden
bovenliggende
commit
6a341ad022
2 gewijzigde bestanden met toevoegingen van 145 en 11 verwijderingen
  1. 51 0
      specs/tech-architecture/RALPH_PROGRESS.md
  2. 94 11
      specs/tech-architecture/flare-storage-nas.md

+ 51 - 0
specs/tech-architecture/RALPH_PROGRESS.md

@@ -49,3 +49,54 @@ Watch out for:
 
 ## Loop log
 - round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…
+
+## 2026-11-11 — R2 (flare-storage-nas): purge instead of migrate; what we serve is superseded
+
+**Goal quadrant: impact × effort — delete 783 GiB of dead data and find out whether the
+fleet is even on the current model build.** Both landed.
+
+The operator rejected migrating the old cache (“*Syncing those old flare cache to store
+does not make much sense honestly*”) and was right on the facts, harder than expected:
+
+- Every node held the **Nov-2** build `checkpoint-expanded-scales/gb10-tp4-202611021936`
+  (261 GiB × 3 nodes) with **nothing holding it open** — verified per node via
+  `/proc/<rank>/fd`, `/proc/<rank>/maps` **and** `lsof +D` before deleting. Freed
+  **783 GiB**: dgx1 98 % full → 283 G free, dgx2 → 467 G, dgx3 → 470 G. Rank PIDs
+  untouched (7-day uptime), `dgx1:30000/v1/models` still answering.
+- dgx1's root is **915 GiB**, not 1.8 TB — it was 22 G from full. Plans sized on dgx4's
+  disk miss this.
+- dgx1's `package_cache` had silently **drifted** to an empty `~/flare-pkgs/gb10`
+  skeleton: `flare models list` said “No local model versions found”, while the live
+  rank used `~/.cache/flare`. Fix that before anything points at the NAS.
+- The started rsync (105 GiB in) was killed and the partial NAS copy deleted; Phase 2 is
+  marked **SUPERSEDED, do not run**.
+
+**The real finding: the fleet serves a superseded package from a repo that no longer
+exists.** The manifest inside `model.safetensors` names
+`…/DeepSeek-V4.1-Flash-1017-FLARE2` → **404 with a valid token** (401 unauthenticated is
+HF masking a missing repo). Current source: `…/DeepSeek-V4.1-Flash-1017-FLARE`, revision
+= **branch `gb10`**, last commit **2026-11-05** *“Publish the GB10 TP4 package with
+vocabulary-parallel sampling and an FP8 Markov head”*, 187 files / **474.7 GiB**, and it
+**requires `flare-inference ≥ 0.1.0~git20261105002735.0009b89d-1`**.
+
+Consequences recorded in the plan (`Phase 2R`, `## Runtime parameters`, §6 items 4–7):
+
+- **PR #184 rebuild dropped.** It is not in the release (marker string → `0`), and it
+  predates the package's minimum runtime → functionally unusable for the current
+  package, however fast. Prod = release build, everywhere. Installed+identical already
+  (`03c8e463`, `Installed == Candidate` on dgx1/2/3/4/8), but dgx1/2/3/8 **execute
+  deleted inodes** — only dgx4 truly runs the release; the restart window fixes that.
+- **Launch parameters taken from the package's own README**, port **stays 30000**
+  (operator). Only delta in practice: `--mtp-adaptive-verification-alpha 0.75` added to
+  both `dsv41-flash-1017-4x-flare-gb10*` recipes. `--transfer-streams 1` (RoCE poison
+  workaround) and `--served-model-name` stay. `--memory-limit` / `--kv-cache-limit`:
+  upstream says keep the `memory` default, so nothing.
+- Next (R3): `flare download … --revision gb10` into the NAS store (474.7 GiB), verify
+  with `flare models list --model-cache`, then a window to cut over and delete the
+  superseded 477 G on dgx1 + 261 G dirs on the ranks.
+
+Skill updated (`~/.agents/skills/flare`): rule 1 now checks `readlink /proc/<pid>/exe`
+for `(deleted)` — hash equality does not prove a node runs the installed build; rule 2
+reversed to “prod runs the latest release build”, with the package-gates-runtime reason;
+rule 5 adds “keep the flag set aligned with the package README, deviate only where
+documented”.

+ 94 - 11
specs/tech-architecture/flare-storage-nas.md

@@ -16,8 +16,10 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
 - [x] NAS throughput measured (was open item 6.3)
 - [x] **Stale model data purged from the nodes — 783 GiB freed**, dgx1 from 98 % full
       (22 G free) to 283 G free; serving never interrupted
-- [ ] PR #184 binary rebuilt on the Nov-5 base and staged (Phase 0.1) — needs dgx5
-      powered on, and is now **required** to load the current package at all
+- [x] Launch parameters aligned with the current `gb10` package recipe; **API port
+      stays 30000** (operator directive), `--mtp-adaptive-verification-alpha 0.75` added
+- [ ] **Every node running the release build** — installed everywhere already, but
+      dgx1/2/3/8 still execute PR-184 inodes; **PR #184 rebuild dropped** (§6 item 5)
 - [ ] **Fresh `gb10` package acquired onto the NAS store and deployed; superseded
       build deleted** (Phase 2R — replaces the "migrate the old store" plan)
 - [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
@@ -36,8 +38,8 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       verify with `flare models list --model-cache /mnt/nas/ai/flare/store`
 - [ ] **R4 (Q1)** — Phase 2R second half: fix dgx1's `package_cache` drift (it points
       at an empty `~/flare-pkgs/gb10`), confirm the launched rank cmdline (open item 6.1)
-- [ ] **R5 (Q1)** — PR #184 rebuild **on the Nov-5 base** and staging (needs dgx5 on;
-      prerequisite for both any restart and loading the new package)
+- [ ] **R5 (Q1)** — Phase 0.1 as rewritten: pre-flight that the packaged release is
+      the binary every node starts into at cutover (no PR-184 rebuild — dropped)
 - [ ] **R6 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4
 - [ ] **R7 (Q2)** — Phase 4 sparks rollout (window; ranks stop)
 - [ ] **R8 (Q2)** — Phase 5 prod4 rollout (window; ranks stop)
@@ -132,12 +134,19 @@ previous phase failed.
 
 ### Phase 0 — prerequisites (no FLARE restart yet)
 
-1. **Rebuild the runtime before any agent restart.** The installed binary is the
-   PPA build (`0.1.0~git20261105002735.0009b89d-1`), which lacks PR #184. The
-   running agents still hold the PR-184 inode; restarting without the rebuild
-   drops the io_uring Engram reader on dgx1/2/3/8. See `references/build.md` in
-   the `flare` skill, or `flare-upstream.pin` + `pr184-on-main` here. Verify:
-   `strings <binary> | grep -c "embedding read worker panicked"` → `1`.
+1. **Every node must be *running* the packaged release build before any cutover**
+   (rewritten 2026-11-11, operator directive: “the flare itself must be the latest
+   version and properly updated on all nodes”). The release **is** installed and
+   identical everywhere — `Installed == Candidate ==
+   0.1.0~git20261105002735.0009b89d-1`, `/usr/bin/flare` = `03c8e463` on
+   dgx1/2/3/4/8 — but dgx1/2/3/8 *execute deleted inodes* (the PR #184 binaries), so
+   only dgx4 actually runs the release. `readlink /proc/<pid>/exe` showing `(deleted)`
+   is the symptom; the fix is a restart onto the packaged file, never `cp`.
+   **The PR #184 rebuild is dropped**: PR #184 is not in the release (`strings -a
+   /usr/bin/flare | grep -c "embedding read worker panicked"` → **0**), and the current
+   `gb10` package cannot be loaded by anything older than `20261105002735` — so the
+   release build wins. Revisit only if/when the PR is merged upstream.
+   Keep the rollback binaries (`flare.orig-20261106`) until the cutover is verified.
 2. Confirm NAS health and space, from every node that will use it:
    ```bash
    ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
@@ -464,6 +473,44 @@ rsync -aH /mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V/deployments/ ~/.cache/fla
 flare orbit run <group> --cluster sparks
 ```
 
+## Runtime parameters (2026-11-11)
+
+Decision: **take the launch parameters from the current `gb10` package**, not from what
+the lane happens to run today — with one deliberate exception, **the API port stays
+30000** (every lane here serves on 30000; `8000` in their example is just their host
+convention).
+
+Their recipe vs. the live rank 0 command line on dgx1 (`tr '\0' ' ' <
+/proc/$(pgrep -f "[f]lare run" | head -1)/cmdline`):
+
+| flag | upstream `gb10` | live | action |
+|---|---|---|---|
+| `--max-concurrency` | `32` | `32` | keep |
+| `--mtp-draft-tokens` | `5` | `5` | keep |
+| `--mtp-adaptive-verification` | yes | yes | keep |
+| `--mtp-adaptive-verification-alpha` | **`0.75`** | **absent** | **added to both recipes** |
+| `--embedding-table-storage` | `disk` | `disk` | keep |
+| `--embedding-table-resident-scales` | yes | yes | keep |
+| `--embedding-table-cache-size` | `1GiB` | `1073741824` | equivalent (both = 1 GiB) |
+| `--transfer-streams 1` | absent | `1` | **keep ours** — RoCE workaround, `flare-roce-poison.md` |
+| `--host` | `0.0.0.0` | `0.0.0.0` | keep |
+| `--port` | `8000` | **`30000`** | **keep 30000** |
+| `--served-model-name deepseek-v4.1-flash` | absent | set | keep ours (client-facing name) |
+
+`alpha 0.75` is theirs on purpose: *“favor longer verification rounds, which raises
+decode throughput with four or more concurrent requests and coding throughput”* — i.e.
+it helps exactly the lane shape we run (c32, TP4).
+
+They also document memory/KV knobs (`--memory-limit=-12G`, `--kv-cache-limit
+memory|context|concurrency`) and recommend **leaving both at the default `memory`**:
+*“Every GiB reserved or cached away from the default reduces the KV capacity that
+Flare can plan.”* Their stated budget — ~11.4 GiB KV per GPU, ~13.1 M tokens, 17–18 GiB
+of 121 GiB left free — matches what we measure, so no change. The 1 GiB Engram row
+cache is their recommended default; they suggest 4–8 GiB only if memory is spare and
+the startup `Plan cache` line confirms room, and our own 48 GiB experiment failed with
+*“selected concurrency and verification width exceed available memory”* — so 1 GiB
+stays.
+
 ## 5. Risks and their fallbacks
 
 | risk | detection | fallback |
@@ -477,7 +524,15 @@ flare orbit run <group> --cluster sparks
 ## 6. What must be measured, not assumed
 
 1. Whether the agent's `--package-cache` path can be redirected by config or needs
-   a launcher `extra_flags` change (phase 2 note).
+   a launcher `extra_flags` change (phase 2 note). **Partly answered 2026-11-11, and it
+   is worse than assumed:** on dgx1 the CLI's configured storage is
+   `/home/kilork/flare-pkgs/gb10`, an **empty 68 K skeleton** — `flare models list`
+   says *“No local model versions found”* and `flare models inspect` cannot see the
+   live package id — while the real 477 G store is in `~/.cache/flare/packages`
+   because rank 0 is launched with `--package-cache /home/kilork/.cache/flare`. So
+   redirection *works*, but CLI and agent are already pointed at different trees on
+   one node. Unify them (launcher `local_dir` + agent flag) **before** either points
+   at the NAS, or the NAS store will inherit the same split-brain.
 2. Whether FLARE tolerates the bind mount across a real stage (phase 3 gate).
 3. ~~Real NAS read throughput~~ **measured 2026-11-11** (`dd iflag=direct`, 2 GiB,
    so no page-cache lying): the NAS is **10 GbE, not 1 GbE**.
@@ -493,3 +548,31 @@ flare orbit run <group> --cluster sparks
    small files would load ~25× slower than the bulk figure suggests — count the
    files in a rank deployment before trusting the 2–3 s number, and keep `weights/`
    local regardless.
+4. **Package revision currency must be verified against the source before any storage
+   work** (2026-11-11: the fleet served a superseded Nov-3 build whose source repo now
+   404s — see Phase 2R). To read what a local package *is*, do not trust the tree
+   layout; read the identity out of the payload (the `flare.manifest` tensor is
+   protobuf, but the strings are legible):
+   ```bash
+   strings -a <pkg>/model.safetensors | grep -aoE "https://huggingface.co/[^ ]*" | sort -u
+   ```
+   and compare with `https://huggingface.co/api/models/<repo>/commits/<revision>`.
+   The HF API masks a *missing* repo as 401 when unauthenticated, so always re-check
+   with a token before concluding a repo is gated. A valid token is
+   `export HF_TOKEN=…` in `~/.zshenv` **on the nodes** — there is none on the mac.
+5. **`flare-inference` must be the latest release and actually running on every node.**
+   Measured 2026-11-11: on dgx1/2/3/4/8 `Installed == Candidate ==
+   0.1.0~git20261105002735.0009b89d-1` and `/usr/bin/flare` is the same file
+   (`03c8e463`) everywhere — but dgx1/2/3/8 *execute* deleted inodes (the PR #184
+   binaries), so only dgx4 is truly on the release build. The package's
+   `≥ 20261105002735` requirement makes this a hard gate: **no node can load the
+   current `gb10` revision until it restarts onto the packaged binary.**
+   PR #184 is **not** in the release (`strings -a /usr/bin/flare | grep -c
+   "embedding read worker panicked"` → 0), so running the latest build means giving up
+   the patched Engram reader until it is merged upstream.
+6. **Node root disks are not uniform** — dgx1 is **915 GiB**, dgx4 is 1.8 TiB. dgx1 was
+   the one at 98 % full; a plan sized from dgx4's disk would have missed the whole
+   reason this migration exists.
+7. Package stores across dgx1–dgx3 share one id (`7cf40bec…-35842073`), and the layout
+   here is **flat**: `du -l` reports **0 hardlinks / 184 files**, so `rsync -H` was
+   never load-bearing on this fleet.