Pārlūkot izejas kodu

docs(flare-storage): the Nov-5 runtime cannot acquire the current package; v0.1.0 staged fleet-wide

Measured, not inferred: with the installed Nov-5 apt binary the gb10 acquisition
stalls at "Verify package" forever - 0 B for minutes, holding the entries and
staging locks, one TLS handshake after another to HF's eu-west-3 LFS CDN, rx_bytes
flat. The identical command with upstream's tagged v0.1.0 binary streams at once
(55-78 MiB/s). An out-of-date runtime is a functional failure here.

Channels established: NVIDIA/spark-local-inference 301-redirects to
NVIDIA/hyperflare-inference; upstream publishes release archives from version tags
(commit 94fbce5f6), tag v0.1.0 (2026-11-11T19:31Z), flare-linux-arm64.tgz
11,427,812 B sha256 7ba5ba43... -> 26,054,848 B binary. APT lags: after a clean
apt-get update, madison offers only 20261105002735 / 20261030013211 / 20261027170349.
So: GitHub tags primary, APT a floor, source build the backup (operator's call).
PR #184 still unmerged (marker 0 in v0.1.0) - dropping it stands.

Staged v0.1.0 on dgx1/2/3/4/8 at /usr/local/share/hyperflare/v0.1.0/flare; every
node re-downloaded and verified the archive sha256, and the resulting binary is
identical on all five (e3fe697e..., 26,054,848 B). /usr/bin/flare and
~/.local/lib/flare-agent/flare untouched - nothing serving was touched. Documented
that the agent path, not /usr/bin, is what the unit starts, with the temp+mv swap
and the readlink/sha256 verification.

Acquisition in flight on dgx4 into /mnt/nas/ai/flare/store. Three undocumented
requirements recorded: --tensor-parallel 4 (default is the local GPU count, so the
upstream example dies with "package exports no deployment with tensor-parallel
size 1" on a lone Spark), non-interactive publisher trust (flare download has no
--dangerously-* flags; needs --publisher-keys + --trust-store, key c8215a75... and
the fleet-uniform trust.toml 26e5ae90...), and --token-file for the LFS transport.

Also: SGLang (Ripple eval) stopped on dgx4 - 103 GiB -> 4 GiB RAM used, port 8010
closed; pkill -f sglang killed the ssh session (pattern self-match), always
self-exclude. New ## Checks runtime probe (staged/installed/running three-way) and
a note that only rank 0 serves the API, so peer curl :30000 -> 000 is normal.
Alexander Korolev 1 nedēļu atpakaļ
vecāks
revīzija
3a5ecd1750

+ 3 - 8
specs/tech-architecture/.rounds-state.flare-storage-nas.json

@@ -1,12 +1,12 @@
 {
   "objective": "/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md",
-  "round": 2,
+  "round": 3,
   "maxRounds": 12,
   "turnsBudget": 60,
   "progressEntries": 12,
   "verifyTimeout": 120,
   "minutesBudget": null,
-  "lastObjectiveHash": "35158cc0-17743",
+  "lastObjectiveHash": "755fc9ef-33963",
   "attemptsBudget": 3,
   "attemptsUsed": 1,
   "lastCheck": null,
@@ -16,11 +16,6 @@
       "subject": "Compile table + conclusion",
       "firstSeen": 1
     },
-    {
-      "id": 11,
-      "subject": "Rebuild flare with PR184 on the Nov-5 base before restart",
-      "firstSeen": 1
-    },
     {
       "id": 12,
       "subject": "Migrate flare model root to NAS in a maintenance window",
@@ -30,5 +25,5 @@
   "active": true,
   "finished": false,
   "lastError": null,
-  "updated": "2026-11-11T21:53:43.180Z"
+  "updated": "2026-11-11T22:10:11.979Z"
 }

+ 64 - 3
specs/tech-architecture/RALPH_PROGRESS.md

@@ -47,9 +47,6 @@ Watch out for:
   command reports them as `off`).
 - R4 (PR #184 rebuild) needs dgx5 powered on and must precede any agent restart.
 
-## Loop log
-- round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…
-
 ## 2026-11-11 — R2 (flare-storage-nas): purge instead of migrate; what we serve is superseded
 
 **Goal quadrant: impact × effort — delete 783 GiB of dead data and find out whether the
@@ -100,3 +97,67 @@ for `(deleted)` — hash equality does not prove a node runs the installed build
 reversed to “prod runs the latest release build”, with the package-gates-runtime reason;
 rule 5 adds “keep the flag set aligned with the package README, deviate only where
 documented”.
+
+## Loop log
+- round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…
+- round 2 · -- · 74 turns · continue → continue · (no ROUND line)
+
+## 2026-11-11 — R3 (flare-storage-nas): the runtime is the blocker, not the disk
+
+**Goal quadrant: impact × effort — acquire the current `gb10` package onto the NAS
+store (R3).** The download is running; the surprise is that it could not have been run
+with the runtime we had installed.
+
+**The Nov-5 apt build cannot acquire the current package.** With `/usr/bin/flare`
+(`0.1.0~git20261105002735.0009b89d-1`, installed == candidate on all five powered
+nodes) the acquisition sits at `Verify package`, **0 B for minutes**: it holds
+`store/locks/entries/checkpoint-expanded-scales--gb10-tp4-…lock` and
+`store/locks/staging/<pid>-0.lock` and makes one TLS handshake after another to
+`13.36.183.231` (HF's eu-west-3 LFS CDN) with `rx_bytes` flat. The **same command**
+with upstream's tagged binary streams instantly. So the operator's instinct was right:
+1105 is too old — and this is functional, not a lost optimisation.
+
+**Channels, established rather than assumed:** `github.com/NVIDIA/spark-local-inference`
+→ **301 → `NVIDIA/hyperflare-inference`** (branch `main`, pushed 2026-11-11, 801 ★);
+upstream publishes **release archives from version tags** (commit `94fbce5f6`), tag
+**`v0.1.0`** (2026-11-11T19:31Z) with `flare-linux-arm64.tgz` = 11,427,812 B,
+sha256 `7ba5ba43…`, unpacking to a 26,054,848 B `flare`. The **APT repo lags**: after a
+clean `apt-get update`, `apt-cache madison flare-inference` offers only
+`20261105002735 / 20261030013211 / 20261027170349`. Source build stays the backup path.
+PR #184 is still not merged (`embedding read worker panicked` marker → `0` in v0.1.0),
+which independently confirms dropping it.
+
+**What landed:**
+- **Acquisition in flight on dgx4** (detached `setsid nohup`, v0.1.0 binary):
+  `flare download … --revision gb10 --tensor-parallel 4 --model-cache
+  /mnt/nas/ai/flare/store --publisher-keys … --token-file …` → 55–78 MiB/s into the NAS
+  store, 14 GiB by 22:10 (160 s). Upstream's example needs three things it does not
+  document: `--tensor-parallel 4` (default is the local GPU count →
+  `package exports no deployment with tensor-parallel size 1`), non-interactive
+  publisher trust (`flare download` has no `--dangerously-*` flags: it wants
+  `--publisher-keys` + `--trust-store`; copied dgx1's key `c8215a75…` and the
+  fleet-uniform `trust.toml` `26e5ae90…`), and `--token-file` for the LFS transport.
+- **v0.1.0 staged identically on dgx1/2/3/4/8** at
+  `/usr/local/share/hyperflare/v0.1.0/flare` — each node re-downloaded and checked the
+  archive sha256; resulting binary identical on all five (`e3fe697e…`, 26,054,848 B).
+  `/usr/bin/flare` and `~/.local/lib/flare-agent/flare` untouched, so nothing serving
+  was touched. Documented the actual swap for the window (`/usr/bin/flare` is *not* what
+  runs — the agent executes `~/.local/lib/flare-agent/flare`).
+- **SGLang stopped on dgx4** (the Ripple eval server): 103 GiB → 4 GiB RAM used, 116 GiB
+  available, port 8010 closed, no leftovers. Weights kept on disk (54 G + 7.6 G).
+  Note: `pkill -f sglang` killed my own ssh session (the pattern matches the remote
+  command line) — always self-exclude.
+- Added a `## Checks` runtime probe: `staged=e3fe697e installed=03c8e463
+  running=…flare-agent/flare (deleted)` on all four serving nodes, `running=` empty on
+  dgx4 (no rank) — exactly the three-way state the milestone needs to close.
+- Recorded that **only rank 0 serves the API** (peer `curl :30000` → `000` is normal),
+  so `dgx2 api=000` is not an outage.
+
+Objective file: Phase 0.1 corrected with the channel table and the measured hang;
+Phase 2R rewritten with the command that actually works plus the three undocumented
+requirements; milestones/round plan updated; `## Checks` gained the runtime probe.
+Skill: `SKILL.md` rule 2 now ranks channels (GitHub tags → APT (lagging floor) →
+source); `references/build.md` gained “Distribution channels (2026-11-11)” with the
+staging recipe; `references/troubleshooting.md` gained the acquisition section (verify
+hang, `--tensor-parallel 4`, publisher trust, token file, pkill self-match, rank-0-only
+API).

+ 132 - 9
specs/tech-architecture/flare-storage-nas.md

@@ -18,8 +18,11 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       (22 G free) to 283 G free; serving never interrupted
 - [x] Launch parameters aligned with the current `gb10` package recipe; **API port
       stays 30000** (operator directive), `--mtp-adaptive-verification-alpha 0.75` added
-- [ ] **Every node running the release build** — installed everywhere already, but
-      dgx1/2/3/8 still execute PR-184 inodes; **PR #184 rebuild dropped** (§6 item 5)
+- [ ] **Every node on the v0.1.0 runtime** — **staged identically on dgx1/2/3/4/8 at
+      `/usr/local/share/hyperflare/v0.1.0/flare`** (archive `7ba5ba43…`, binary
+      `e3fe697e…` 26,054,848 B, same sha256 on all five), nothing in `/usr/bin` or the
+      agent path touched yet. Remaining: dgx5–7 (off) + the temp+`mv` swap and one
+      restart window; **PR #184 rebuild dropped** (§6 item 5)
 - [ ] **Fresh `gb10` package acquired onto the NAS store and deployed; superseded
       build deleted** (Phase 2R — replaces the "migrate the old store" plan)
 - [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
@@ -35,7 +38,11 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
       local package is a superseded build from a deleted repo (Phase 2R)
 - [ ] **R3 (Q1)** — Phase 2R first half: download the current `gb10` package
       (474.7 GiB) into the NAS store with `flare download … --revision gb10`,
-      verify with `flare models list --model-cache /mnt/nas/ai/flare/store`
+      verify with `flare models list --model-cache /mnt/nas/ai/flare/store`.
+      **In flight 2026-11-11 22:07 on dgx4**; the Nov-5 apt binary could not do it at
+      all (§ Phase 0.1), the tagged v0.1.0 binary does it at 55–78 MiB/s ⇒ ETA ≈ 2 h.
+      Also done in this round: v0.1.0 **staged** on dgx1/2/3/4/8, and SGLang (Ripple
+      eval server) **stopped on dgx4** — 103 GiB → 4 GiB RAM used
 - [ ] **R4 (Q1)** — Phase 2R second half: fix dgx1's `package_cache` drift (it points
       at an empty `~/flare-pkgs/gb10`), confirm the launched rank cmdline (open item 6.1)
 - [ ] **R5 (Q1)** — Phase 0.1 as rewritten: pre-flight that the packaged release is
@@ -47,7 +54,26 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
 
 ## Checks
 
-Run from the controller; every node must report `fstab=1 live=1`:
+**Only rank 0 serves the API.** `dgx1:30000/v1/models` answering is the health probe;
+`curl` on dgx2/dgx3/dgx8 returns `000` there by design (peers have no listening port) —
+do not read that as an outage. Peer health is `pgrep -fc "[f]lare run"` = 1.
+
+The runtime check a round should run (staged-vs-running, never inferred from
+`flare --version`, which prints `0.1.0` for every build):
+
+```bash
+for h in dgx1 dgx2 dgx3 dgx4 dgx8; do printf "%-6s " $h
+  ssh -o BatchMode=yes $h.lan 'printf "staged=%s installed=%s running=%s\n" \
+    "$(sha256sum /usr/local/share/hyperflare/v0.1.0/flare 2>/dev/null | cut -c1-8)" \
+    "$(sha256sum /usr/bin/flare | cut -c1-8)" \
+    "$(readlink /proc/$(pgrep -f "[f]lare run" | head -1)/exe 2>/dev/null)"' 2>/dev/null || echo off
+done
+```
+
+Expected today: `staged=e3fe697e installed=03c8e463 running=…flare-agent/flare (deleted)`.
+All three equal to `e3fe697e` (and no `(deleted)`) is the gate for this milestone.
+
+For the NAS mount, run from the controller; every node must report `fstab=1 live=1`:
 
 ```bash
 for h in dgx1 dgx2 dgx3 dgx4 dgx5 dgx6 dgx7 dgx8; do printf "%-6s " $h
@@ -147,6 +173,59 @@ previous phase failed.
    `gb10` package cannot be loaded by anything older than `20261105002735` — so the
    release build wins. Revisit only if/when the PR is merged upstream.
    Keep the rollback binaries (`flare.orig-20261106`) until the cutover is verified.
+
+   **Corrected 2026-11-11 (R3): “the latest apt build” is no longer the latest runtime,
+   and it cannot acquire the current package.**
+   - The project was renamed again: `NVIDIA/spark-local-inference` → **301 →
+     `NVIDIA/hyperflare-inference`** (default branch `main`, pushed 2026-11-11, 801 ★).
+     (`github.com/hyperflare/hyperflare` is an unrelated 2017 repo — do not use it.)
+   - Upstream now ships **first-class release archives from version tags** (commit
+     `94fbce5f6` “Publish release archives from version tags”). Tag **`v0.1.0`**,
+     published **2026-11-11T19:31Z**, assets: `flare-linux-arm64.tgz` **11,427,812 B**,
+     `sha256 7ba5ba4387cfe56a2e8205d711625addf26633219ec670ca33c23fd046d22d84`
+     (unpacks to `flare/flare`, 26,054,848 B, plus LICENSE/NOTICE/THIRD_PARTY_NOTICES).
+     Contents of v0.1.0 include PR #29 *Multi-host disk prefix caching*, PR #35/#36
+     *requested logprob token IDs / System One scoring*, ARM SHA-256 acceleration.
+     PR #184 is still **not** merged (marker string → `0` in the v0.1.0 binary too).
+   - The APT repo (`ppa.local-inference-lab.ai`, still the documented channel in the
+     `gb10` README, and its install stanza is unchanged: `lil-flare-archive.asc` +
+     `flare.sources` + `apt install flare-inference`) **lags**: after a clean
+     `apt-get update`, `apt-cache madison flare-inference` offers only
+     `20261105002735 / 20261030013211 / 20261027170349`. There is no newer apt build.
+   - **Measured consequence, not speculation:** with the installed Nov-5 binary the
+     acquisition of the current `gb10` revision **never gets past `Verify package`** —
+     0 B for >120 s, holding `locks/entries/checkpoint-expanded-scales--gb10-tp4-…lock`
+     and `locks/staging/<pid>-0.lock`, making one TLS handshake after another to
+     `13.36.183.231` (HF's eu-west-3 LFS CDN) with no payload transfer. The **same
+     command with the v0.1.0 binary streams immediately** (`Acquire core`, 55–58 MiB/s,
+     5 GiB in 90 s). So: **runtime channel = GitHub release archives (primary), source
+     build (backup); apt is only a floor.**
+   - `v0.1.0` is not yet packaged by us, so until the cutover the fleet still *runs*
+     Nov-5 while *acquiring* with the tagged release binary. Two consequences: install
+     the chosen build on **every** node in the window (one identical binary, temp+`mv`),
+     and record the sha256 of whatever we install.
+
+   **Staged 2026-11-11 (R3), zero disruption:** the verified archive is unpacked on
+   **dgx1, dgx2, dgx3, dgx4, dgx8** to `/usr/local/share/hyperflare/v0.1.0/`
+   (`flare`, `LICENSE`, `NOTICE`, `THIRD_PARTY_NOTICES.md`, `archive.sha256`,
+   `binary.sha256`). Every node independently re-downloaded and checked
+   `archive sha256 7ba5ba43…`; the resulting binary is `26,054,848 B`,
+   `sha256 e3fe697eab1f97ff…` — **identical on all five**, and each runs
+   `flare --version` → `flare 0.1.0`. `/usr/bin/flare` and
+   `~/.local/lib/flare-agent/flare` were **not touched**, so nothing serving can be
+   affected. dgx5–7 are off and still need this.
+
+   The swap in the window is deliberately boring, and note that `/usr/bin/flare` is
+   **not** what runs — the agent executes `~/.local/lib/flare-agent/flare`:
+   ```bash
+   # per node, in the window (rename, never cp — see the SIGBUS rule)
+   sudo install -m 0755 /usr/local/share/hyperflare/v0.1.0/flare /usr/bin/flare.tmp-v010
+   sudo mv /usr/bin/flare.tmp-v010 /usr/bin/flare
+   # the agent's own copy is what the unit starts; refresh it the same way, then
+   # restart the ranks together (one group, one window). Verify afterwards:
+   readlink /proc/$(pgrep -f "[f]lare run" | head -1)/exe   # must NOT say (deleted)
+   sha256sum /usr/bin/flare ~/.local/lib/flare-agent/flare  # both must be e3fe697e…
+   ```
 2. Confirm NAS health and space, from every node that will use it:
    ```bash
    ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
@@ -344,14 +423,58 @@ not old caches). The `hf` token, when one is needed, is `export HF_TOKEN=…` in
 `~/.zshenv` **on the nodes** — there is no token on the mac:
 
 ```bash
-package_path=$(flare download \
-  local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE --revision gb10)
-# to land it on the NAS store rather than the local cache, point the CLI at it:
-flare download local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE --revision gb10 \
-  --model-cache /mnt/nas/ai/flare/store
+# Verified working form (run on dgx4, the idle node, with the NAS mounted):
+~/hyperflare/flare/flare download local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE \
+  --revision gb10 \
+  --tensor-parallel 4 \
+  --model-cache /mnt/nas/ai/flare/store \
+  --publisher-keys ~/.cache/flare/publishers/937c0143f6856d3ee8ad164f62e3d804bd825d7a8750d0accb9a5849f703a8c1.json \
+  --token-file ~/.config/hf/token
 flare models list --model-cache /mnt/nas/ai/flare/store     # must list the new version
 ```
 
+Three things the upstream example does **not** tell you (all hit in R3):
+
+1. **`--tensor-parallel 4` is mandatory here.** Their example works because they run it
+   on the controller of a four-Spark group; on a lone Spark the default is the local
+   GPU count (1), and the acquisition dies with
+   `Error: "package exports no deployment with tensor-parallel size 1"` after fetching
+   the 35 MiB envelope.
+2. **Publisher trust must be non-interactive.** `flare download` has *no*
+   `--dangerously-approve-key` / `--dangerously-accept-publisher` flags (those exist on
+   `orbit run` only) — it takes `--publisher-keys <json>` and `--trust-store <path>`.
+   The two files to copy from a node that already accepted the key:
+   `~/.config/flare/trust.toml` (**fleet-uniform**: dgx1 and dgx2 have the same file,
+   270 B, `sha256 26e5ae90568572a28cbea9fb4b274ddca4127f4edf00b2f23086a2079db78464`,
+   containing the `937c0143…` fingerprint) and
+   `~/.cache/flare/publishers/937c0143….json` (219 B,
+   `sha256 c8215a75db94b6dc4ee2992789acb675527caf2806380b80b3b98d1c79698a38`, from dgx1
+   — dgx2/dgx4's `publishers/` directories are empty). Copy them with `base64 -w0` and
+   **compare sha256 on both ends**; do not overwrite a trust store without checking it
+   first (dgx4 had its own copy of the same file).
+3. **A token file helps even for a public repo** — `--token-file ~/.config/hf/token`
+   (mode 600, built from `export HF_TOKEN=…` in `~/.zshenv` on the node) keeps the LFS
+   transport off anonymous rate limits over 475 GiB. `--model-cache` is a *flag*; it
+   does not touch `[paths]` in config, so it causes no drift on the node.
+
+Everything lands under `/mnt/nas/ai/flare/store/` as
+`transport/hub/v1/{blobs,repos}/…` plus `staging/`, `locks/`, `refs/`, `models/` —
+the `transport/` default comes from `--cache-dir` (“parent directory for the shared
+Hugging Face download transport cache [default: the model storage directory]”), so all
+475 GiB is written once, on the NAS, and renames inside it are cheap.
+
+Progress of the acquisition started 2026-11-11 22:07 on dgx4 (detached, survives the
+session — `setsid nohup`):
+
+```bash
+tail -c 400 ~/flare-download-gb10.log          # symlink to the newest log
+du -sh /mnt/nas/ai/flare/store                 # bytes landed on the NAS
+pgrep -af "[f]lare download"                   # still alive?
+```
+
+Observed rate **55–58 MiB/s** (WAN-bound, not NAS-bound) ⇒ 474.7 GiB ≈ **2.5 h**.
+Measured inbound on `enP7s7`: 1482 MiB / 20 s ≈ 74 MiB/s.
+
 The upstream launch recipe (`SPARK-DS4-GUIDE.md` / the `gb10` README) is worth
 aligning our launcher flags with — note it differs from what prod4 runs today:
 

+ 1 - 0
specs/tech-architecture/rounds-history.flare-storage-nas.jsonl

@@ -1 +1,2 @@
 {"timestamp":"2026-11-11T21:49:43.957Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":1,"goal":"Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every powered node (quadrant Q1)","quadrant":null,"turns":27,"marker":"continue","decision":"continue","reason":null,"check":null}
+{"timestamp":"2026-11-11T22:03:33.500Z","objective":"/Volumes/main/projects/git.kilork.org/sm121/launcher/specs/tech-architecture/flare-storage-nas.md","round":2,"goal":null,"quadrant":null,"turns":74,"marker":"continue","decision":"continue","reason":null,"check":null}