|
@@ -18,8 +18,11 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
(22 G free) to 283 G free; serving never interrupted
|
|
(22 G free) to 283 G free; serving never interrupted
|
|
|
- [x] Launch parameters aligned with the current `gb10` package recipe; **API port
|
|
- [x] Launch parameters aligned with the current `gb10` package recipe; **API port
|
|
|
stays 30000** (operator directive), `--mtp-adaptive-verification-alpha 0.75` added
|
|
stays 30000** (operator directive), `--mtp-adaptive-verification-alpha 0.75` added
|
|
|
-- [ ] **Every node running the release build** — installed everywhere already, but
|
|
|
|
|
- dgx1/2/3/8 still execute PR-184 inodes; **PR #184 rebuild dropped** (§6 item 5)
|
|
|
|
|
|
|
+- [ ] **Every node on the v0.1.0 runtime** — **staged identically on dgx1/2/3/4/8 at
|
|
|
|
|
+ `/usr/local/share/hyperflare/v0.1.0/flare`** (archive `7ba5ba43…`, binary
|
|
|
|
|
+ `e3fe697e…` 26,054,848 B, same sha256 on all five), nothing in `/usr/bin` or the
|
|
|
|
|
+ agent path touched yet. Remaining: dgx5–7 (off) + the temp+`mv` swap and one
|
|
|
|
|
+ restart window; **PR #184 rebuild dropped** (§6 item 5)
|
|
|
- [ ] **Fresh `gb10` package acquired onto the NAS store and deployed; superseded
|
|
- [ ] **Fresh `gb10` package acquired onto the NAS store and deployed; superseded
|
|
|
build deleted** (Phase 2R — replaces the "migrate the old store" plan)
|
|
build deleted** (Phase 2R — replaces the "migrate the old store" plan)
|
|
|
- [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
|
|
- [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
|
|
@@ -35,7 +38,11 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
local package is a superseded build from a deleted repo (Phase 2R)
|
|
local package is a superseded build from a deleted repo (Phase 2R)
|
|
|
- [ ] **R3 (Q1)** — Phase 2R first half: download the current `gb10` package
|
|
- [ ] **R3 (Q1)** — Phase 2R first half: download the current `gb10` package
|
|
|
(474.7 GiB) into the NAS store with `flare download … --revision gb10`,
|
|
(474.7 GiB) into the NAS store with `flare download … --revision gb10`,
|
|
|
- verify with `flare models list --model-cache /mnt/nas/ai/flare/store`
|
|
|
|
|
|
|
+ verify with `flare models list --model-cache /mnt/nas/ai/flare/store`.
|
|
|
|
|
+ **In flight 2026-11-11 22:07 on dgx4**; the Nov-5 apt binary could not do it at
|
|
|
|
|
+ all (§ Phase 0.1), the tagged v0.1.0 binary does it at 55–78 MiB/s ⇒ ETA ≈ 2 h.
|
|
|
|
|
+ Also done in this round: v0.1.0 **staged** on dgx1/2/3/4/8, and SGLang (Ripple
|
|
|
|
|
+ eval server) **stopped on dgx4** — 103 GiB → 4 GiB RAM used
|
|
|
- [ ] **R4 (Q1)** — Phase 2R second half: fix dgx1's `package_cache` drift (it points
|
|
- [ ] **R4 (Q1)** — Phase 2R second half: fix dgx1's `package_cache` drift (it points
|
|
|
at an empty `~/flare-pkgs/gb10`), confirm the launched rank cmdline (open item 6.1)
|
|
at an empty `~/flare-pkgs/gb10`), confirm the launched rank cmdline (open item 6.1)
|
|
|
- [ ] **R5 (Q1)** — Phase 0.1 as rewritten: pre-flight that the packaged release is
|
|
- [ ] **R5 (Q1)** — Phase 0.1 as rewritten: pre-flight that the packaged release is
|
|
@@ -47,7 +54,26 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
|
|
|
|
|
## Checks
|
|
## Checks
|
|
|
|
|
|
|
|
-Run from the controller; every node must report `fstab=1 live=1`:
|
|
|
|
|
|
|
+**Only rank 0 serves the API.** `dgx1:30000/v1/models` answering is the health probe;
|
|
|
|
|
+`curl` on dgx2/dgx3/dgx8 returns `000` there by design (peers have no listening port) —
|
|
|
|
|
+do not read that as an outage. Peer health is `pgrep -fc "[f]lare run"` = 1.
|
|
|
|
|
+
|
|
|
|
|
+The runtime check a round should run (staged-vs-running, never inferred from
|
|
|
|
|
+`flare --version`, which prints `0.1.0` for every build):
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+for h in dgx1 dgx2 dgx3 dgx4 dgx8; do printf "%-6s " $h
|
|
|
|
|
+ ssh -o BatchMode=yes $h.lan 'printf "staged=%s installed=%s running=%s\n" \
|
|
|
|
|
+ "$(sha256sum /usr/local/share/hyperflare/v0.1.0/flare 2>/dev/null | cut -c1-8)" \
|
|
|
|
|
+ "$(sha256sum /usr/bin/flare | cut -c1-8)" \
|
|
|
|
|
+ "$(readlink /proc/$(pgrep -f "[f]lare run" | head -1)/exe 2>/dev/null)"' 2>/dev/null || echo off
|
|
|
|
|
+done
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Expected today: `staged=e3fe697e installed=03c8e463 running=…flare-agent/flare (deleted)`.
|
|
|
|
|
+All three equal to `e3fe697e` (and no `(deleted)`) is the gate for this milestone.
|
|
|
|
|
+
|
|
|
|
|
+For the NAS mount, run from the controller; every node must report `fstab=1 live=1`:
|
|
|
|
|
|
|
|
```bash
|
|
```bash
|
|
|
for h in dgx1 dgx2 dgx3 dgx4 dgx5 dgx6 dgx7 dgx8; do printf "%-6s " $h
|
|
for h in dgx1 dgx2 dgx3 dgx4 dgx5 dgx6 dgx7 dgx8; do printf "%-6s " $h
|
|
@@ -147,6 +173,59 @@ previous phase failed.
|
|
|
`gb10` package cannot be loaded by anything older than `20261105002735` — so the
|
|
`gb10` package cannot be loaded by anything older than `20261105002735` — so the
|
|
|
release build wins. Revisit only if/when the PR is merged upstream.
|
|
release build wins. Revisit only if/when the PR is merged upstream.
|
|
|
Keep the rollback binaries (`flare.orig-20261106`) until the cutover is verified.
|
|
Keep the rollback binaries (`flare.orig-20261106`) until the cutover is verified.
|
|
|
|
|
+
|
|
|
|
|
+ **Corrected 2026-11-11 (R3): “the latest apt build” is no longer the latest runtime,
|
|
|
|
|
+ and it cannot acquire the current package.**
|
|
|
|
|
+ - The project was renamed again: `NVIDIA/spark-local-inference` → **301 →
|
|
|
|
|
+ `NVIDIA/hyperflare-inference`** (default branch `main`, pushed 2026-11-11, 801 ★).
|
|
|
|
|
+ (`github.com/hyperflare/hyperflare` is an unrelated 2017 repo — do not use it.)
|
|
|
|
|
+ - Upstream now ships **first-class release archives from version tags** (commit
|
|
|
|
|
+ `94fbce5f6` “Publish release archives from version tags”). Tag **`v0.1.0`**,
|
|
|
|
|
+ published **2026-11-11T19:31Z**, assets: `flare-linux-arm64.tgz` **11,427,812 B**,
|
|
|
|
|
+ `sha256 7ba5ba4387cfe56a2e8205d711625addf26633219ec670ca33c23fd046d22d84`
|
|
|
|
|
+ (unpacks to `flare/flare`, 26,054,848 B, plus LICENSE/NOTICE/THIRD_PARTY_NOTICES).
|
|
|
|
|
+ Contents of v0.1.0 include PR #29 *Multi-host disk prefix caching*, PR #35/#36
|
|
|
|
|
+ *requested logprob token IDs / System One scoring*, ARM SHA-256 acceleration.
|
|
|
|
|
+ PR #184 is still **not** merged (marker string → `0` in the v0.1.0 binary too).
|
|
|
|
|
+ - The APT repo (`ppa.local-inference-lab.ai`, still the documented channel in the
|
|
|
|
|
+ `gb10` README, and its install stanza is unchanged: `lil-flare-archive.asc` +
|
|
|
|
|
+ `flare.sources` + `apt install flare-inference`) **lags**: after a clean
|
|
|
|
|
+ `apt-get update`, `apt-cache madison flare-inference` offers only
|
|
|
|
|
+ `20261105002735 / 20261030013211 / 20261027170349`. There is no newer apt build.
|
|
|
|
|
+ - **Measured consequence, not speculation:** with the installed Nov-5 binary the
|
|
|
|
|
+ acquisition of the current `gb10` revision **never gets past `Verify package`** —
|
|
|
|
|
+ 0 B for >120 s, holding `locks/entries/checkpoint-expanded-scales--gb10-tp4-…lock`
|
|
|
|
|
+ and `locks/staging/<pid>-0.lock`, making one TLS handshake after another to
|
|
|
|
|
+ `13.36.183.231` (HF's eu-west-3 LFS CDN) with no payload transfer. The **same
|
|
|
|
|
+ command with the v0.1.0 binary streams immediately** (`Acquire core`, 55–58 MiB/s,
|
|
|
|
|
+ 5 GiB in 90 s). So: **runtime channel = GitHub release archives (primary), source
|
|
|
|
|
+ build (backup); apt is only a floor.**
|
|
|
|
|
+ - `v0.1.0` is not yet packaged by us, so until the cutover the fleet still *runs*
|
|
|
|
|
+ Nov-5 while *acquiring* with the tagged release binary. Two consequences: install
|
|
|
|
|
+ the chosen build on **every** node in the window (one identical binary, temp+`mv`),
|
|
|
|
|
+ and record the sha256 of whatever we install.
|
|
|
|
|
+
|
|
|
|
|
+ **Staged 2026-11-11 (R3), zero disruption:** the verified archive is unpacked on
|
|
|
|
|
+ **dgx1, dgx2, dgx3, dgx4, dgx8** to `/usr/local/share/hyperflare/v0.1.0/`
|
|
|
|
|
+ (`flare`, `LICENSE`, `NOTICE`, `THIRD_PARTY_NOTICES.md`, `archive.sha256`,
|
|
|
|
|
+ `binary.sha256`). Every node independently re-downloaded and checked
|
|
|
|
|
+ `archive sha256 7ba5ba43…`; the resulting binary is `26,054,848 B`,
|
|
|
|
|
+ `sha256 e3fe697eab1f97ff…` — **identical on all five**, and each runs
|
|
|
|
|
+ `flare --version` → `flare 0.1.0`. `/usr/bin/flare` and
|
|
|
|
|
+ `~/.local/lib/flare-agent/flare` were **not touched**, so nothing serving can be
|
|
|
|
|
+ affected. dgx5–7 are off and still need this.
|
|
|
|
|
+
|
|
|
|
|
+ The swap in the window is deliberately boring, and note that `/usr/bin/flare` is
|
|
|
|
|
+ **not** what runs — the agent executes `~/.local/lib/flare-agent/flare`:
|
|
|
|
|
+ ```bash
|
|
|
|
|
+ # per node, in the window (rename, never cp — see the SIGBUS rule)
|
|
|
|
|
+ sudo install -m 0755 /usr/local/share/hyperflare/v0.1.0/flare /usr/bin/flare.tmp-v010
|
|
|
|
|
+ sudo mv /usr/bin/flare.tmp-v010 /usr/bin/flare
|
|
|
|
|
+ # the agent's own copy is what the unit starts; refresh it the same way, then
|
|
|
|
|
+ # restart the ranks together (one group, one window). Verify afterwards:
|
|
|
|
|
+ readlink /proc/$(pgrep -f "[f]lare run" | head -1)/exe # must NOT say (deleted)
|
|
|
|
|
+ sha256sum /usr/bin/flare ~/.local/lib/flare-agent/flare # both must be e3fe697e…
|
|
|
|
|
+ ```
|
|
|
2. Confirm NAS health and space, from every node that will use it:
|
|
2. Confirm NAS health and space, from every node that will use it:
|
|
|
```bash
|
|
```bash
|
|
|
ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
|
|
ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
|
|
@@ -344,14 +423,58 @@ not old caches). The `hf` token, when one is needed, is `export HF_TOKEN=…` in
|
|
|
`~/.zshenv` **on the nodes** — there is no token on the mac:
|
|
`~/.zshenv` **on the nodes** — there is no token on the mac:
|
|
|
|
|
|
|
|
```bash
|
|
```bash
|
|
|
-package_path=$(flare download \
|
|
|
|
|
- local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE --revision gb10)
|
|
|
|
|
-# to land it on the NAS store rather than the local cache, point the CLI at it:
|
|
|
|
|
-flare download local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE --revision gb10 \
|
|
|
|
|
- --model-cache /mnt/nas/ai/flare/store
|
|
|
|
|
|
|
+# Verified working form (run on dgx4, the idle node, with the NAS mounted):
|
|
|
|
|
+~/hyperflare/flare/flare download local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE \
|
|
|
|
|
+ --revision gb10 \
|
|
|
|
|
+ --tensor-parallel 4 \
|
|
|
|
|
+ --model-cache /mnt/nas/ai/flare/store \
|
|
|
|
|
+ --publisher-keys ~/.cache/flare/publishers/937c0143f6856d3ee8ad164f62e3d804bd825d7a8750d0accb9a5849f703a8c1.json \
|
|
|
|
|
+ --token-file ~/.config/hf/token
|
|
|
flare models list --model-cache /mnt/nas/ai/flare/store # must list the new version
|
|
flare models list --model-cache /mnt/nas/ai/flare/store # must list the new version
|
|
|
```
|
|
```
|
|
|
|
|
|
|
|
|
|
+Three things the upstream example does **not** tell you (all hit in R3):
|
|
|
|
|
+
|
|
|
|
|
+1. **`--tensor-parallel 4` is mandatory here.** Their example works because they run it
|
|
|
|
|
+ on the controller of a four-Spark group; on a lone Spark the default is the local
|
|
|
|
|
+ GPU count (1), and the acquisition dies with
|
|
|
|
|
+ `Error: "package exports no deployment with tensor-parallel size 1"` after fetching
|
|
|
|
|
+ the 35 MiB envelope.
|
|
|
|
|
+2. **Publisher trust must be non-interactive.** `flare download` has *no*
|
|
|
|
|
+ `--dangerously-approve-key` / `--dangerously-accept-publisher` flags (those exist on
|
|
|
|
|
+ `orbit run` only) — it takes `--publisher-keys <json>` and `--trust-store <path>`.
|
|
|
|
|
+ The two files to copy from a node that already accepted the key:
|
|
|
|
|
+ `~/.config/flare/trust.toml` (**fleet-uniform**: dgx1 and dgx2 have the same file,
|
|
|
|
|
+ 270 B, `sha256 26e5ae90568572a28cbea9fb4b274ddca4127f4edf00b2f23086a2079db78464`,
|
|
|
|
|
+ containing the `937c0143…` fingerprint) and
|
|
|
|
|
+ `~/.cache/flare/publishers/937c0143….json` (219 B,
|
|
|
|
|
+ `sha256 c8215a75db94b6dc4ee2992789acb675527caf2806380b80b3b98d1c79698a38`, from dgx1
|
|
|
|
|
+ — dgx2/dgx4's `publishers/` directories are empty). Copy them with `base64 -w0` and
|
|
|
|
|
+ **compare sha256 on both ends**; do not overwrite a trust store without checking it
|
|
|
|
|
+ first (dgx4 had its own copy of the same file).
|
|
|
|
|
+3. **A token file helps even for a public repo** — `--token-file ~/.config/hf/token`
|
|
|
|
|
+ (mode 600, built from `export HF_TOKEN=…` in `~/.zshenv` on the node) keeps the LFS
|
|
|
|
|
+ transport off anonymous rate limits over 475 GiB. `--model-cache` is a *flag*; it
|
|
|
|
|
+ does not touch `[paths]` in config, so it causes no drift on the node.
|
|
|
|
|
+
|
|
|
|
|
+Everything lands under `/mnt/nas/ai/flare/store/` as
|
|
|
|
|
+`transport/hub/v1/{blobs,repos}/…` plus `staging/`, `locks/`, `refs/`, `models/` —
|
|
|
|
|
+the `transport/` default comes from `--cache-dir` (“parent directory for the shared
|
|
|
|
|
+Hugging Face download transport cache [default: the model storage directory]”), so all
|
|
|
|
|
+475 GiB is written once, on the NAS, and renames inside it are cheap.
|
|
|
|
|
+
|
|
|
|
|
+Progress of the acquisition started 2026-11-11 22:07 on dgx4 (detached, survives the
|
|
|
|
|
+session — `setsid nohup`):
|
|
|
|
|
+
|
|
|
|
|
+```bash
|
|
|
|
|
+tail -c 400 ~/flare-download-gb10.log # symlink to the newest log
|
|
|
|
|
+du -sh /mnt/nas/ai/flare/store # bytes landed on the NAS
|
|
|
|
|
+pgrep -af "[f]lare download" # still alive?
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Observed rate **55–58 MiB/s** (WAN-bound, not NAS-bound) ⇒ 474.7 GiB ≈ **2.5 h**.
|
|
|
|
|
+Measured inbound on `enP7s7`: 1482 MiB / 20 s ≈ 74 MiB/s.
|
|
|
|
|
+
|
|
|
The upstream launch recipe (`SPARK-DS4-GUIDE.md` / the `gb10` README) is worth
|
|
The upstream launch recipe (`SPARK-DS4-GUIDE.md` / the `gb10` README) is worth
|
|
|
aligning our launcher flags with — note it differs from what prod4 runs today:
|
|
aligning our launcher flags with — note it differs from what prod4 runs today:
|
|
|
|
|
|