|
@@ -16,8 +16,10 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
- [x] NAS throughput measured (was open item 6.3)
|
|
- [x] NAS throughput measured (was open item 6.3)
|
|
|
- [x] **Stale model data purged from the nodes — 783 GiB freed**, dgx1 from 98 % full
|
|
- [x] **Stale model data purged from the nodes — 783 GiB freed**, dgx1 from 98 % full
|
|
|
(22 G free) to 283 G free; serving never interrupted
|
|
(22 G free) to 283 G free; serving never interrupted
|
|
|
-- [ ] PR #184 binary rebuilt on the Nov-5 base and staged (Phase 0.1) — needs dgx5
|
|
|
|
|
- powered on, and is now **required** to load the current package at all
|
|
|
|
|
|
|
+- [x] Launch parameters aligned with the current `gb10` package recipe; **API port
|
|
|
|
|
+ stays 30000** (operator directive), `--mtp-adaptive-verification-alpha 0.75` added
|
|
|
|
|
+- [ ] **Every node running the release build** — installed everywhere already, but
|
|
|
|
|
+ dgx1/2/3/8 still execute PR-184 inodes; **PR #184 rebuild dropped** (§6 item 5)
|
|
|
- [ ] **Fresh `gb10` package acquired onto the NAS store and deployed; superseded
|
|
- [ ] **Fresh `gb10` package acquired onto the NAS store and deployed; superseded
|
|
|
build deleted** (Phase 2R — replaces the "migrate the old store" plan)
|
|
build deleted** (Phase 2R — replaces the "migrate the old store" plan)
|
|
|
- [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
|
|
- [ ] Bind-mount layout validated on an idle node (Phase 3 gate)
|
|
@@ -36,8 +38,8 @@ beyond it. Every command is copy-pasteable and every phase has a gate.
|
|
|
verify with `flare models list --model-cache /mnt/nas/ai/flare/store`
|
|
verify with `flare models list --model-cache /mnt/nas/ai/flare/store`
|
|
|
- [ ] **R4 (Q1)** — Phase 2R second half: fix dgx1's `package_cache` drift (it points
|
|
- [ ] **R4 (Q1)** — Phase 2R second half: fix dgx1's `package_cache` drift (it points
|
|
|
at an empty `~/flare-pkgs/gb10`), confirm the launched rank cmdline (open item 6.1)
|
|
at an empty `~/flare-pkgs/gb10`), confirm the launched rank cmdline (open item 6.1)
|
|
|
-- [ ] **R5 (Q1)** — PR #184 rebuild **on the Nov-5 base** and staging (needs dgx5 on;
|
|
|
|
|
- prerequisite for both any restart and loading the new package)
|
|
|
|
|
|
|
+- [ ] **R5 (Q1)** — Phase 0.1 as rewritten: pre-flight that the packaged release is
|
|
|
|
|
+ the binary every node starts into at cutover (no PR-184 rebuild — dropped)
|
|
|
- [ ] **R6 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4
|
|
- [ ] **R6 (Q2)** — Phase 3 bind-mount validation gate on idle dgx4
|
|
|
- [ ] **R7 (Q2)** — Phase 4 sparks rollout (window; ranks stop)
|
|
- [ ] **R7 (Q2)** — Phase 4 sparks rollout (window; ranks stop)
|
|
|
- [ ] **R8 (Q2)** — Phase 5 prod4 rollout (window; ranks stop)
|
|
- [ ] **R8 (Q2)** — Phase 5 prod4 rollout (window; ranks stop)
|
|
@@ -132,12 +134,19 @@ previous phase failed.
|
|
|
|
|
|
|
|
### Phase 0 — prerequisites (no FLARE restart yet)
|
|
### Phase 0 — prerequisites (no FLARE restart yet)
|
|
|
|
|
|
|
|
-1. **Rebuild the runtime before any agent restart.** The installed binary is the
|
|
|
|
|
- PPA build (`0.1.0~git20261105002735.0009b89d-1`), which lacks PR #184. The
|
|
|
|
|
- running agents still hold the PR-184 inode; restarting without the rebuild
|
|
|
|
|
- drops the io_uring Engram reader on dgx1/2/3/8. See `references/build.md` in
|
|
|
|
|
- the `flare` skill, or `flare-upstream.pin` + `pr184-on-main` here. Verify:
|
|
|
|
|
- `strings <binary> | grep -c "embedding read worker panicked"` → `1`.
|
|
|
|
|
|
|
+1. **Every node must be *running* the packaged release build before any cutover**
|
|
|
|
|
+ (rewritten 2026-11-11, operator directive: “the flare itself must be the latest
|
|
|
|
|
+ version and properly updated on all nodes”). The release **is** installed and
|
|
|
|
|
+ identical everywhere — `Installed == Candidate ==
|
|
|
|
|
+ 0.1.0~git20261105002735.0009b89d-1`, `/usr/bin/flare` = `03c8e463` on
|
|
|
|
|
+ dgx1/2/3/4/8 — but dgx1/2/3/8 *execute deleted inodes* (the PR #184 binaries), so
|
|
|
|
|
+ only dgx4 actually runs the release. `readlink /proc/<pid>/exe` showing `(deleted)`
|
|
|
|
|
+ is the symptom; the fix is a restart onto the packaged file, never `cp`.
|
|
|
|
|
+ **The PR #184 rebuild is dropped**: PR #184 is not in the release (`strings -a
|
|
|
|
|
+ /usr/bin/flare | grep -c "embedding read worker panicked"` → **0**), and the current
|
|
|
|
|
+ `gb10` package cannot be loaded by anything older than `20261105002735` — so the
|
|
|
|
|
+ release build wins. Revisit only if/when the PR is merged upstream.
|
|
|
|
|
+ Keep the rollback binaries (`flare.orig-20261106`) until the cutover is verified.
|
|
|
2. Confirm NAS health and space, from every node that will use it:
|
|
2. Confirm NAS health and space, from every node that will use it:
|
|
|
```bash
|
|
```bash
|
|
|
ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
|
|
ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1
|
|
@@ -464,6 +473,44 @@ rsync -aH /mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V/deployments/ ~/.cache/fla
|
|
|
flare orbit run <group> --cluster sparks
|
|
flare orbit run <group> --cluster sparks
|
|
|
```
|
|
```
|
|
|
|
|
|
|
|
|
|
+## Runtime parameters (2026-11-11)
|
|
|
|
|
+
|
|
|
|
|
+Decision: **take the launch parameters from the current `gb10` package**, not from what
|
|
|
|
|
+the lane happens to run today — with one deliberate exception, **the API port stays
|
|
|
|
|
+30000** (every lane here serves on 30000; `8000` in their example is just their host
|
|
|
|
|
+convention).
|
|
|
|
|
+
|
|
|
|
|
+Their recipe vs. the live rank 0 command line on dgx1 (`tr '\0' ' ' <
|
|
|
|
|
+/proc/$(pgrep -f "[f]lare run" | head -1)/cmdline`):
|
|
|
|
|
+
|
|
|
|
|
+| flag | upstream `gb10` | live | action |
|
|
|
|
|
+|---|---|---|---|
|
|
|
|
|
+| `--max-concurrency` | `32` | `32` | keep |
|
|
|
|
|
+| `--mtp-draft-tokens` | `5` | `5` | keep |
|
|
|
|
|
+| `--mtp-adaptive-verification` | yes | yes | keep |
|
|
|
|
|
+| `--mtp-adaptive-verification-alpha` | **`0.75`** | **absent** | **added to both recipes** |
|
|
|
|
|
+| `--embedding-table-storage` | `disk` | `disk` | keep |
|
|
|
|
|
+| `--embedding-table-resident-scales` | yes | yes | keep |
|
|
|
|
|
+| `--embedding-table-cache-size` | `1GiB` | `1073741824` | equivalent (both = 1 GiB) |
|
|
|
|
|
+| `--transfer-streams 1` | absent | `1` | **keep ours** — RoCE workaround, `flare-roce-poison.md` |
|
|
|
|
|
+| `--host` | `0.0.0.0` | `0.0.0.0` | keep |
|
|
|
|
|
+| `--port` | `8000` | **`30000`** | **keep 30000** |
|
|
|
|
|
+| `--served-model-name deepseek-v4.1-flash` | absent | set | keep ours (client-facing name) |
|
|
|
|
|
+
|
|
|
|
|
+`alpha 0.75` is theirs on purpose: *“favor longer verification rounds, which raises
|
|
|
|
|
+decode throughput with four or more concurrent requests and coding throughput”* — i.e.
|
|
|
|
|
+it helps exactly the lane shape we run (c32, TP4).
|
|
|
|
|
+
|
|
|
|
|
+They also document memory/KV knobs (`--memory-limit=-12G`, `--kv-cache-limit
|
|
|
|
|
+memory|context|concurrency`) and recommend **leaving both at the default `memory`**:
|
|
|
|
|
+*“Every GiB reserved or cached away from the default reduces the KV capacity that
|
|
|
|
|
+Flare can plan.”* Their stated budget — ~11.4 GiB KV per GPU, ~13.1 M tokens, 17–18 GiB
|
|
|
|
|
+of 121 GiB left free — matches what we measure, so no change. The 1 GiB Engram row
|
|
|
|
|
+cache is their recommended default; they suggest 4–8 GiB only if memory is spare and
|
|
|
|
|
+the startup `Plan cache` line confirms room, and our own 48 GiB experiment failed with
|
|
|
|
|
+*“selected concurrency and verification width exceed available memory”* — so 1 GiB
|
|
|
|
|
+stays.
|
|
|
|
|
+
|
|
|
## 5. Risks and their fallbacks
|
|
## 5. Risks and their fallbacks
|
|
|
|
|
|
|
|
| risk | detection | fallback |
|
|
| risk | detection | fallback |
|
|
@@ -477,7 +524,15 @@ flare orbit run <group> --cluster sparks
|
|
|
## 6. What must be measured, not assumed
|
|
## 6. What must be measured, not assumed
|
|
|
|
|
|
|
|
1. Whether the agent's `--package-cache` path can be redirected by config or needs
|
|
1. Whether the agent's `--package-cache` path can be redirected by config or needs
|
|
|
- a launcher `extra_flags` change (phase 2 note).
|
|
|
|
|
|
|
+ a launcher `extra_flags` change (phase 2 note). **Partly answered 2026-11-11, and it
|
|
|
|
|
+ is worse than assumed:** on dgx1 the CLI's configured storage is
|
|
|
|
|
+ `/home/kilork/flare-pkgs/gb10`, an **empty 68 K skeleton** — `flare models list`
|
|
|
|
|
+ says *“No local model versions found”* and `flare models inspect` cannot see the
|
|
|
|
|
+ live package id — while the real 477 G store is in `~/.cache/flare/packages`
|
|
|
|
|
+ because rank 0 is launched with `--package-cache /home/kilork/.cache/flare`. So
|
|
|
|
|
+ redirection *works*, but CLI and agent are already pointed at different trees on
|
|
|
|
|
+ one node. Unify them (launcher `local_dir` + agent flag) **before** either points
|
|
|
|
|
+ at the NAS, or the NAS store will inherit the same split-brain.
|
|
|
2. Whether FLARE tolerates the bind mount across a real stage (phase 3 gate).
|
|
2. Whether FLARE tolerates the bind mount across a real stage (phase 3 gate).
|
|
|
3. ~~Real NAS read throughput~~ **measured 2026-11-11** (`dd iflag=direct`, 2 GiB,
|
|
3. ~~Real NAS read throughput~~ **measured 2026-11-11** (`dd iflag=direct`, 2 GiB,
|
|
|
so no page-cache lying): the NAS is **10 GbE, not 1 GbE**.
|
|
so no page-cache lying): the NAS is **10 GbE, not 1 GbE**.
|
|
@@ -493,3 +548,31 @@ flare orbit run <group> --cluster sparks
|
|
|
small files would load ~25× slower than the bulk figure suggests — count the
|
|
small files would load ~25× slower than the bulk figure suggests — count the
|
|
|
files in a rank deployment before trusting the 2–3 s number, and keep `weights/`
|
|
files in a rank deployment before trusting the 2–3 s number, and keep `weights/`
|
|
|
local regardless.
|
|
local regardless.
|
|
|
|
|
+4. **Package revision currency must be verified against the source before any storage
|
|
|
|
|
+ work** (2026-11-11: the fleet served a superseded Nov-3 build whose source repo now
|
|
|
|
|
+ 404s — see Phase 2R). To read what a local package *is*, do not trust the tree
|
|
|
|
|
+ layout; read the identity out of the payload (the `flare.manifest` tensor is
|
|
|
|
|
+ protobuf, but the strings are legible):
|
|
|
|
|
+ ```bash
|
|
|
|
|
+ strings -a <pkg>/model.safetensors | grep -aoE "https://huggingface.co/[^ ]*" | sort -u
|
|
|
|
|
+ ```
|
|
|
|
|
+ and compare with `https://huggingface.co/api/models/<repo>/commits/<revision>`.
|
|
|
|
|
+ The HF API masks a *missing* repo as 401 when unauthenticated, so always re-check
|
|
|
|
|
+ with a token before concluding a repo is gated. A valid token is
|
|
|
|
|
+ `export HF_TOKEN=…` in `~/.zshenv` **on the nodes** — there is none on the mac.
|
|
|
|
|
+5. **`flare-inference` must be the latest release and actually running on every node.**
|
|
|
|
|
+ Measured 2026-11-11: on dgx1/2/3/4/8 `Installed == Candidate ==
|
|
|
|
|
+ 0.1.0~git20261105002735.0009b89d-1` and `/usr/bin/flare` is the same file
|
|
|
|
|
+ (`03c8e463`) everywhere — but dgx1/2/3/8 *execute* deleted inodes (the PR #184
|
|
|
|
|
+ binaries), so only dgx4 is truly on the release build. The package's
|
|
|
|
|
+ `≥ 20261105002735` requirement makes this a hard gate: **no node can load the
|
|
|
|
|
+ current `gb10` revision until it restarts onto the packaged binary.**
|
|
|
|
|
+ PR #184 is **not** in the release (`strings -a /usr/bin/flare | grep -c
|
|
|
|
|
+ "embedding read worker panicked"` → 0), so running the latest build means giving up
|
|
|
|
|
+ the patched Engram reader until it is merged upstream.
|
|
|
|
|
+6. **Node root disks are not uniform** — dgx1 is **915 GiB**, dgx4 is 1.8 TiB. dgx1 was
|
|
|
|
|
+ the one at 98 % full; a plan sized from dgx4's disk would have missed the whole
|
|
|
|
|
+ reason this migration exists.
|
|
|
|
|
+7. Package stores across dgx1–dgx3 share one id (`7cf40bec…-35842073`), and the layout
|
|
|
|
|
+ here is **flat**: `du -l` reports **0 hardlinks / 184 files**, so `rsync -H` was
|
|
|
|
|
+ never load-bearing on this fleet.
|