# FLARE storage: expanded checkpoint on node NVMe, everything else on NAS (2026-11-10) Goal, in the operator's words: **only the expanded checkpoint stays on the sparks NVMe** (260 GiB for `DeepSeek-V4.1-Flash` tp4) and **all other model artefacts live on the NAS**. This document is written to be executed by someone with no context beyond it. Every command is copy-pasteable and every phase has a gate. ## Milestones - [x] NAS mount made boot-proof on every participating node (`nofail`, `timeo=100`, `x-systemd.mount-timeout=30`) — verified live, serving untouched - [x] dgx8's missing NAS mount restored (it was in fstab but never mounted) - [x] dgx1's stale NFS state cleared without a reboot; node pinned to the protocol that works there (4.0) — see the correction in Phase 1 - [x] Pre-migration state of dgx1/2/3/4/8 captured locally and on the NAS - [x] NAS throughput measured (was open item 6.3) - [x] **Stale model data purged from the nodes — 783 GiB freed**, dgx1 from 98 % full (22 G free) to 283 G free; serving never interrupted - [x] Launch parameters aligned with the current `gb10` package recipe; **API port stays 30000** (operator directive), `--mtp-adaptive-verification-alpha 0.75` added - [ ] **Every node on the v0.1.0 runtime** — **staged identically on dgx1/2/3/4/8 at `/usr/local/share/hyperflare/v0.1.0/flare`** (archive `7ba5ba43…`, binary `e3fe697e…` 26,054,848 B, same sha256 on all five). **Swapped and running on dgx4** (2026-11-12: `/usr/bin` + agent copy both `e3fe697e`, agent `exe` not `(deleted)`, `flare.orig-20261112` backups) — the swap is rehearsed and known-good. Remaining: dgx1/2/3/8 (needs the window, they hold the ranks) and dgx5–7 (off). **PR #184 rebuild dropped** (§6 item 5). Note `~/.local/lib/flare-agent/flare` is **not dpkg-owned** — apt never updates what the unit actually runs - [x] **Current `gb10` core acquired onto the NAS store and verified** — `checkpoint-expanded-scales/gb10-tp4-202611120031`, 474.66 GiB, **TP 4 · ready**, signatures verified, visible from dgx1/dgx4 over NFS (Phase 2R) - [ ] That core **deployed to the ranks, and the superseded build deleted** (needs the window; dgx1's 477 G and the ranks' 261 G dirs are still the live data) - [x] **Bind-mount layout validated on idle dgx4 (Phase 3 gate)** — a rank served from a NAS-backed `deployments/` (TP1 test bed), negative control failed correctly, no writes/rename by FLARE, fstab bind survives reboot with `Requires=mnt-nas.mount`. Remaining for the layout: apply it to the TP4 rank slices in the window (R7/R8). - [x] **Ranks do not depend on the NAS — proven, and the `package_cache` drift fixed.** A rank started **and answered a chat completion with `/mnt/nas` unmounted**, as long as its model lives in a local store, so the storage design has no NAS dependency on the rank-restart path. `paths.package_cache` turned out to be a **CLI fallback** (see §“What the binary says about cache roots”), not the rank's setting, so R4's NAS value was reverted to `/home/kilork/.cache/flare` on dgx1 + dgx4 (backups `config.toml.bak-20261112-rank-store`); dgx1 stayed `api=200` throughout - [ ] Sparks rolled out (Phase 4, needs a window) - [ ] prod4 rolled out (Phase 5, needs a window) - [ ] Verification matrix run and rollback drilled (Phase 6) ## Round plan - [x] **R1 (Q1)** — Phase 0.2–0.4 + Phase 1 on dgx1/2/3/4/8 - [x] **R2 (Q1)** — stale-data purge instead of migration (operator decision): **783 GiB freed** on dgx1/2/3, all ranks stayed up; also identified that every local package is a superseded build from a deleted repo (Phase 2R) - [x] **R3 (Q1)** — current `gb10` core **acquired and verified on the NAS store**: `models/checkpoint-expanded-scales/gb10-tp4-202611120031`, 474.66 GiB, `payload v2`, deployment `db20209d361f473b8452f24ad517fa0d` **TP 4 · ready**, 474.63 GiB in **181 hardlinked objects**, signatures verified by the publisher, `flare models list` correct from **dgx1 (prod) and dgx4**. 2 h 8 min, avg 63 MiB/s (peak 104). Note the gate command is `flare models list` / `--core-cache`, not `--model-cache` (see the vocabulary section) - [x] **R4 (Q1)** — `package_cache` now `/mnt/nas/ai/flare/store` on dgx4 + dgx1 (backups kept; dgx1 rank untouched, `api=200`, same PID). Answered the config question: v0.1.0 still honours `paths.package_cache`, the launcher never passes the cache flag, and the flag renamed `--package-cache` → `--core-cache` — see the new “v0.1.0 vocabulary and paths” section. Still open: confirm the *launched* cmdline after the first restart (needs the window) - [x] **R5 (Q1)** — Phase 0.1 pre-flight done, and the swap **rehearsed end-to-end on idle dgx4**: `/usr/bin/flare` and `~/.local/lib/flare-agent/flare` both `e3fe697e`, agent restarted (`systemctl --user restart flare-agent.service`, new pid, `readlink /proc//exe` **without `(deleted)`**), Nov-5 backups kept at both paths as `flare.orig-20261112`. The mechanism behind the fleet's deleted inodes is proven: **`dpkg -S ~/.local/lib/flare-agent/flare` → owned by no package**, so apt refreshes `/usr/bin/flare` and *never* the copy the unit ExecStarts. dgx4 is therefore ahead of dgx1/2/3/8 — do **not** add it to a group until the window swaps the others (one identical binary per running group) - [x] **R6a (Q1)** — Phase 3 bind-mount gate, part 1 on idle dgx4: v0.1.0 accepts the layout, `--stage-only` passes **Verify core** through a NAS bind, mount survives five stage attempts. Also found: the plan's Phase 3 procedure was **invalid** (a standalone model-dir copy is rejected in v0.1.0) and there is a **data-loss trap** in Phase 4 cleanup — see “Phase 3 gate results”. No data was lost. - [x] **R6b (Q1)** — **closed without a window**, on a single-GPU core (see “The single-GPU test bed”): a rank **served chat completions with `deployments/` bind-mounted from the NAS**, and the negative control (bind an empty NAS dir → rank refuses to start) proves the data comes *through* the bind. Across a complete load+serve cycle FLARE wrote **nothing** into the bind (0 files newer than rank start) and never replaced it (same inode, still mounted after serving and after reboot) ⇒ **option D is not needed**. Also validated the **fstab-persisted bind across a reboot of dgx4**: mounted at boot, its unit provably `Requires=mnt-nas.mount`, and a rank came up from it **22 s after boot**. Steady-state inference touches the NAS **0 MiB**. - [ ] **R7 (Q2)** — Phase 4 sparks rollout (window; ranks stop) - [ ] **R8 (Q2)** — Phase 5 prod4 rollout (window; ranks stop) - [ ] **R9 (Q3)** — Phase 6 verification matrix and rollback drill ## Checks **Only rank 0 serves the API.** `dgx1:30000/v1/models` answering is the health probe; `curl` on dgx2/dgx3/dgx8 returns `000` there by design (peers have no listening port) — do not read that as an outage. Peer health is `pgrep -fc "[f]lare run"` = 1. The runtime check a round should run (staged-vs-running, never inferred from `flare --version`, which prints `0.1.0` for every build): ```bash for h in dgx1 dgx2 dgx3 dgx4 dgx8; do printf "%-6s " $h ssh -o BatchMode=yes $h.lan 'A=$(pgrep -f "[f]lare run" | head -1); B=$(pgrep -f "[f]lare agent" | head -1) printf "staged=%s /usrbin=%s agent=%s exe=%s\n" \ "$(sha256sum /usr/local/share/hyperflare/v0.1.0/flare 2>/dev/null | cut -c1-8)" \ "$(sha256sum /usr/bin/flare | cut -c1-8)" \ "$(sha256sum ~/.local/lib/flare-agent/flare | cut -c1-8)" \ "$(readlink /proc/${A:-$B}/exe 2>/dev/null)"' 2>/dev/null || echo off done ``` **`agent=` is the column that matters.** The unit starts `%h/.local/lib/flare-agent/flare`, and `dpkg` owns no such file, so `/usrbin=` can move while the runtime does not. Expected now (dgx4 swapped, ranks still on the Nov-5 build): ``` dgx1 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted) dgx4 staged=e3fe697e /usrbin=e3fe697e agent=e3fe697e exe=/home/kilork/.local/lib/flare-agent/flare ``` Gate for the milestone: `agent=e3fe697e` on every node, and no `exe` ending in `(deleted)`. A node reporting `agent=03c8e463` after an apt upgrade is **not updated**. For the NAS mount, run from the controller; every node must report `fstab=1 live=1`: ```bash for h in dgx1 dgx2 dgx3 dgx4 dgx5 dgx6 dgx7 dgx8; do printf "%-6s " $h ssh -o BatchMode=yes -o ConnectTimeout=5 $h.lan 'printf "%s fstab=%s live=%s ranks=%s\n" \ "$(hostname -s)" "$(grep /mnt/nas /etc/fstab | grep -c nofail)" \ "$(findmnt -no OPTIONS /mnt/nas | grep -c "timeo=100")" "$(pgrep -fc "[f]lare run")"' \ 2>/dev/null || echo "off"; done ``` ## 1. Measured baseline (2026-11-10, read-only) | node | role | `models/` | `packages/` | rank cmdline cache | |---|---|---|---|---| | dgx8 | sparks rank 3 | 261 GiB | — | `--package-cache /home/kilork/.cache/flare` | | dgx1 | prod4 rank 0 | 261 GiB | 477 GiB | `--package-cache /home/kilork/.cache/flare` | Anatomy of one model directory — **this split is the whole plan**: ``` ~/.cache/flare/models/checkpoint-expanded-scales// model.safetensors 35 MiB signed envelope (manifest + locations) deployments//rank-/ 788 KiB (rank 0) … 1.8 GiB (rank 1-3) weights/ 260 GiB ← the expanded checkpoint ``` Package store (CLI side, rank 0 only): ``` ~/.cache/flare/packages/-/ model.safetensors 35 MiB deployments/ 7.2 GiB weights/ 470 GiB ← second copy of the same content ``` NAS: `/mnt/nas` = 33 TiB, 1.1 TiB used, `/mnt/nas/ai/flare` effectively empty. ### What the move actually saves — read before committing - sparks ranks 1–3: **1.8 GiB each** (261 → 260 GiB). Small. - rank 0 (prod4): **477 GiB** (the duplicate package store) plus 7.2 GiB of deployments. - Future model versions stop multiplying: only their `weights/` ever lands on NVMe. If the intent is "NVMe holds one checkpoint and nothing else", this plan delivers it. If the intent was primarily disk relief on sparks, the win is small and the cheap part (phase 2, the package store) may be enough. ## 2. Target layout ``` # every rank node, local NVMe /home/kilork/.cache/flare/models/// weights/ real directory 260 GiB model.safetensors bind mount → NAS/nodes///model.safetensors deployments/ bind mount → NAS/nodes///deployments # NAS /mnt/nas/ai/flare/ store/ paths.package_cache — CLI store, packages/, blobs/, staging/, refs/ (rank 0's 477 GiB lives here, once) nodes////{model.safetensors,deployments/} per-node small artefacts (bind targets) ``` Only `weights/` is allowed on node NVMe. Everything else resolves through the bind mounts, so FLARE's view of a model directory is unchanged. ## 3. Why bind mounts and not symlinks FLARE's own error strings include *"a package primary escapes its directory"* and *"staged entry is not a regular file"* — a symlink pointing outside the model directory is exactly what those checks reject. Bind mounts are invisible to the process: the path stays inside the directory and the entry is a real file. Do not substitute symlinks. **Known risk:** if FLARE ever replaces `deployments/` by rename (`rm -rf` + `mv`), the mount point disappears and writes fall back to local disk. Phase 3 is a gate that measures this; if it happens, fall back to **option D**: leave `deployments/` local (1.8 GiB) and move only the package store. Do not fight it. ## 4. Phases Each phase is independently revertible. Do not start a phase whose gate the previous phase failed. ### Phase 0 — prerequisites (no FLARE restart yet) 1. **Every node must be *running* the packaged release build before any cutover** (rewritten 2026-11-11, operator directive: “the flare itself must be the latest version and properly updated on all nodes”). The release **is** installed and identical everywhere — `Installed == Candidate == 0.1.0~git20261105002735.0009b89d-1`, `/usr/bin/flare` = `03c8e463` on dgx1/2/3/4/8 — but dgx1/2/3/8 *execute deleted inodes* (the PR #184 binaries), so only dgx4 actually runs the release. `readlink /proc//exe` showing `(deleted)` is the symptom; the fix is a restart onto the packaged file, never `cp`. **The PR #184 rebuild is dropped**: PR #184 is not in the release (`strings -a /usr/bin/flare | grep -c "embedding read worker panicked"` → **0**), and the current `gb10` package cannot be loaded by anything older than `20261105002735` — so the release build wins. Revisit only if/when the PR is merged upstream. Keep the rollback binaries (`flare.orig-20261106`) until the cutover is verified. **Corrected 2026-11-11 (R3): “the latest apt build” is no longer the latest runtime, and it cannot acquire the current package.** - The project was renamed again: `NVIDIA/spark-local-inference` → **301 → `NVIDIA/hyperflare-inference`** (default branch `main`, pushed 2026-11-11, 801 ★). (`github.com/hyperflare/hyperflare` is an unrelated 2017 repo — do not use it.) - Upstream now ships **first-class release archives from version tags** (commit `94fbce5f6` “Publish release archives from version tags”). Tag **`v0.1.0`**, published **2026-11-11T19:31Z**, assets: `flare-linux-arm64.tgz` **11,427,812 B**, `sha256 7ba5ba4387cfe56a2e8205d711625addf26633219ec670ca33c23fd046d22d84` (unpacks to `flare/flare`, 26,054,848 B, plus LICENSE/NOTICE/THIRD_PARTY_NOTICES). Contents of v0.1.0 include PR #29 *Multi-host disk prefix caching*, PR #35/#36 *requested logprob token IDs / System One scoring*, ARM SHA-256 acceleration. PR #184 is still **not** merged (marker string → `0` in the v0.1.0 binary too). - The APT repo (`ppa.local-inference-lab.ai`, still the documented channel in the `gb10` README, and its install stanza is unchanged: `lil-flare-archive.asc` + `flare.sources` + `apt install flare-inference`) **lags**: after a clean `apt-get update`, `apt-cache madison flare-inference` offers only `20261105002735 / 20261030013211 / 20261027170349`. There is no newer apt build. - **Measured consequence, not speculation:** with the installed Nov-5 binary the acquisition of the current `gb10` revision **never gets past `Verify package`** — 0 B for >120 s, holding `locks/entries/checkpoint-expanded-scales--gb10-tp4-…lock` and `locks/staging/-0.lock`, making one TLS handshake after another to `13.36.183.231` (HF's eu-west-3 LFS CDN) with no payload transfer. The **same command with the v0.1.0 binary streams immediately** (`Acquire core`, 55–58 MiB/s, 5 GiB in 90 s). So: **runtime channel = GitHub release archives (primary), source build (backup); apt is only a floor.** - `v0.1.0` is not yet packaged by us, so until the cutover the fleet still *runs* Nov-5 while *acquiring* with the tagged release binary. Two consequences: install the chosen build on **every** node in the window (one identical binary, temp+`mv`), and record the sha256 of whatever we install. **Staged 2026-11-11 (R3), zero disruption:** the verified archive is unpacked on **dgx1, dgx2, dgx3, dgx4, dgx8** to `/usr/local/share/hyperflare/v0.1.0/` (`flare`, `LICENSE`, `NOTICE`, `THIRD_PARTY_NOTICES.md`, `archive.sha256`, `binary.sha256`). Every node independently re-downloaded and checked `archive sha256 7ba5ba43…`; the resulting binary is `26,054,848 B`, `sha256 e3fe697eab1f97ff…` — **identical on all five**, and each runs `flare --version` → `flare 0.1.0`. `/usr/bin/flare` and `~/.local/lib/flare-agent/flare` were **not touched**, so nothing serving can be affected. dgx5–7 are off and still need this. **Use `host-scripts/flare-swap.sh `** — written and tested for this (2026-11-12: verified idempotent no-op on dgx4, and it correctly **refused** on dgx8 with a rank running, exit 1, `tmp-swap` files count 0, nothing disturbed). It swaps *both* paths by temp+`mv`, keeps a dated `*.orig-` backup at each, restarts only the agent unit, and verifies sha256 **and** `/proc//exe` for `(deleted)` before exiting 0. Inside the window it is one command per node: ```bash ssh dgxN.lan 'bash -s -- /usr/local/share/hyperflare/v0.1.0/flare' < host-scripts/flare-swap.sh # ranks are already stopped in the window; the script still refuses unless FORCE=1 FORCE=1 ssh dgxN.lan 'bash -s -- /usr/local/share/hyperflare/v0.1.0/flare' < host-scripts/flare-swap.sh ``` The swap is deliberately boring, and note that `/usr/bin/flare` is **not** what runs — the agent executes `~/.local/lib/flare-agent/flare` (and `dpkg` owns neither copy): ```bash # per node, in the window (rename, never cp — see the SIGBUS rule) sudo install -m 0755 /usr/local/share/hyperflare/v0.1.0/flare /usr/bin/flare.tmp-v010 sudo mv /usr/bin/flare.tmp-v010 /usr/bin/flare # the agent's own copy is what the unit starts; refresh it the same way, then # restart the ranks together (one group, one window). Verify afterwards: readlink /proc/$(pgrep -f "[f]lare run" | head -1)/exe # must NOT say (deleted) sha256sum /usr/bin/flare ~/.local/lib/flare-agent/flare # both must be e3fe697e… ``` 2. Confirm NAS health and space, from every node that will use it: ```bash ping -c1 -W2 nas.lan && df -h /mnt/nas | tail -1 ``` 3. Record the current state of each node for rollback — **done 2026-11-11**, files are on each node and in `/mnt/nas/ai/flare/` (`flare-before-gx10-dgxN.txt`). Hostnames are `gx10-dgxN`, **not** the `dgxN.lan` aliases — build filenames from `hostname -s` and do not assume the alias resolves to a filename: ```bash du -sh ~/.cache/flare/* | sort -h > /tmp/flare-before-$(hostname -s).txt findmnt -no TARGET,SOURCE,OPTIONS /mnt/nas awk 'NR>1' /proc/fs/nfsfs/servers | sort -u | wc -l # NFS client IDs tr '\0' ' ' < /proc/$(pgrep -f "[f]lare run" | head -1)/cmdline # exact launch command ``` The captured rank cmdline is the recovery recipe if a node has to be restarted outside `orbit` (the controller is currently unpaired), e.g. rank 0 on dgx1: `flare run ~/.cache/flare/packages/7cf40bec…-35842073/model.safetensors --package-cache ~/.cache/flare --rank 0 --rendezvous 0.0.0.0:29800 --max-concurrency 32 --mtp-draft-tokens 5 --mtp-adaptive-verification --embedding-table-storage disk --embedding-table-cache-size 1073741824 --embedding-table-resident-scales --transfer-streams 1 --host 0.0.0.0 --port 30000 --served-model-name deepseek-v4.1-flash --publisher-keys … --dangerously-approve-key … --dangerously-accept-publisher local-inference-lab --gpus 0` 4. ~~dgx8 has the NAS in fstab but not mounted~~ **fixed 2026-11-11** — mounted at `vers=4.1` and made non-blocking with the phase 1 options. **Pre-window disk math — do this before scheduling the window (2026-11-12).** The acquired core's `weights/` is **470 G**, i.e. *bigger* than the old 260 GiB the plan was written against, and per-node free space is the one thing a window cannot recover: | node | root | free now | superseded data freed by the window | note | |---|---|---|---|---| | dgx1 | 915 G | 283 G | 477 G package store (open by rank 0 until restart) | **tight**: ~760 G free afterwards | | dgx2 | 1.8 T | 467 G | 261 G model dir | comfortable | | dgx3 | 1.8 T | 470 G | 261 G model dir | comfortable | | dgx4 | 1.8 T | 1.5 T | — (idle) | the staging test bed | | dgx8 | 1.8 T | ~470 G | 261 G model dir | comfortable | A peer's local footprint is **not** the store total: on the old build dgx2 held 261 G where the package store was 477 G, and a rank slice is only 3 files / 1.3 GiB. The exact new per-rank number is unknown until a real stage happens — **measure it on dgx4 first** (1.5 T free), because if a rank needs the full 470 G locally then dgx1 lands at ~40 % free and must not receive any further local copies. Never start a window whose aggregate `weights/` exceeds a node's free space after its own deletion: a rank that cannot write its staging tree takes the whole group down. Gate: binary staged and verified on every node (`## Checks` probe), agent binary swapped wherever ranks are not running, NAS mounted on every participating node, and the table above re-measured the day of the window. ### Phase 1 — make the NAS mount boot-proof The existing line (all nodes) is: ``` nas.lan:/sna /mnt/nas nfs4 rw,hard,rsize=1048576,wsize=1048576,timeo=600,retrans=5,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0 ``` Two problems for this plan: a NAS hiccup blocks boot, and `hard` + long `timeo` means a stale mount hangs reads (this NAS has already produced a `server not responding` outage that needed reboots). Applied everywhere, **with one per-node exception** (see below): ``` nas.lan:/sna /mnt/nas nfs4 rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576,timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=4.1 0 0 ``` Do **not** use a blanket `sed` over the `nas.lan:/sna` line — it silently rewrites the protocol version too. Rewrite the entry by its mount-point field and pass the protocol per node — use **`host-scripts/set-nas-mount-opts.sh [remount]`** (proven idempotent on dgx4 on 2026-11-11; it backs up fstab, rewrites the entry, and plain-`umount`s before mounting). For reference, it is equivalent to: ```bash sudo cp -a /etc/fstab /etc/fstab.bak-$(date +%F-%H%M) sudo python3 - "$VERS" <<'PY' import sys vers = sys.argv[1] path, out, hit = "/etc/fstab", [], False for line in open(path).read().splitlines(True): f = line.split() if len(f) >= 3 and f[1] == "/mnt/nas": opts = ("rw,hard,nofail,x-systemd.mount-timeout=30,rsize=1048576,wsize=1048576," "timeo=100,retrans=3,noatime,nodiratime,actimeo=600,nconnect=8,vers=" + vers) out.append("nas.lan:/sna\t/mnt/nas\tnfs4\t%s\t0\t0\n" % opts); hit = True else: out.append(line) assert hit, "no /mnt/nas entry in fstab" open(path, "w").write("".join(out)) PY sudo systemctl daemon-reload sudo umount /mnt/nas && sudo mount /mnt/nas # umount fails safely if something holds it findmnt -no OPTIONS /mnt/nas | grep -o 'timeo=100' ``` `nofail` is an fstab-only option: it never appears in live mount options, so verify the *live* change with `timeo=100`, not with `nofail`. **dgx1 exception — applied and verified 2026-11-11.** dgx1 cannot mount NFSv4.1 at all: a fresh `vers=4.1` mount hangs even with a single clean client and with `nconnect=1`, while dgx2/3/4/8 mount 4.1 instantly. Its fstab is therefore pinned to `vers=4.0` with the same other options. Its two stale client IDs were cleared without a reboot (`umount`, not `umount -l`, with zero open files — that is the procedure; a reboot was not needed). Gate: mount present, `nofail` in effect, a `sudo mount -a` is a no-op. **Passed on dgx1/2/3/4/8 (2026-11-11):** live options show `timeo=100`, `mount -a` returns 0 without remounting, all rank PIDs kept their 7-day uptime and the API on `dgx1:30000` still answered. dgx5–7 are off and still need this. ### Phase 2 — package store (the 477 GiB) to NAS — **SUPERSEDED, do not run** > **Superseded on 2026-11-11 (R2).** Two independent reasons: > 1. The operator rejected it: *“Syncing those old flare cache to store does not make > much sense honestly.”* Copying a superseded build onto the NAS is not migration, > it is archiving garbage. > 2. It is superseded by fact: the local package is an **out-of-date build whose source > repo no longer exists** (see Phase 2R). The rsync was started, reached 105 GiB, > was killed, and the partial copy was deleted. What replaced it: **purge the stale data** (done, below) and **acquire the current revision onto the NAS store** (Phase 2R). #### Purge performed on 2026-11-11 Every node held the **Nov-2 build** `checkpoint-expanded-scales/gb10-tp4-202611021936` (261 GiB each) with nothing holding it open — verified three ways per node before deleting (`/proc//fd`, `/proc//maps`, and `lsof +D`): | node | before | after | freed | |---|---|---|---| | dgx1 | 848 G used, **22 G free (98 %)** | 587 G used, **283 G free** | 261 G | | dgx2 | 664 G used, 206 G free | 403 G used, 467 G free | 261 G | | dgx3 | 661 G used, 209 G free | 400 G used, 470 G free | 261 G | **Total 783 GiB freed.** Rank PIDs were unchanged (7-day uptime) and `dgx1:30000/v1/models` kept answering. Note dgx1's root is **915 GiB**, not 1.8 TB like dgx4 — the 261 GiB was 28 % of that disk. Still present and **not deletable while serving**: - dgx1 `~/.cache/flare/packages/7cf40bec…-35842073` — 477 G, and rank 0 has its `weights/engram-*.safetensors` (92 G each) open. Space returns only when the rank exits, so this one dies in the Phase 5 window, not before. - dgx2/dgx3 `models/…202611032142` and dgx8 `models/…202611022100` (261 G each) — the live ranks read these. For reference, the original (not executed) procedure was: ```bash sudo mkdir -p /mnt/nas/ai/flare/store sudo chown kilork:kilork /mnt/nas/ai/flare/store # rsync -H is mandatory: format-2 packages are hardlink stores rsync -aH --info=progress2 ~/.cache/flare/packages/ /mnt/nas/ai/flare/store/ ``` Point the CLI at it, in `~/.config/flare/config.toml` on the node that downloads (rank 0 / controller): ```toml [paths] package_cache = "/mnt/nas/ai/flare/store" ``` Then remove the local copy only after a verified download check: ```bash flare models list --model-cache /mnt/nas/ai/flare/store # must list the version rm -rf ~/.cache/flare/packages ``` Gate: `flare models list` sees the version on NAS, `df -h /` shows the space back, and the next `orbit run` still starts (see phase 4 gate for the full test). **Open item to verify, not assume:** ranks are launched with an explicit `--package-cache /home/kilork/.cache/flare` that does **not** come from `[paths]` (the agent synthesises it). Check the launched cmdline after a start: ```bash pgrep -af "[f]lare run" | head -1 | tr ' ' '\n' | grep -A1 -e package-cache -e model-cache ``` If that path needs to change, it is a **launcher** change (`[flare].extra_flags` / `[flare.clusters.].extra_flags` in the sm121 config, `src/native/flare_orbit.rs::extra_flags_for`) — never a hand edit on a node. ### Phase 2R — replace, don't migrate (2026-11-11) **What we are actually serving is superseded, and its source is gone.** The acquired package's manifest (read out of the `flare.manifest` tensor inside `model.safetensors` — it is protobuf, not JSON, but the strings are legible) names: ``` source https://huggingface.co/local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE2 model checkpoint-expanded-scales identity gb10 publisher local-inference-lab ``` That repo answers **404 with a valid token** (401 unauthenticated is HF masking a missing repo). The current source is the same publisher **without the `2`**, and `gb10` is a *branch* of it: | | | |---|---| | repo | `local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE` (public, no token needed) | | revision | branch `gb10` — last commit **2026-11-05T01:42:45** *“Publish the GB10 TP4 package with vocabulary-parallel sampling and an FP8 Markov head”* | | size | **187 files, 474.7 GiB** (we hold 184 files / 477 GiB from a Nov-3 pull) | | requires | `flare-inference` **`0.1.0~git20261105002735.0009b89d-1` or newer** — “earlier versions cannot load it” | That version is exactly what the fleet has *installed* on every node. The binaries *running* are the older PR-184 inodes, so **no node can load the new package until the Nov-5-based rebuild (R5) lands** — the same conclusion #11 reached, now with a functional reason instead of a performance one. Download into the NAS store (this is what the store is *for* — fresh acquisitions, not old caches). The `hf` token, when one is needed, is `export HF_TOKEN=…` in `~/.zshenv` **on the nodes** — there is no token on the mac: ```bash # Verified working form (run on dgx4, the idle node, with the NAS mounted): ~/hyperflare/flare/flare download local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE \ --revision gb10 \ --tensor-parallel 4 \ --model-cache /mnt/nas/ai/flare/store \ --publisher-keys ~/.cache/flare/publishers/937c0143f6856d3ee8ad164f62e3d804bd825d7a8750d0accb9a5849f703a8c1.json \ --token-file ~/.config/hf/token flare models list --model-cache /mnt/nas/ai/flare/store # must list the new version ``` Three things the upstream example does **not** tell you (all hit in R3): 1. **`--tensor-parallel 4` is mandatory here.** Their example works because they run it on the controller of a four-Spark group; on a lone Spark the default is the local GPU count (1), and the acquisition dies with `Error: "package exports no deployment with tensor-parallel size 1"` after fetching the 35 MiB envelope. 2. **Publisher trust must be non-interactive.** `flare download` has *no* `--dangerously-approve-key` / `--dangerously-accept-publisher` flags (those exist on `orbit run` only) — it takes `--publisher-keys ` and `--trust-store `. The two files to copy from a node that already accepted the key: `~/.config/flare/trust.toml` (**fleet-uniform**: dgx1 and dgx2 have the same file, 270 B, `sha256 26e5ae90568572a28cbea9fb4b274ddca4127f4edf00b2f23086a2079db78464`, containing the `937c0143…` fingerprint) and `~/.cache/flare/publishers/937c0143….json` (219 B, `sha256 c8215a75db94b6dc4ee2992789acb675527caf2806380b80b3b98d1c79698a38`, from dgx1 — dgx2/dgx4's `publishers/` directories are empty). Copy them with `base64 -w0` and **compare sha256 on both ends**; do not overwrite a trust store without checking it first (dgx4 had its own copy of the same file). 3. **A token file helps even for a public repo** — `--token-file ~/.config/hf/token` (mode 600, built from `export HF_TOKEN=…` in `~/.zshenv` on the node) keeps the LFS transport off anonymous rate limits over 475 GiB. `--model-cache` is a *flag*; it does not touch `[paths]` in config, so it causes no drift on the node. Everything lands under `/mnt/nas/ai/flare/store/` as `transport/hub/v1/{blobs,repos}/…` plus `staging/`, `locks/`, `refs/`, `models/` — the `transport/` default comes from `--cache-dir` (“parent directory for the shared Hugging Face download transport cache [default: the model storage directory]”), so all 475 GiB is written once, on the NAS, and renames inside it are cheap. Progress of the acquisition started 2026-11-11 22:07 on dgx4 (detached, survives the session — `setsid nohup`): ```bash tail -c 400 ~/flare-download-gb10.log # symlink to the newest log du -sh /mnt/nas/ai/flare/store # bytes landed on the NAS pgrep -af "[f]lare download" # still alive? ``` Observed rate **55 → 90 MiB/s as the transfer warms up** (WAN-bound, not NAS-bound: inbound on `enP7s7` measured 1482 MiB / 20 s ≈ 74 MiB/s while the CLI reported 86 MiB/s) ⇒ 474.7 GiB in roughly **1.5 h** from 22:07. Trust the CLI's own elapsed timer in the log, not the wall clock: node clocks are **UTC** and the mac is not, so `date` on the two disagree by about an hour. The upstream launch recipe (`SPARK-DS4-GUIDE.md` / the `gb10` README) is worth aligning our launcher flags with — note it differs from what prod4 runs today: ```bash flare orbit run "spark1:${package_path}" \ --cluster sparks --name deepseek1017 \ --max-concurrency 32 --mtp-draft-tokens 5 --mtp-adaptive-verification \ --mtp-adaptive-verification-alpha 0.75 \ --embedding-table-storage disk \ --embedding-table-resident-scales --embedding-table-cache-size 1GiB \ --host 0.0.0.0 --port 8000 --detach ``` (Differences from our running rank 0: it adds `--mtp-adaptive-verification-alpha 0.75`, uses port 8000 not 30000, and names the group. Orbit ships each peer only the data it needs — “You do not need four full downloads.”) Order of work: **R5 rebuild → download fresh to NAS (R3) → window: stop ranks, delete the superseded 477 G on dgx1 and the 261 G dirs everywhere, start on the new package → then the bind-mount layout (Phase 3+) on top of the new version dirs.** Doing the bind-mount work on the old version dirs would be wasted: the paths contain the version name, and every fstab entry would have to be rewritten. > **Version ids are mutable; deployment ids are not.** The core was first acquired as > `gb10-tp4-202611112207`. That version's *materialisation* was destroyed by a recursive > delete through a live bind (see “Phase 3 gate results”), and re-running `download` > re-linked it from local blobs as **`gb10-tp4-202611120031`** — which is the id used by > every command in this document now. The deployment id > **`db20209d361f473b8452f24ad517fa0d` (TP 4 · ready)** is content-derived and **did not > change**, as did the inventory (`474.66 GiB`, `181 hardlinked objects`). Recovery cost: > **34 MB of network**, the remaining 509.6 GB re-read from local blobs, 1251 s. > Current state: `1 model · 1 version`, `[DEFAULT]` = `gb10-tp4-202611120031`. #### Acquisition result (2026-11-12 00:16 UTC) — measured, use these numbers ``` store/…/models/checkpoint-expanded-scales/gb10-tp4-202611120031/ model.safetensors 33 MiB mode 0444 (read-only) weights/ 470 G deployments/db20209d361f473b8452f24ad517fa0d/ rank-0 rank-1 rank-2 rank-3 1.3 G each, 5.1 G total ``` `flare models list`: `1 model · 1 version`, `Format payload v2`, `Disk used 474.66 GiB`, `Shared files 474.63 GiB · 181 hardlinked objects`, `Deployment db20209d361f473b8452f24ad517fa0d · TP 4 · ready`, `Context / prefill 1,048,576 / 8,192 tokens`, `Pages / state rows 128 / 448`. `flare models inspect checkpoint-expanded-scales`: name **DeepSeek-V4.1-Flash**, publisher *local-inference-lab (signatures verified)*, **sm_121**, vocab 129,280, **concurrency 32 · 4 ranks**, speculation `mtp · widths 1,2,3,4,5,6`. Acquisition accounting: `network 509,628,494,618 B`, `local source 509,653,222,270 B`, `verification 509,646,855,013 B`, `materialized 0 B`, **elapsed 7682 s (2 h 08 m)**, average 63 MiB/s (peak 104 MiB/s measured on `du`). **No 2× leftover**: the transport cache was folded into `blobs/` — it peaked at 475 G next to a growing `blobs/`, and at the end `transport/` is 88 K and `blobs/` is the whole 475 G. So the earlier 2×-peak warning applies to the *middle* of the run only, and `models gc` was not needed. **This changes the Phase 3 arithmetic, in our favour.** A rank slice is **three files**: `initial-state.safetensors` 1,321,056,820 B, `kernels.safetensors` 39,429,126 B, `graphs.safetensors` 4,207,295 B. Measured directly on the real files from dgx4 (`dd iflag=direct`, so no page-cache lying — note this `dd` has no `status=nocache`, do not add it): the whole 1.3 GiB slice reads in **1.29 s at 1.1 GB/s** (`initial-state` 1.242 s, `kernels` 0.038 s, `graphs` 0.006 s at 689 MB/s). `nconnect=8` with 4 MiB reads beats the earlier 1 MiB estimate of 660–820 MiB/s. **So a NAS-backed `deployments/` costs ~1.3 s per rank at load** — the small-file fear in §6 item 3 does not apply to this package, and Phase 3 has every prospect of passing on the numbers, not by hope. `model.safetensors` is mode 0444, so bind-mount it read-write and do not expect to replace it. ### v0.1.0 vocabulary and paths — read before running any command (2026-11-11, R4) The runtime we are moving to renamed the artefact. Measured against both binaries on dgx4 (`/usr/bin/flare` = Nov-5, `/usr/local/share/hyperflare/v0.1.0/flare`): | | Nov-5 build (installed) | v0.1.0 (staged) | |---|---|---| | artefact name | package | **core** (“Serve a local **core**”, “Download a prepared **core**”) | | serve/orbit flag | `--package-cache` | **`--core-cache`** (`run`, `orbit run`, `models list`) | | download flag | `--model-cache` | `--model-cache` (unchanged — the two flags coexist, do not “fix” one into the other) | | default store root | `~/.cache/flare/packages` | `~/.cache/flare/cores` | | config key | `paths.package_cache` | **still `paths.package_cache`** — verified: with `package_cache` set, `models list` reports that path under both binaries | | store root layout | `blobs/ refs/ staging/ locks/ sources/ origins/ models/` | same, plus `transport/` (the HF download cache) | Consequences that actually bite: 1. **No automatic migration exists** — neither binary contains a `migrat*`-with-core/package string. An upgraded node simply looks somewhere else unless `paths.package_cache` says otherwise. Since the fleet pins `package_cache` in `~/.config/flare/config.toml`, **that key is what keeps the upgrade from orphaning 474 GiB** — do not let a node end up without it. 2. **The launcher needs no change for this.** It never passes the cache flag (`grep -rn "package-cache" src/` → only a comment in `flare_package.rs`); the agent synthesises it. So the flag rename travels with the binary swap, not with our config. Confirm in the window anyway: `tr '\0' ' ' < /proc/$(pgrep -f "[f]lare run" | head -1)/cmdline`. 3. **`flare models` is now a store manager** — `list, inspect, download, downloads, select, remove, gc`. Use `models remove` / `models gc` for the superseded build instead of `rm -rf`, and `models select` to pin the default version. `models downloads` inspects the transport cache. **Open question for the verification step:** whether `transport/` keeps a second copy of the payload (⇒ ~950 GiB on the NAS until `models gc`); check `du -sh /mnt/nas/ai/flare/store/*` and `models downloads` once the acquisition lands. **Half-answered mid-flight**: at 39 GiB downloaded, *all* of it sits in `transport/` (40 K in `locks`, 24 K in `staging`, 8 K in `blobs`) — so materialisation into `models/` happens only after the whole payload is cached, and `staging/trees/-` accumulates one tree per attempt (the killed Nov-5 attempt is still there as `1800521-0`). 4. `flare trust` exists now (`revoke`), for publisher trust management. 5. **Node clocks disagree by an hour** (dgx1 `23:14` while dgx4 `22:14` for the same instant). Never compare timestamps across nodes; filename your backups with the local node's clock and compare *mtime within a node* only. ### R4 — CLI store pointed at the NAS store (done 2026-11-11) Applied on **dgx4** (the acquiring node) and **dgx1** (prod rank 0): ``` package_cache = "/mnt/nas/ai/flare/store" # NAS store (2026-11-11, R4); was /home/kilork/flare-pkgs/gb10 (empty skeleton) ``` Backups: `~/.config/flare/config.toml.bak-2026-11-11-2214-nas-store` (dgx4), `…-2314-nas-store` (dgx1). Verify: `flare models list` prints `Storage /mnt/nas/ai/flare/store`. **dgx1's rank was untouched** — same PID (533700), 7.5-day uptime, `api=200` — because the rank's cache path comes from the agent's explicit flag, not from this key. What the old value was doing: it deliberately equalled the recipe's `local_dir` (`/home/kilork/flare-pkgs/gb10`) so that the launcher's rsync mirror would see `src == dst` and skip copying, materialising the ~500 GB package once. In practice the mirror never filled up (the agent writes to `~/.cache/flare/packages`), so the key pointed at a 68 K empty skeleton and `flare models list` reported “No local model versions found” while 477 G sat elsewhere. With the store on the NAS, the `local_dir` mirror is obsolete — retire it in the Phase 5 window rather than now, so the launcher keeps a working path until the cutover. ### Phase 3 — validate the bind-mount model on an idle node (gate) Do this **before** touching dgx1–dgx3. Use **dgx4**: it is idle (no ranks), and now boots from the new MP600 Mini. The paths for this gate now exist — do not invent them. Source (NAS): `/mnt/nas/ai/flare/store/models/checkpoint-expanded-scales/gb10-tp4-202611120031/` with `model.safetensors` (0444), `weights/` (470 G) and `deployments/db20209d361f473b8452f24ad517fa0d/rank-/` (3 files, 1.3 G each). Put the local copy at `~/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-202611120031/` (v0.1.0's default root is `~/.cache/flare/cores`, but this fleet pins `paths.package_cache`, so use whatever that key says on the node under test). 1. Copy one model's small artefacts from a running node: ```bash ssh dgx8.lan 'tar -C ~/.cache/flare/models// -cf - model.safetensors deployments' \ | tar -C /tmp -xf - ``` 2. Build a fake local model dir whose `weights/` is a symlink to nothing and whose `deployments/` is a **bind mount from the NAS copy**: ```bash mkdir -p ~/.cache/flare/models///weights mv /tmp/model.safetensors /tmp/deployments ~/.cache/flare/models/// sudo mkdir -p /mnt/nas/ai/flare/nodes/dgx4// sudo mv ~/.cache/flare/models///deployments /mnt/nas/ai/flare/nodes/dgx4/// sudo mount --bind /mnt/nas/ai/flare/nodes/dgx4///deployments \ ~/.cache/flare/models///deployments ``` 3. Ask FLARE what it sees: ```bash flare inspect ~/.cache/flare/models///model.safetensors | head -20 flare models list ``` 4. If a stage/launch is possible on dgx4 (`flare run --stage-only` with a dummy agent), run it and then check the mount survived: ```bash findmnt -no TARGET ~/.cache/flare/models///deployments ``` **Empty output = FLARE replaced the directory → take option D.** Gate: `flare inspect` shows the deployment, and `findmnt` still reports the bind mount after any stage operation. ### Phase 3 gate results (2026-11-12, dgx4) — procedure corrected, one data-loss trap **What passed.** With the exact Phase 4 shape — per-node artefacts on the NAS under `nodes/gx10-dgx4///deployments//`, bind-mounted over the model directory's `deployments/` inside a store root — `flare run --stage-only` **passed `Verify core`** and reached `Wait for tensor-parallel hosts` (the furthest a lone node can get). The bind was unchanged before and after: same inode, same mount, `findmnt` still reporting the NAS source. v0.1.0's inventory layer is also bind-safe: `flare models list` over a locally-assembled root reported the full inventory (`474.66 GiB`, `181 hardlinked objects`) while `blobs/` and `weights/` were bind mounts onto a **different device** (st_dev 80 = NFS vs 66306 = local NVMe). > **Correction to what R5 concluded here (2026-11-12, R6).** R5 blamed a missing > store-root scaffolding for the `open core directory: No such file or directory` > failure. That was a **misattribution**. The real cause, proved by the negative control > below: that error is what v0.1.0 reports when **`deployments//rank-N` > is missing or one level off**. R5's failing binds pointed at > `deployments/` *as* `deployments/`, so the expected `rank-N` level vanished. Bind a > directory that contains `/rank-N` and the very same tree verifies and serves. A > store does indeed have no metadata files (purely structural), and the model dir does > have to live inside its store — but do **not** go copying `blobs/ refs/ staging/` > around on the strength of the old wording; that was never what the error meant. The plan's original Phase 3 step 1–2 procedure was still wrong in another way: a copied `model.safetensors` alone is not a core. Working test shape (verified): ```bash S=/mnt/nas/ai/flare/store; R2=$HOME/flare-bindtest # any local store root mkdir -p $R2 && sudo mount --bind $S $R2 # root scaffolding M=checkpoint-expanded-scales; V=gb10-tp4-202611120031 D=$R2/models/$M/$V; DEP=db20209d361f473b8452f24ad517fa0d N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V # per-node artefacts on NAS sudo mount --bind $N/deployments $D/deployments # per-node slice on top flare run $D/model.safetensors --package-cache $R2 --rank 0 \ --rendezvous 127.0.0.1:29814 --gpus 0 --stage-only findmnt -no TARGET,SOURCE $D/deployments # must still print the bind ``` **Data-loss trap — this is the most important line in the document.** A `rm -rf` over a tree that still has **live bind mounts deletes through them**, into the NAS. That is how the freshly acquired core's `models///weights/` + `deployments/` materialisation was destroyed during test cleanup on dgx4: the `rmdir` of the two mount points failed with `EBUSY` (which looks like “it refused”), while every file *inside* them was deleted on the NAS. **Unmount first, always**, or make the refusal real: ```bash findmnt -rno TARGET | grep -F "$D" | sort -r | xargs -r -n1 sudo umount # deepest first sudo rm -rf --one-file-system "$D" # belt and braces ``` Nothing was lost, because the store is content-addressed: `blobs/` still held the whole **475 G** (link count dropped to 1), and `nodes/gx10-dgx4/` still held all four deployment slices (5.2 G). Recovery is just re-running `flare download` into the same store — all blobs are local, so it re-materialises the hardlink farm instead of re-downloading. **Do not panic at `Disk used 32.4 MiB` in `models list`; check `du -sh /blobs` first.** Three measurement gotchas that look like corruption and are not: - `du -sh a b c` **deduplicates hardlinks across its arguments** — measured `du -sh $S/models $S/blobs` as `475G` + `6.2M`, i.e. “the blobs vanished”, when in fact each directory alone is 475 G. Measure one directory per call. - NFS `actimeo=600` dir caching makes a live artefact look **missing** while test binds are stacked (`cp: cannot stat …/rank-0: No such file or directory`); the file was there the whole time. Unmount the binds before debugging. - `cp -al` between two NAS paths failed with **`Invalid cross-device link`** even though both were under `/mnt/nas`. Use `cp -a` for per-node artefacts (1.3 GiB, cheap). **What is still unproven (R6b).** strace shows FLARE opens with dirfd-relative paths — `openat(7, "tp4", O_RDONLY|O_NOFOLLOW|O_DIRECTORY)` — so grepping strace for full paths **undercounts reads**; the positive test that a deployment file was read *through* the bind needs a completed stage, which needs the other three ranks. The `O_NOFOLLOW` in that same line is the escape check, and it is why **symlinks remain banned** here. If a completed stage turns out to rename over `deployments/` or to hardlink into it (cross-device hardlinks fail with EXDEV, as just demonstrated on NFS), take **option D** without arguing: leave `deployments/` local (1.3 GiB) and move only the package store. ### The single-GPU test bed — how to close any layout question in 6 minutes, no window Every “does FLARE tolerate X” question above was answered on **one idle node** with a 25 GB single-GPU core instead of waiting for the TP4 group. Keep this recipe — it is the cheapest way to test a storage or launch change: ```bash # local-inference-lab/Qwen3.8-27B-FLARE, branch gb10 → 25.5 GB, 317 s at ~80 MiB/s R=$HOME/teststore; mkdir -p $R flare download local-inference-lab/Qwen3.8-27B-FLARE --tensor-parallel 1 \ --model-cache $R --publisher-keys ~/.cache/flare/publishers/937c0143*.json \ --token-file ~/.config/hf/token # → model Qwen3.8-27B-NVIDIA-NVFP4-QAD-MXFP8 · gb10-tp1- · payload v2 # 23.77 GiB · 69 hardlinked objects · Deployment gb10-tp1 · TP 1 · ready # context/prefill 262,144 / 4,096 · weights 22 G · deployments 2.5 G · envelope 18 M flare run $R/models///model.safetensors --package-cache $R --gpus 0 \ --host 127.0.0.1 --port 30080 --publisher-keys ~/.cache/flare/publishers/937c0143*.json curl -s http://127.0.0.1:30080/v1/models # 200 in ~20 s; chat completions work ``` Gotchas measured on the way: **`--rank` and `--rendezvous` are rejected for a single-host core** (`--rendezvous applies only to a multi-host core`), the deployment id here is human-readable (`gb10-tp1`) instead of a hash, this model streams into `reasoning_content` with `content: null`, a 262k context makes the rank reserve ~109 GiB of RAM (`MemAvailable 117 → 8 GiB`), and after `pkill` of a rank allow **~10 s** for the port to free before restarting (unmapping 109 GiB is not instant — restarting after 3 s produced silent failures here). **Proven with it** (`deployments/` moved to `nodes////deployments` on the NAS and bind-mounted over the model directory's `deployments/`): - rank serves: `api=200`, `20 s` (warm) / `21 s` (cold, after `drop_caches` on the idle node); - negative control — bind a NAS dir with **no** rank slice → `[FAILED] Verify core … open core directory: No such file or directory`, i.e. no hidden fallback to disk; - **nothing written into the bind**: `find $D/deployments -newermt "$START"` → 0 files, inode identical, mount intact after serving, after restart, and after reboot; - **steady-state inference pulls 0 MiB** over the network — the artefacts are read at load, so the NAS cost of a bind is bounded by the rank slice (1.3 GiB for TP4, measured 1.29 s at 1.1 GB/s) and happens once per start. **Boot persistence (fstab) — validated by rebooting idle dgx4**, the exact stanza Phase 4 installs, and nothing else in `/etc/fstab` touched (backup `/etc/fstab.bak-20261112-fsbind`): ``` /ai/flare/nodes////deployments /home/kilork/.cache/flare/models///deployments none bind,nofail,x-systemd.mount-timeout=30,x-systemd.requires=/mnt/nas,x-systemd.requires-mounts-for=/mnt/nas 0 0 ``` After reboot: `findmnt` shows the bind, `systemctl show -p Requires` contains **`mnt-nas.mount`** (so the bind can never be mounted before the NFS root), and a rank started from it **22 s after boot** with the bind intact. Removing it again: delete the fstab line, `daemon-reload`, **unmount deepest-first, then** `rm -rf --one-file-system` (see the trap above). ## 5. What the binary says about cache roots (read before editing any config) From `flare run --help` in the installed build (not inference): > `package_cache PACKAGE_CACHE` — **Fallback** prepared-model cache root when > `--model-cache` is absent [default: `paths.package_cache`, else `~/.cache/flare/packages`] So `paths.package_cache` is the **CLI's fallback**, not the rank's cache. Proof from the fleet: dgx2/dgx3 config says `package_cache = "/home/kilork/flare-pkgs/gb10"`, yet their captured rank command lines both carry `--package-cache /home/kilork/.cache/flare` — the group spec / agent passes it **explicitly**. Measured drift across participating nodes (2026-11-12), all three values different: | node | `paths.package_cache` | rank actually used | |---|---|---| | dgx1 | `~/.cache/flare` (fixed 2026-11-12) | `~/.cache/flare` | | dgx2, dgx3 | `~/flare-pkgs/gb10` ← skeleton, drift | `~/.cache/flare` | | dgx4 | `~/.cache/flare` | — (no ranks) | | dgx8 | `~/.cache/flare/packages` ← old default | `~/.cache/flare` | **Normalise dgx2/dgx3/dgx8 to `~/.cache/flare` inside the window** (CLI-only edit, but serving nodes are not edited outside a window by rule). And never point this key at the NAS: it is the fallback, and a fallback that resolves off a network share is a rank that might one day try to load 474 G over NFS. **Availability, measured rather than assumed:** with `/mnt/nas` fully unmounted on dgx4 (`lsof +D /mnt/nas` → 0 holders, clean `umount`), the TP1 rank started and returned a chat completion. The NAS is therefore **not** on the rank-restart path as long as each rank's model lives in a local store — which is exactly what this plan guarantees (`weights/` local, small artefacts bind-mounted). The bind itself *does* depend on the NAS at start (a missing bind means no `rank-N` → the rank refuses to start), so `nofail,x-systemd.requires-mounts-for=/mnt/nas` in fstab is load-bearing, not decoration. Broken entries are labelled, not silently picked: `flare models list` reported `gb10-tp4-202611112207 [UNREADABLE] … cache entry … has no primary` for the R5 debris (12 K, 0 files). Verified `findmnt` showed 0 binds under it, then removed it with `rm -rf --one-file-system`; dgx1/2/3/8 report no such entries. ### FLARE already ships the distribution tools the plan hand-rolls Also straight from the help text — worth trying **instead of rsync** during the window: - `orbit push --to NODE[,…]` — “one per rank in rank order, or any number to receive the complete package”, with `--streams N` “parallel TCP streams per destination file **over the fabric**”, and `--roce-ring` / `--roce-chain` placement. - `orbit stage` — “Stage package data on nodes **without starting ranks**”; `orbit ls`, `orbit status` (“each node's capacity, groups, and package cache”), and `cache list [MODEL] [--json]` (“show what every node's package cache holds”). - `flare run PACKAGE` accepts a local path, `MODEL[@VERSION]`, or a HF repo — **“local models resolve first”**, and `--verify-weights` exists to re-check stored ranges. If `orbit push` writes rank slices directly into each node's local store, Phase 4's rsync steps (and their `EXDEV`/hardlink traps) disappear. Try it on dgx4 with the TP1 core first. ### Phase 4 — roll out to sparks (window; ranks stop) Repeat per node in rank order, one node at a time, verifying before moving on. 1. Stop the group (needs a working controller; see the re-pair note in `flare-bringup.md`): ```bash flare orbit stop --cluster sparks ``` 2. Move the node's small artefacts to NAS and bind them back: ```bash V=gb10-tp4-; M=checkpoint-expanded-scales D=/home/kilork/.cache/flare/models/$M/$V N=/mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V sudo mkdir -p "$N" rsync -aH --remove-source-files "$D/model.safetensors" "$N/" rsync -aH "$D/deployments/" "$N/deployments/" # cp -a/rsync, never cp -al: NAS↔NAS hardlinks can EXDEV # NOTHING here may run while a bind is mounted at the target — rm deletes through it: findmnt -rno TARGET | grep -F "$D" | sort -r | xargs -r -n1 sudo umount rm -rf --one-file-system "$D/deployments" sudo mount --bind "$N/deployments" "$D/deployments" ``` Persist it in `/etc/fstab` (bind mounts are ordinary fstab entries and are mounted before the agent starts): ``` /mnt/nas/ai/flare/nodes/dgx5/checkpoint-expanded-scales/gb10-tp4-/deployments /home/kilork/.cache/flare/models/checkpoint-expanded-scales/gb10-tp4-/deployments none bind,nofail,x-systemd.requires=/mnt/nas,x-systemd.requires-mounts-for=/mnt/nas 0 0 ``` 3. Start the group and verify on the node: ```bash findmnt -no TARGET,SOURCE ~/.cache/flare/models/$M/$V/deployments du -sh ~/.cache/flare/models/$M/$V/* | sort -h # weights ~260G, rest tiny pgrep -af "[f]lare run" | head -1 ``` 4. Only then move to the next node. Gate: ranks serve, `findmnt` shows the bind mounts, `du` shows ~260 GiB of `weights/` and nothing else of size. ### Phase 5 — prod4 (dgx1–dgx4), same steps Same procedure with `--cluster prod4`. dgx1 is rank 0 and is the one that gains the 477 GiB; it is also the node whose `packages/` phase 2 removed. ### Phase 6 — verification and rollback Verification, per node: ```bash du -sh ~/.cache/flare/models/*/*/* # weights dominates findmnt -no TARGET,SOURCE | grep deployments df -h / /mnt/nas flare orbit status --cluster sparks # agents reachable, groups listed curl -s localhost:/v1/models | head # API answers (port from the rank cmdline) ``` Rollback (per node, no data loss — the NAS copy is authoritative): ```bash sudo umount ~/.cache/flare/models/$M/$V/deployments sudo sed -i "\|$N/deployments|d" /etc/fstab rsync -aH /mnt/nas/ai/flare/nodes/$(hostname -s)/$M/$V/deployments/ ~/.cache/flare/models/$M/$V/deployments/ flare orbit run --cluster sparks ``` ## Runtime parameters (2026-11-11) Decision: **take the launch parameters from the current `gb10` package**, not from what the lane happens to run today — with one deliberate exception, **the API port stays 30000** (every lane here serves on 30000; `8000` in their example is just their host convention). Their recipe vs. the live rank 0 command line on dgx1 (`tr '\0' ' ' < /proc/$(pgrep -f "[f]lare run" | head -1)/cmdline`): | flag | upstream `gb10` | live | action | |---|---|---|---| | `--max-concurrency` | `32` | `32` | keep | | `--mtp-draft-tokens` | `5` | `5` | keep | | `--mtp-adaptive-verification` | yes | yes | keep | | `--mtp-adaptive-verification-alpha` | **`0.75`** | **absent** | **added to both recipes** | | `--embedding-table-storage` | `disk` | `disk` | keep | | `--embedding-table-resident-scales` | yes | yes | keep | | `--embedding-table-cache-size` | `1GiB` | `1073741824` | equivalent (both = 1 GiB) | | `--transfer-streams 1` | absent | `1` | **keep ours** — RoCE workaround, `flare-roce-poison.md` | | `--host` | `0.0.0.0` | `0.0.0.0` | keep | | `--port` | `8000` | **`30000`** | **keep 30000** | | `--served-model-name deepseek-v4.1-flash` | absent | set | keep ours (client-facing name) | `alpha 0.75` is theirs on purpose: *“favor longer verification rounds, which raises decode throughput with four or more concurrent requests and coding throughput”* — i.e. it helps exactly the lane shape we run (c32, TP4). They also document memory/KV knobs (`--memory-limit=-12G`, `--kv-cache-limit memory|context|concurrency`) and recommend **leaving both at the default `memory`**: *“Every GiB reserved or cached away from the default reduces the KV capacity that Flare can plan.”* Their stated budget — ~11.4 GiB KV per GPU, ~13.1 M tokens, 17–18 GiB of 121 GiB left free — matches what we measure, so no change. The 1 GiB Engram row cache is their recommended default; they suggest 4–8 GiB only if memory is spare and the startup `Plan cache` line confirms room, and our own 48 GiB experiment failed with *“selected concurrency and verification width exceed available memory”* — so 1 GiB stays. ## 5. Risks and their fallbacks | risk | detection | fallback | |---|---|---| | FLARE replaces `deployments/` by rename; bind mount lost | `findmnt` empty after a stage | **Option D**: deployments local, only the package store moves | | NAS stall at boot | node boots with `/mnt/nas` missing | `nofail` (phase 1) keeps boot going; agent then writes deployments locally — re-sync after recovery | | NAS outage during serving | reads hang (`hard` mount) | bind mounts are only touched at load; keep `timeo=100,retrans=3`; do not put `staging/` on NAS | | hardlink-blind copy duplicates the package | `du` doubles after a copy | always `rsync -aH` | | mixed flare binaries after a restart | rendezvous `incompatible protocol` | phase 0 rebuild first, restart all ranks together | ## 6. What must be measured, not assumed 1. Whether the agent's `--package-cache` path can be redirected by config or needs a launcher `extra_flags` change (phase 2 note). **Partly answered 2026-11-11, and it is worse than assumed:** on dgx1 the CLI's configured storage is `/home/kilork/flare-pkgs/gb10`, an **empty 68 K skeleton** — `flare models list` says *“No local model versions found”* and `flare models inspect` cannot see the live package id — while the real 477 G store is in `~/.cache/flare/packages` because rank 0 is launched with `--package-cache /home/kilork/.cache/flare`. So redirection *works*, but CLI and agent are already pointed at different trees on one node. Unify them (launcher `local_dir` + agent flag) **before** either points at the NAS, or the NAS store will inherit the same split-brain. 2. Whether FLARE tolerates the bind mount across a real stage (phase 3 gate). 3. ~~Real NAS read throughput~~ **measured 2026-11-11** (`dd iflag=direct`, 2 GiB, so no page-cache lying): the NAS is **10 GbE, not 1 GbE**. | node | write 2 GiB | read 1 MiB | read 8 KiB | |---|---|---|---| | dgx8 (v4.1) | 659 MiB/s | 731 MiB/s | 28 MiB/s | | dgx1 (v4.0) | 663 MiB/s | 822 MiB/s | 32 MiB/s | So a 1.8 GiB `deployments/` directory reads in **2–3 s**, and the protocol choice costs nothing. The real constraint is small-file/short-read work: at 8 KiB the same share runs at ~30 MiB/s, so a `deployments/` split into many small files would load ~25× slower than the bulk figure suggests — count the files in a rank deployment before trusting the 2–3 s number, and keep `weights/` local regardless. 4. **Package revision currency must be verified against the source before any storage work** (2026-11-11: the fleet served a superseded Nov-3 build whose source repo now 404s — see Phase 2R). To read what a local package *is*, do not trust the tree layout; read the identity out of the payload (the `flare.manifest` tensor is protobuf, but the strings are legible): ```bash strings -a /model.safetensors | grep -aoE "https://huggingface.co/[^ ]*" | sort -u ``` and compare with `https://huggingface.co/api/models//commits/`. The HF API masks a *missing* repo as 401 when unauthenticated, so always re-check with a token before concluding a repo is gated. A valid token is `export HF_TOKEN=…` in `~/.zshenv` **on the nodes** — there is none on the mac. 5. **`flare-inference` must be the latest release and actually running on every node.** Measured 2026-11-11: on dgx1/2/3/4/8 `Installed == Candidate == 0.1.0~git20261105002735.0009b89d-1` and `/usr/bin/flare` is the same file (`03c8e463`) everywhere — but dgx1/2/3/8 *execute* deleted inodes (the PR #184 binaries), so only dgx4 is truly on the release build. The package's `≥ 20261105002735` requirement makes this a hard gate: **no node can load the current `gb10` revision until it restarts onto the packaged binary.** PR #184 is **not** in the release (`strings -a /usr/bin/flare | grep -c "embedding read worker panicked"` → 0), so running the latest build means giving up the patched Engram reader until it is merged upstream. 6. **Node root disks are not uniform** — dgx1 is **915 GiB**, dgx4 is 1.8 TiB. dgx1 was the one at 98 % full; a plan sized from dgx4's disk would have missed the whole reason this migration exists. 7. Package stores across dgx1–dgx3 share one id (`7cf40bec…-35842073`), and the layout here is **flat**: `du -l` reports **0 hardlinks / 184 files**, so `rsync -H` was never load-bearing on this fleet.