RALPH_PROGRESS.md 23 KB

Progress

Round 1 (2026-11-11) — Phase 0.2–0.4 + Phase 1 on dgx1/2/3/4/8

Done and verified.

  • Phase 1 applied on dgx1, dgx2, dgx3, dgx4, dgx8: NAS entry rewritten to rw,hard,nofail,x-systemd.mount-timeout=30,…,timeo=100,retrans=3,… with a backup /etc/fstab.bak-<date>-<time> on each node. Verified live (timeo=100 in findmnt), sudo mount -a returns 0 without remounting. Serving untouched: rank PIDs kept their 7-day uptime on dgx1/2/3/8 and dgx1:30000/v1/models still answered afterwards.
  • dgx8 NAS mount restored — it was in fstab but had never been mounted.
  • dgx1: no reboot was needed. The two stale NFS client IDs were cleared with a plain (non-lazy) umount while it had zero open files. Then the real finding: dgx1 cannot mount NFSv4.1 at all — a fresh vers=4.1 mount hangs with one clean client and with nconnect=1, while dgx2/3/4/8 mount 4.1 instantly. Its fstab is pinned to vers=4.0 (with the new timeouts). Do not "fix" it back to 4.1.
  • Phase 0.3 state capture for dgx1/2/3/4/8: on each node at /tmp/flare-before-gx10-dgxN.txt and copied to /mnt/nas/ai/flare/. Includes the exact rank launch cmdline (recovery recipe while the orbit controller is unpaired).
  • Open item 6.3 measured: the NAS is 10 GbE, not 1 GbE — 659–822 MiB/s at 1 MiB reads, but only ~30 MiB/s at 8 KiB. So a 1.8 GiB deployments/ reads in 2–3 s; many-small-file deployments would load ~25× slower. v4.0 vs v4.1: no measurable difference.

Objective file updated: added ## Milestones, ## Round plan, ## Checks (the check passes on all five powered nodes: fstab=1 live=1), rewrote Phase 1 (the original blanket sed would have silently forced 4.1 onto dgx1 and hung it; and nofail never shows up in live options, so verify with timeo=100), and filled in §6.3.

Next round

R2 (Q1): Phase 2 first half — rsync -aH ~/.cache/flare/packages/ /mnt/nas/ai/flare/store/ from dgx1 (rank 0, the node holding the 477 GiB). At the measured ~660 MiB/s write that is ~12 minutes, so run it in the foreground-ish with --info=progress2 and then verify with flare models list --model-cache /mnt/nas/ai/flare/store. Do not delete the local copy in the same round — that is R3, gated on the models list result.

Watch out for:

  • Never echo 3 > /proc/sys/vm/drop_caches on a serving node — the 260 GiB of weights are served from page cache. Use dd iflag=direct for any storage test.
  • dgx5/6/7 are off; they still need Phase 1 when they come up (the ## Checks command reports them as off).
  • R4 (PR #184 rebuild) needs dgx5 powered on and must precede any agent restart.

2026-11-11 — R2 (flare-storage-nas): purge instead of migrate; what we serve is superseded

Goal quadrant: impact × effort — delete 783 GiB of dead data and find out whether the fleet is even on the current model build. Both landed.

The operator rejected migrating the old cache (“*Syncing those old flare cache to store does not make much sense honestly*”) and was right on the facts, harder than expected:

  • Every node held the Nov-2 build checkpoint-expanded-scales/gb10-tp4-202611021936 (261 GiB × 3 nodes) with nothing holding it open — verified per node via /proc/<rank>/fd, /proc/<rank>/maps and lsof +D before deleting. Freed 783 GiB: dgx1 98 % full → 283 G free, dgx2 → 467 G, dgx3 → 470 G. Rank PIDs untouched (7-day uptime), dgx1:30000/v1/models still answering.
  • dgx1's root is 915 GiB, not 1.8 TB — it was 22 G from full. Plans sized on dgx4's disk miss this.
  • dgx1's package_cache had silently drifted to an empty ~/flare-pkgs/gb10 skeleton: flare models list said “No local model versions found”, while the live rank used ~/.cache/flare. Fix that before anything points at the NAS.
  • The started rsync (105 GiB in) was killed and the partial NAS copy deleted; Phase 2 is marked SUPERSEDED, do not run.

The real finding: the fleet serves a superseded package from a repo that no longer exists. The manifest inside model.safetensors names …/DeepSeek-V4.1-Flash-1017-FLARE2 → 404 with a valid token (401 unauthenticated is HF masking a missing repo). Current source: …/DeepSeek-V4.1-Flash-1017-FLARE, revision = branch gb10, last commit 2026-11-05 “Publish the GB10 TP4 package with vocabulary-parallel sampling and an FP8 Markov head”, 187 files / 474.7 GiB, and it requires flare-inference ≥ 0.1.0~git20261105002735.0009b89d-1.

Consequences recorded in the plan (Phase 2R, ## Runtime parameters, §6 items 4–7):

  • PR #184 rebuild dropped. It is not in the release (marker string → 0), and it predates the package's minimum runtime → functionally unusable for the current package, however fast. Prod = release build, everywhere. Installed+identical already (03c8e463, Installed == Candidate on dgx1/2/3/4/8), but dgx1/2/3/8 execute deleted inodes — only dgx4 truly runs the release; the restart window fixes that.
  • Launch parameters taken from the package's own README, port stays 30000 (operator). Only delta in practice: --mtp-adaptive-verification-alpha 0.75 added to both dsv41-flash-1017-4x-flare-gb10* recipes. --transfer-streams 1 (RoCE poison workaround) and --served-model-name stay. --memory-limit / --kv-cache-limit: upstream says keep the memory default, so nothing.
  • Next (R3): flare download … --revision gb10 into the NAS store (474.7 GiB), verify with flare models list --model-cache, then a window to cut over and delete the superseded 477 G on dgx1 + 261 G dirs on the ranks.

Skill updated (~/.agents/skills/flare): rule 1 now checks readlink /proc/<pid>/exe for (deleted) — hash equality does not prove a node runs the installed build; rule 2 reversed to “prod runs the latest release build”, with the package-gates-runtime reason; rule 5 adds “keep the flag set aligned with the package README, deviate only where documented”.

2026-11-11 — R3 (flare-storage-nas): the runtime is the blocker, not the disk

Goal quadrant: impact × effort — acquire the current gb10 package onto the NAS store (R3). The download is running; the surprise is that it could not have been run with the runtime we had installed.

The Nov-5 apt build cannot acquire the current package. With /usr/bin/flare (0.1.0~git20261105002735.0009b89d-1, installed == candidate on all five powered nodes) the acquisition sits at Verify package, 0 B for minutes: it holds store/locks/entries/checkpoint-expanded-scales--gb10-tp4-…lock and store/locks/staging/<pid>-0.lock and makes one TLS handshake after another to 13.36.183.231 (HF's eu-west-3 LFS CDN) with rx_bytes flat. The same command with upstream's tagged binary streams instantly. So the operator's instinct was right: 1105 is too old — and this is functional, not a lost optimisation.

Channels, established rather than assumed: github.com/NVIDIA/spark-local-inference → 301 → NVIDIA/hyperflare-inference (branch main, pushed 2026-11-11, 801 ★); upstream publishes release archives from version tags (commit 94fbce5f6), tag v0.1.0 (2026-11-11T19:31Z) with flare-linux-arm64.tgz = 11,427,812 B, sha256 7ba5ba43…, unpacking to a 26,054,848 B flare. The APT repo lags: after a clean apt-get update, apt-cache madison flare-inference offers only 20261105002735 / 20261030013211 / 20261027170349. Source build stays the backup path. PR #184 is still not merged (embedding read worker panicked marker → 0 in v0.1.0), which independently confirms dropping it.

What landed:

  • Acquisition in flight on dgx4 (detached setsid nohup, v0.1.0 binary): flare download … --revision gb10 --tensor-parallel 4 --model-cache /mnt/nas/ai/flare/store --publisher-keys … --token-file … → 55–78 MiB/s into the NAS store, 14 GiB by 22:10 (160 s). Upstream's example needs three things it does not document: --tensor-parallel 4 (default is the local GPU count → package exports no deployment with tensor-parallel size 1), non-interactive publisher trust (flare download has no --dangerously-* flags: it wants --publisher-keys + --trust-store; copied dgx1's key c8215a75… and the fleet-uniform trust.toml 26e5ae90…), and --token-file for the LFS transport.
  • v0.1.0 staged identically on dgx1/2/3/4/8 at /usr/local/share/hyperflare/v0.1.0/flare — each node re-downloaded and checked the archive sha256; resulting binary identical on all five (e3fe697e…, 26,054,848 B). /usr/bin/flare and ~/.local/lib/flare-agent/flare untouched, so nothing serving was touched. Documented the actual swap for the window (/usr/bin/flare is not what runs — the agent executes ~/.local/lib/flare-agent/flare).
  • SGLang stopped on dgx4 (the Ripple eval server): 103 GiB → 4 GiB RAM used, 116 GiB available, port 8010 closed, no leftovers. Weights kept on disk (54 G + 7.6 G). Note: pkill -f sglang killed my own ssh session (the pattern matches the remote command line) — always self-exclude.
  • Added a ## Checks runtime probe: staged=e3fe697e installed=03c8e463 running=…flare-agent/flare (deleted) on all four serving nodes, running= empty on dgx4 (no rank) — exactly the three-way state the milestone needs to close.
  • Recorded that only rank 0 serves the API (peer curl :30000 → 000 is normal), so dgx2 api=000 is not an outage.

Objective file: Phase 0.1 corrected with the channel table and the measured hang; Phase 2R rewritten with the command that actually works plus the three undocumented requirements; milestones/round plan updated; ## Checks gained the runtime probe. Skill: SKILL.md rule 2 now ranks channels (GitHub tags → APT (lagging floor) → source); references/build.md gained “Distribution channels (2026-11-11)” with the staging recipe; references/troubleshooting.md gained the acquisition section (verify hang, --tensor-parallel 4, publisher trust, token file, pkill self-match, rank-0-only API).

2026-11-11 — R4 (flare-storage-nas): v0.1.0 renamed the artefact; the CLI store is on the NAS

Goal quadrant: impact × effort — close R3's verification and do R4 (the config drift) while the 475 GiB acquisition streams. R4 is done and verified; R3's gate is still streaming (see below).

The runtime we are moving to speaks a different vocabulary, measured not assumed. Side by side on dgx4 (/usr/bin/flare Nov-5 vs the staged v0.1.0): “prepared package” → prepared “core”, --package-cache → --core-cache (on run, orbit run, models list), default store root ~/.cache/flare/packages → ~/.cache/flare/cores, and flare download keeps --model-cache (the two flags coexist — do not “fix” one into the other). But paths.package_cache is unchanged: with that key set, both binaries report the configured path from models list. And nothing migrates automatically — neither binary has a migrat* string for core/package. So that one config key is what prevents an upgrade from orphaning hundreds of GB.

R4 applied: package_cache = "/mnt/nas/ai/flare/store" on dgx4 (acquiring node) and dgx1 (prod rank 0), replacing the value that pointed at /home/kilork/flare-pkgs/gb10 — a 68 K empty skeleton. Backups config.toml.bak-2026-11-11-2214-nas-store (dgx4) / …-2314-nas-store (dgx1). Verified: flare models list → Storage /mnt/nas/ai/flare/store, and dgx1's rank is untouched — same PID 533700, 7.5-day uptime, api=200 — because the rank's cache path comes from the agent's explicit flag, not this key. The old value was deliberate (it equalled the recipe's local_dir so the rsync mirror saw src == dst), but the mirror never filled, so the key was pointing at nothing; retire local_dir in the Phase 5 window, not now.

The launcher needs no change for the flag rename — grep -rn "package-cache" src/ finds only a comment; the agent synthesises the flag, so it travels with the binary swap. Still to confirm in the window: the launched cmdline itself (open item 6.1).

Store mechanics learned mid-flight, which change the verification step: at 39 GiB downloaded, all bytes sit in <store>/transport/ — models/, blobs/, refs/ are still empty skeletons, and staging/trees/<pid>-<n> has one tree per attempt (the killed Nov-5 attempt is still there). So materialisation happens only after the whole payload is cached, expect a peak near 2× the payload, and flare models gc is the tool for it — v0.1.0's flare models gained download · downloads · select · remove · gc, i.e. real store management (models remove/gc beat rm -rf, models select pins a default version). flare trust also exists now.

Node clocks disagree by an hour (dgx1 23:14 vs dgx4 22:14 at the same instant) — never compare timestamps across nodes.

Acquisition state at last check: 38.1 GiB at 420 s, 93 MiB/s, alive, one process, --tensor-parallel 4, target ≈ 23:32 UTC. R3's gate (models list showing the version) is the next round's first action if it does not land this round.

Objective file: new “v0.1.0 vocabulary and paths” section (side-by-side table + the five consequences that bite), “R4 — CLI store pointed at the NAS store” (applied value, backup paths, verification, why the old value existed), R4 ticked. Skill references/storage.md: same vocabulary table, the no-migration warning, the agent synthesises-the-flag note, the models subcommands, and the transport-first layout with its 2× peak.

2026-11-12 — R4 close (flare-storage-nas): the current gb10 core is on the NAS and verified

R3's gate passed. The acquisition that started 22:07 UTC finished at 00:16 UTC (7682 s, avg 63 MiB/s, peak 104) and flare models list now reports, from both dgx4 and dgx1 (prod) over NFS:

checkpoint-expanded-scales  gb10-tp4-202611112207  [DEFAULT]
Format payload v2 · Disk used 474.66 GiB · Shared files 474.63 GiB · 181 hardlinked objects
Deployment db20209d361f473b8452f24ad517fa0d · TP 4 · ready
Context / prefill 1,048,576 / 8,192 tokens · Pages / state rows 128 / 448

models inspect confirms identity: DeepSeek-V4.1-Flash, local-inference-lab (signatures verified), sm_121, vocab 129,280, concurrency 32 · 4 ranks, speculation mtp · widths 1–6.

Two numbers in the plan were wrong and are now measured. (1) The store does not keep a 2× copy: transport/ peaked at 475 G while blobs/ grew beside it, and at the end transport/ is 88 K and blobs/ is the whole 475 G — so no models gc was needed, and the 2× warning only applies mid-run. (2) A rank slice is three files — initial-state.safetensors 1,321,056,820 B + kernels.safetensors 39,429,126 B + graphs.safetensors 4,207,295 B = 1.3 GiB each, deployments/ 5.1 G total, weights/ 470 G, model.safetensors 33 M at mode 0444. At our measured NAS read rates that is ~2 s per rank, so the small-file worry in §6 item 3 simply does not apply to this package — the bind-mount phase is viable on the numbers, not by hope.

Serving survived the 2 h of NAS writes untouched: dgx1/2/3/8 all ranks=1, dgx1 api=200, uptime continuing, every mount still timeo=100.

Milestones restructured honestly: acquisition done, deployment of the core + deletion of the superseded build still open (it needs the window — dgx1's 477 G and the ranks' 261 G dirs are what is actually serving).

Next: R6 (Phase 3 gate) on dgx4 — now unblocked, with real paths in the doc; then the window (R7/R8) which is also what lands v0.1.0 as the running binary and frees the superseded 477 G.

2026-11-12 — R5 round (flare-storage-nas): the bind-mount design works, and I broke a core while testing it

Round goal was R6 (Phase 3 gate) on idle dgx4. Half of it is now proven, and the plan gained its most important safety rule.

Proven on dgx4 with the staged v0.1.0 binary, using the real Phase 4 shape (per-node artefacts under nodes/gx10-dgx4/<model>/<version>/deployments/<id>/ on the NAS, bind-mounted over the model directory's deployments/ inside a store root): flare run --stage-only passed Verify core and reached Wait for tensor-parallel hosts, and the bind was byte-identical before and after five stage attempts (same inode, findmnt still showing the NAS source). The inventory layer is bind-safe too: models list over a locally-assembled root reported 474.66 GiB · 181 hardlinked objects while blobs/ and weights/ were bind mounts onto a different device (NFS st_dev 80 vs local NVMe 66306). strace explains the symlink ban precisely: the escape check is openat(7, "tp4", O_RDONLY|O_NOFOLLOW|O_DIRECTORY).

The plan's own Phase 3 procedure was invalid and is rewritten. Copying model.safetensors + deployments out of a store and pointing run/inspect at the copy fails in v0.1.0 with open core directory: No such file or directory (os error 2) — the envelope resolves its core within the store that contains it, and a store has no metadata files at all (purely structural), so the model path must live inside a coherent store root with blobs/ refs/ sources/ origins/ staging/ locks/ present. The verified command shape is in the doc.

Then I destroyed the core's materialisation. Cleanup ran rm -rf over a tree that still had two live bind mounts into the NAS store. rm unmounts nothing: the rmdir of the mount points failed EBUSY — which reads like a refusal — while every file inside them was deleted on the NAS. models list went to Disk used 32.4 MiB. No data was lost: the store is content-addressed, so blobs/ still held the whole 475 G (link count 1) and nodes/gx10-dgx4/ still held all four deployment slices. Recovery is flare download into the same store, which re-links from local blobs; it is running now (pid 2037594, ~/flare-rematerialise.log). There is no verify/repair subcommand — download is the repair path.

Three false alarms that look like corruption, all documented so the next round does not chase them: du -sh a b dedups hardlinks across arguments (that is how blobs read as 6.2M before I had actually broken anything); NFS actimeo=600 dir caching made a live rank-0 look missing while binds were stacked; and cp -al between two NAS paths fails Invalid cross-device link — per-node artefacts need cp -a.

Rule added to the skill as hard rule 6 (unmount deepest-first before recursive deletes, rm -rf --one-file-system, then blob-first triage), plus a store-triage section in references/troubleshooting.md and a validated bind-layout section in references/storage.md. R6 is split: R6a done, R6b needs a completed TP4 stage (peers ⇒ the window) to prove deployment reads traverse the bind and that FLARE never renames over / hardlinks into deployments/ — either would take option D.

Serving untouched all round: dgx1/2/3/8 ranks=1, dgx1 api=200. dgx4 has 1.5 T free and no ranks, and now holds the first real nodes/<node>/… artefacts (5.2 G).

Second half of the same round: R5 landed, and the window now has a tool

R5 is done, and it is not just paperwork — it produced the pre-cutover runtime table and the mechanism behind the fleet's mystery:

dgx1  staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
dgx2  staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
dgx3  staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
dgx4  staged=e3fe697e /usrbin=e3fe697e agent=e3fe697e exe=/home/kilork/.local/lib/flare-agent/flare
dgx8  staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)

Every serving node runs a (deleted) inode, i.e. production is running neither the Nov-5 apt build nor v0.1.0 — it is the pre-apt (PR-184) binary held alive by the process. The reason apt never fixed this is now proven instead of inferred: the unit ExecStarts %h/.local/lib/flare-agent/flare and dpkg -S owns no such file, so every apt upgrade moved /usr/bin/flare and left the running path alone. The ## Checks probe prints agent= for exactly this reason — it is the only column describing what executes.

Then the swap was rehearsed on idle dgx4 (temp+mv at both paths, systemctl --user restart flare-agent.service, sha + readlink /proc/PID/exe verification, flare.orig-20261112 backups) and turned into host-scripts/flare-swap.sh so the window runs one reviewed command per node instead of hand-typed lines. Its two tests both behaved: on dgx4 it was a verified no-op with exit 0; on dgx8, with a rank running, it refused (exit 1) leaving shas unchanged, zero tmp-swap files behind, and the rank untouched. That refusal is the behaviour that protects prod during a window.

A trap for the window, found by reading the numbers again: the new core's weights/ is 470 G, not the 260 GiB this plan was written against. dgx1's root is 915 G with 283 G free, and its 477 G package store is only released when rank 0 restarts — so dgx1 comes to ~760 G free during the window, and if a rank turns out to need the full 470 G locally, that is the only node where the arithmetic is close. Hence a new gate: measure the real per-rank footprint on dgx4 (1.5 T free) before rolling sparks/prod4, and re-measure the free-space table the day of the window.

Repair of my own damage is still running healthily on dgx4 (pid 2037594, read_bytes +370 MB/s against a 475 G blob re-verification, NIC idle — the progress bar counts network bytes only, which is why it reads 32.4 MiB the whole time). When it lands, models list returns to 474.66 GiB.

Recovery confirmed inside the same round. The repair finished at 1251 s: network 33,966,105 bytes · local source 509,653,222,270 · verification 509,646,855,013 — i.e. 34 MB re-downloaded, 509.6 GB re-read from local blobs. The store now shows 1 model · 1 version, [DEFAULT] gb10-tp4-202611120031, 474.66 GiB, 181 hardlinked objects, deployment db20209d361f473b8452f24ad517fa0d · TP 4 · ready. Two durable facts: version ids are mutable (a re-materialisation mints a new one, so scripts must query rather than hardcode) and deployment ids are content-derived and stable (every reference to db20209d… in the plan stayed valid). flare models remove wants MODEL@VERSION, not MODEL/VERSION. Per-node artefacts were rebuilt for the new version (5.1 G). NAS at 1.6 T/33 T. Serving untouched throughout: dgx1/2/3/8 ranks=1, dgx1 api=200.

Loop log

  • round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…
  • round 2 · -- · 74 turns · continue → continue · (no ROUND line)
  • round 3 · -- · 36 turns · continue → continue · outcome
  • round 4 · -- · 28 turns · continue → continue · outcome
  • round 5 · -- · 54 turns · continue → continue · validate the bind-mount model on idle dgx4 — Phase 3 gate…