RALPH_PROGRESS.md 22 KB

Progress

Round 1 (2026-11-11) — Phase 0.2–0.4 + Phase 1 on dgx1/2/3/4/8

Done and verified.

  • Phase 1 applied on dgx1, dgx2, dgx3, dgx4, dgx8: NAS entry rewritten to rw,hard,nofail,x-systemd.mount-timeout=30,…,timeo=100,retrans=3,… with a backup /etc/fstab.bak-<date>-<time> on each node. Verified live (timeo=100 in findmnt), sudo mount -a returns 0 without remounting. Serving untouched: rank PIDs kept their 7-day uptime on dgx1/2/3/8 and dgx1:30000/v1/models still answered afterwards.
  • dgx8 NAS mount restored — it was in fstab but had never been mounted.
  • dgx1: no reboot was needed. The two stale NFS client IDs were cleared with a plain (non-lazy) umount while it had zero open files. Then the real finding: dgx1 cannot mount NFSv4.1 at all — a fresh vers=4.1 mount hangs with one clean client and with nconnect=1, while dgx2/3/4/8 mount 4.1 instantly. Its fstab is pinned to vers=4.0 (with the new timeouts). Do not "fix" it back to 4.1.
  • Phase 0.3 state capture for dgx1/2/3/4/8: on each node at /tmp/flare-before-gx10-dgxN.txt and copied to /mnt/nas/ai/flare/. Includes the exact rank launch cmdline (recovery recipe while the orbit controller is unpaired).
  • Open item 6.3 measured: the NAS is 10 GbE, not 1 GbE — 659–822 MiB/s at 1 MiB reads, but only ~30 MiB/s at 8 KiB. So a 1.8 GiB deployments/ reads in 2–3 s; many-small-file deployments would load ~25× slower. v4.0 vs v4.1: no measurable difference.

Objective file updated: added ## Milestones, ## Round plan, ## Checks (the check passes on all five powered nodes: fstab=1 live=1), rewrote Phase 1 (the original blanket sed would have silently forced 4.1 onto dgx1 and hung it; and nofail never shows up in live options, so verify with timeo=100), and filled in §6.3.

Next round

R2 (Q1): Phase 2 first half — rsync -aH ~/.cache/flare/packages/ /mnt/nas/ai/flare/store/ from dgx1 (rank 0, the node holding the 477 GiB). At the measured ~660 MiB/s write that is ~12 minutes, so run it in the foreground-ish with --info=progress2 and then verify with flare models list --model-cache /mnt/nas/ai/flare/store. Do not delete the local copy in the same round — that is R3, gated on the models list result.

Watch out for:

  • Never echo 3 > /proc/sys/vm/drop_caches on a serving node — the 260 GiB of weights are served from page cache. Use dd iflag=direct for any storage test.
  • dgx5/6/7 are off; they still need Phase 1 when they come up (the ## Checks command reports them as off).
  • R4 (PR #184 rebuild) needs dgx5 powered on and must precede any agent restart.

2026-11-11 — R2 (flare-storage-nas): purge instead of migrate; what we serve is superseded

Goal quadrant: impact × effort — delete 783 GiB of dead data and find out whether the fleet is even on the current model build. Both landed.

The operator rejected migrating the old cache (“*Syncing those old flare cache to store does not make much sense honestly*”) and was right on the facts, harder than expected:

  • Every node held the Nov-2 build checkpoint-expanded-scales/gb10-tp4-202611021936 (261 GiB × 3 nodes) with nothing holding it open — verified per node via /proc/<rank>/fd, /proc/<rank>/maps and lsof +D before deleting. Freed 783 GiB: dgx1 98 % full → 283 G free, dgx2 → 467 G, dgx3 → 470 G. Rank PIDs untouched (7-day uptime), dgx1:30000/v1/models still answering.
  • dgx1's root is 915 GiB, not 1.8 TB — it was 22 G from full. Plans sized on dgx4's disk miss this.
  • dgx1's package_cache had silently drifted to an empty ~/flare-pkgs/gb10 skeleton: flare models list said “No local model versions found”, while the live rank used ~/.cache/flare. Fix that before anything points at the NAS.
  • The started rsync (105 GiB in) was killed and the partial NAS copy deleted; Phase 2 is marked SUPERSEDED, do not run.

The real finding: the fleet serves a superseded package from a repo that no longer exists. The manifest inside model.safetensors names …/DeepSeek-V4.1-Flash-1017-FLARE2 → 404 with a valid token (401 unauthenticated is HF masking a missing repo). Current source: …/DeepSeek-V4.1-Flash-1017-FLARE, revision = branch gb10, last commit 2026-11-05 “Publish the GB10 TP4 package with vocabulary-parallel sampling and an FP8 Markov head”, 187 files / 474.7 GiB, and it requires flare-inference ≥ 0.1.0~git20261105002735.0009b89d-1.

Consequences recorded in the plan (Phase 2R, ## Runtime parameters, §6 items 4–7):

  • PR #184 rebuild dropped. It is not in the release (marker string → 0), and it predates the package's minimum runtime → functionally unusable for the current package, however fast. Prod = release build, everywhere. Installed+identical already (03c8e463, Installed == Candidate on dgx1/2/3/4/8), but dgx1/2/3/8 execute deleted inodes — only dgx4 truly runs the release; the restart window fixes that.
  • Launch parameters taken from the package's own README, port stays 30000 (operator). Only delta in practice: --mtp-adaptive-verification-alpha 0.75 added to both dsv41-flash-1017-4x-flare-gb10* recipes. --transfer-streams 1 (RoCE poison workaround) and --served-model-name stay. --memory-limit / --kv-cache-limit: upstream says keep the memory default, so nothing.
  • Next (R3): flare download … --revision gb10 into the NAS store (474.7 GiB), verify with flare models list --model-cache, then a window to cut over and delete the superseded 477 G on dgx1 + 261 G dirs on the ranks.

Skill updated (~/.agents/skills/flare): rule 1 now checks readlink /proc/<pid>/exe for (deleted) — hash equality does not prove a node runs the installed build; rule 2 reversed to “prod runs the latest release build”, with the package-gates-runtime reason; rule 5 adds “keep the flag set aligned with the package README, deviate only where documented”.

2026-11-11 — R3 (flare-storage-nas): the runtime is the blocker, not the disk

Goal quadrant: impact × effort — acquire the current gb10 package onto the NAS store (R3). The download is running; the surprise is that it could not have been run with the runtime we had installed.

The Nov-5 apt build cannot acquire the current package. With /usr/bin/flare (0.1.0~git20261105002735.0009b89d-1, installed == candidate on all five powered nodes) the acquisition sits at Verify package, 0 B for minutes: it holds store/locks/entries/checkpoint-expanded-scales--gb10-tp4-…lock and store/locks/staging/<pid>-0.lock and makes one TLS handshake after another to 13.36.183.231 (HF's eu-west-3 LFS CDN) with rx_bytes flat. The same command with upstream's tagged binary streams instantly. So the operator's instinct was right: 1105 is too old — and this is functional, not a lost optimisation.

Channels, established rather than assumed: github.com/NVIDIA/spark-local-inference → 301 → NVIDIA/hyperflare-inference (branch main, pushed 2026-11-11, 801 ★); upstream publishes release archives from version tags (commit 94fbce5f6), tag v0.1.0 (2026-11-11T19:31Z) with flare-linux-arm64.tgz = 11,427,812 B, sha256 7ba5ba43…, unpacking to a 26,054,848 B flare. The APT repo lags: after a clean apt-get update, apt-cache madison flare-inference offers only 20261105002735 / 20261030013211 / 20261027170349. Source build stays the backup path. PR #184 is still not merged (embedding read worker panicked marker → 0 in v0.1.0), which independently confirms dropping it.

What landed:

  • Acquisition in flight on dgx4 (detached setsid nohup, v0.1.0 binary): flare download … --revision gb10 --tensor-parallel 4 --model-cache /mnt/nas/ai/flare/store --publisher-keys … --token-file … → 55–78 MiB/s into the NAS store, 14 GiB by 22:10 (160 s). Upstream's example needs three things it does not document: --tensor-parallel 4 (default is the local GPU count → package exports no deployment with tensor-parallel size 1), non-interactive publisher trust (flare download has no --dangerously-* flags: it wants --publisher-keys + --trust-store; copied dgx1's key c8215a75… and the fleet-uniform trust.toml 26e5ae90…), and --token-file for the LFS transport.
  • v0.1.0 staged identically on dgx1/2/3/4/8 at /usr/local/share/hyperflare/v0.1.0/flare — each node re-downloaded and checked the archive sha256; resulting binary identical on all five (e3fe697e…, 26,054,848 B). /usr/bin/flare and ~/.local/lib/flare-agent/flare untouched, so nothing serving was touched. Documented the actual swap for the window (/usr/bin/flare is not what runs — the agent executes ~/.local/lib/flare-agent/flare).
  • SGLang stopped on dgx4 (the Ripple eval server): 103 GiB → 4 GiB RAM used, 116 GiB available, port 8010 closed, no leftovers. Weights kept on disk (54 G + 7.6 G). Note: pkill -f sglang killed my own ssh session (the pattern matches the remote command line) — always self-exclude.
  • Added a ## Checks runtime probe: staged=e3fe697e installed=03c8e463 running=…flare-agent/flare (deleted) on all four serving nodes, running= empty on dgx4 (no rank) — exactly the three-way state the milestone needs to close.
  • Recorded that only rank 0 serves the API (peer curl :30000 → 000 is normal), so dgx2 api=000 is not an outage.

Objective file: Phase 0.1 corrected with the channel table and the measured hang; Phase 2R rewritten with the command that actually works plus the three undocumented requirements; milestones/round plan updated; ## Checks gained the runtime probe. Skill: SKILL.md rule 2 now ranks channels (GitHub tags → APT (lagging floor) → source); references/build.md gained “Distribution channels (2026-11-11)” with the staging recipe; references/troubleshooting.md gained the acquisition section (verify hang, --tensor-parallel 4, publisher trust, token file, pkill self-match, rank-0-only API).

2026-11-11 — R4 (flare-storage-nas): v0.1.0 renamed the artefact; the CLI store is on the NAS

Goal quadrant: impact × effort — close R3's verification and do R4 (the config drift) while the 475 GiB acquisition streams. R4 is done and verified; R3's gate is still streaming (see below).

The runtime we are moving to speaks a different vocabulary, measured not assumed. Side by side on dgx4 (/usr/bin/flare Nov-5 vs the staged v0.1.0): “prepared package” → prepared “core”, --package-cache → --core-cache (on run, orbit run, models list), default store root ~/.cache/flare/packages → ~/.cache/flare/cores, and flare download keeps --model-cache (the two flags coexist — do not “fix” one into the other). But paths.package_cache is unchanged: with that key set, both binaries report the configured path from models list. And nothing migrates automatically — neither binary has a migrat* string for core/package. So that one config key is what prevents an upgrade from orphaning hundreds of GB.

R4 applied: package_cache = "/mnt/nas/ai/flare/store" on dgx4 (acquiring node) and dgx1 (prod rank 0), replacing the value that pointed at /home/kilork/flare-pkgs/gb10 — a 68 K empty skeleton. Backups config.toml.bak-2026-11-11-2214-nas-store (dgx4) / …-2314-nas-store (dgx1). Verified: flare models list → Storage /mnt/nas/ai/flare/store, and dgx1's rank is untouched — same PID 533700, 7.5-day uptime, api=200 — because the rank's cache path comes from the agent's explicit flag, not this key. The old value was deliberate (it equalled the recipe's local_dir so the rsync mirror saw src == dst), but the mirror never filled, so the key was pointing at nothing; retire local_dir in the Phase 5 window, not now.

The launcher needs no change for the flag rename — grep -rn "package-cache" src/ finds only a comment; the agent synthesises the flag, so it travels with the binary swap. Still to confirm in the window: the launched cmdline itself (open item 6.1).

Store mechanics learned mid-flight, which change the verification step: at 39 GiB downloaded, all bytes sit in <store>/transport/ — models/, blobs/, refs/ are still empty skeletons, and staging/trees/<pid>-<n> has one tree per attempt (the killed Nov-5 attempt is still there). So materialisation happens only after the whole payload is cached, expect a peak near 2× the payload, and flare models gc is the tool for it — v0.1.0's flare models gained download · downloads · select · remove · gc, i.e. real store management (models remove/gc beat rm -rf, models select pins a default version). flare trust also exists now.

Node clocks disagree by an hour (dgx1 23:14 vs dgx4 22:14 at the same instant) — never compare timestamps across nodes.

Acquisition state at last check: 38.1 GiB at 420 s, 93 MiB/s, alive, one process, --tensor-parallel 4, target ≈ 23:32 UTC. R3's gate (models list showing the version) is the next round's first action if it does not land this round.

Objective file: new “v0.1.0 vocabulary and paths” section (side-by-side table + the five consequences that bite), “R4 — CLI store pointed at the NAS store” (applied value, backup paths, verification, why the old value existed), R4 ticked. Skill references/storage.md: same vocabulary table, the no-migration warning, the agent synthesises-the-flag note, the models subcommands, and the transport-first layout with its 2× peak.

2026-11-12 — R4 close (flare-storage-nas): the current gb10 core is on the NAS and verified

R3's gate passed. The acquisition that started 22:07 UTC finished at 00:16 UTC (7682 s, avg 63 MiB/s, peak 104) and flare models list now reports, from both dgx4 and dgx1 (prod) over NFS:

checkpoint-expanded-scales  gb10-tp4-202611112207  [DEFAULT]
Format payload v2 · Disk used 474.66 GiB · Shared files 474.63 GiB · 181 hardlinked objects
Deployment db20209d361f473b8452f24ad517fa0d · TP 4 · ready
Context / prefill 1,048,576 / 8,192 tokens · Pages / state rows 128 / 448

models inspect confirms identity: DeepSeek-V4.1-Flash, local-inference-lab (signatures verified), sm_121, vocab 129,280, concurrency 32 · 4 ranks, speculation mtp · widths 1–6.

Two numbers in the plan were wrong and are now measured. (1) The store does not keep a 2× copy: transport/ peaked at 475 G while blobs/ grew beside it, and at the end transport/ is 88 K and blobs/ is the whole 475 G — so no models gc was needed, and the 2× warning only applies mid-run. (2) A rank slice is three files — initial-state.safetensors 1,321,056,820 B + kernels.safetensors 39,429,126 B + graphs.safetensors 4,207,295 B = 1.3 GiB each, deployments/ 5.1 G total, weights/ 470 G, model.safetensors 33 M at mode 0444. At our measured NAS read rates that is ~2 s per rank, so the small-file worry in §6 item 3 simply does not apply to this package — the bind-mount phase is viable on the numbers, not by hope.

Serving survived the 2 h of NAS writes untouched: dgx1/2/3/8 all ranks=1, dgx1 api=200, uptime continuing, every mount still timeo=100.

Milestones restructured honestly: acquisition done, deployment of the core + deletion of the superseded build still open (it needs the window — dgx1's 477 G and the ranks' 261 G dirs are what is actually serving).

Next: R6 (Phase 3 gate) on dgx4 — now unblocked, with real paths in the doc; then the window (R7/R8) which is also what lands v0.1.0 as the running binary and frees the superseded 477 G.

Loop log

  • round 1 · -- · 27 turns · continue → continue · Phase 0.2–0.4 + Phase 1 — NAS mount boot-proof on every p…
  • round 2 · -- · 74 turns · continue → continue · (no ROUND line)
  • round 3 · -- · 36 turns · continue → continue · outcome
  • round 4 · -- · 28 turns · continue → continue · outcome

2026-11-12 — R5 round (flare-storage-nas): the bind-mount design works, and I broke a core while testing it

Round goal was R6 (Phase 3 gate) on idle dgx4. Half of it is now proven, and the plan gained its most important safety rule.

Proven on dgx4 with the staged v0.1.0 binary, using the real Phase 4 shape (per-node artefacts under nodes/gx10-dgx4/<model>/<version>/deployments/<id>/ on the NAS, bind-mounted over the model directory's deployments/ inside a store root): flare run --stage-only passed Verify core and reached Wait for tensor-parallel hosts, and the bind was byte-identical before and after five stage attempts (same inode, findmnt still showing the NAS source). The inventory layer is bind-safe too: models list over a locally-assembled root reported 474.66 GiB · 181 hardlinked objects while blobs/ and weights/ were bind mounts onto a different device (NFS st_dev 80 vs local NVMe 66306). strace explains the symlink ban precisely: the escape check is openat(7, "tp4", O_RDONLY|O_NOFOLLOW|O_DIRECTORY).

The plan's own Phase 3 procedure was invalid and is rewritten. Copying model.safetensors + deployments out of a store and pointing run/inspect at the copy fails in v0.1.0 with open core directory: No such file or directory (os error 2) — the envelope resolves its core within the store that contains it, and a store has no metadata files at all (purely structural), so the model path must live inside a coherent store root with blobs/ refs/ sources/ origins/ staging/ locks/ present. The verified command shape is in the doc.

Then I destroyed the core's materialisation. Cleanup ran rm -rf over a tree that still had two live bind mounts into the NAS store. rm unmounts nothing: the rmdir of the mount points failed EBUSY — which reads like a refusal — while every file inside them was deleted on the NAS. models list went to Disk used 32.4 MiB. No data was lost: the store is content-addressed, so blobs/ still held the whole 475 G (link count 1) and nodes/gx10-dgx4/ still held all four deployment slices. Recovery is flare download into the same store, which re-links from local blobs; it is running now (pid 2037594, ~/flare-rematerialise.log). There is no verify/repair subcommand — download is the repair path.

Three false alarms that look like corruption, all documented so the next round does not chase them: du -sh a b dedups hardlinks across arguments (that is how blobs read as 6.2M before I had actually broken anything); NFS actimeo=600 dir caching made a live rank-0 look missing while binds were stacked; and cp -al between two NAS paths fails Invalid cross-device link — per-node artefacts need cp -a.

Rule added to the skill as hard rule 6 (unmount deepest-first before recursive deletes, rm -rf --one-file-system, then blob-first triage), plus a store-triage section in references/troubleshooting.md and a validated bind-layout section in references/storage.md. R6 is split: R6a done, R6b needs a completed TP4 stage (peers ⇒ the window) to prove deployment reads traverse the bind and that FLARE never renames over / hardlinks into deployments/ — either would take option D.

Serving untouched all round: dgx1/2/3/8 ranks=1, dgx1 api=200. dgx4 has 1.5 T free and no ranks, and now holds the first real nodes/<node>/… artefacts (5.2 G).

Second half of the same round: R5 landed, and the window now has a tool

R5 is done, and it is not just paperwork — it produced the pre-cutover runtime table and the mechanism behind the fleet's mystery:

dgx1  staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
dgx2  staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
dgx3  staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
dgx4  staged=e3fe697e /usrbin=e3fe697e agent=e3fe697e exe=/home/kilork/.local/lib/flare-agent/flare
dgx8  staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)

Every serving node runs a (deleted) inode, i.e. production is running neither the Nov-5 apt build nor v0.1.0 — it is the pre-apt (PR-184) binary held alive by the process. The reason apt never fixed this is now proven instead of inferred: the unit ExecStarts %h/.local/lib/flare-agent/flare and dpkg -S owns no such file, so every apt upgrade moved /usr/bin/flare and left the running path alone. The ## Checks probe prints agent= for exactly this reason — it is the only column describing what executes.

Then the swap was rehearsed on idle dgx4 (temp+mv at both paths, systemctl --user restart flare-agent.service, sha + readlink /proc/PID/exe verification, flare.orig-20261112 backups) and turned into host-scripts/flare-swap.sh so the window runs one reviewed command per node instead of hand-typed lines. Its two tests both behaved: on dgx4 it was a verified no-op with exit 0; on dgx8, with a rank running, it refused (exit 1) leaving shas unchanged, zero tmp-swap files behind, and the rank untouched. That refusal is the behaviour that protects prod during a window.

A trap for the window, found by reading the numbers again: the new core's weights/ is 470 G, not the 260 GiB this plan was written against. dgx1's root is 915 G with 283 G free, and its 477 G package store is only released when rank 0 restarts — so dgx1 comes to ~760 G free during the window, and if a rank turns out to need the full 470 G locally, that is the only node where the arithmetic is close. Hence a new gate: measure the real per-rank footprint on dgx4 (1.5 T free) before rolling sparks/prod4, and re-measure the free-space table the day of the window.

Repair of my own damage is still running healthily on dgx4 (pid 2037594, read_bytes +370 MB/s against a 475 G blob re-verification, NIC idle — the progress bar counts network bytes only, which is why it reads 32.4 MiB the whole time). When it lands, models list returns to 474.66 GiB.