Browse Source

docs(progress): R6 round - bind design proven with a serving TP1 rank on one node; negative control replaces a traffic counter; option D off the table; fstab bind survives reboot; two of my own earlier conclusions corrected (store-root scaffolding, 2564 MiB)

Alexander Korolev 1 tuần trước cách đây
mục cha
commit
40a98e5400
1 tập tin đã thay đổi với 49 bổ sung và 0 xóa
  1. 49 0
      specs/tech-architecture/RALPH_PROGRESS.md

+ 49 - 0
specs/tech-architecture/RALPH_PROGRESS.md

@@ -361,3 +361,52 @@ dgx1 api=200.
 - round 3 · -- · 36 turns · continue → continue · outcome
 - round 4 · -- · 28 turns · continue → continue · outcome
 - round 5 · -- · 54 turns · continue → continue · validate the bind-mount model on idle dgx4 — Phase 3 gate…
+
+## 2026-11-12 — R6 round (flare-storage-nas): the bind design is proven end to end, on one node, without a window
+
+**The round's premise was that R6b needed the window. It did not.** `local-inference-lab/Qwen3.8-27B-FLARE`
+(branch `gb10`, 25.5 GB, 317 s at ~80 MiB/s) yields a **TP 1 · ready** core —
+`Qwen3.8-27B-NVIDIA-NVFP4-QAD-MXFP8 · gb10-tp1-…`, 23.77 GiB, weights 22 G +
+deployments 2.5 G + envelope 18 M — which let dgx4 run a **complete, serving rank** with
+`deployments/` bind-mounted from `nodes/<node>/<model>/<version>/deployments` on the NAS.
+`api=200` in 20 s warm, 21 s cold (page cache dropped on the idle node), and a chat
+completion came back (reasoning model: `content: null`, `reasoning_content` populated).
+
+The four things that mattered, each measured rather than assumed:
+
+- **Reads do traverse the bind** — the negative control is the proof, not the traffic
+  counter: bind a NAS dir containing no rank slice and the rank dies with
+  `Verify core … open core directory: No such file or directory`. Bind one that contains
+  `<id>/rank-N` and it serves. No hidden fallback to local disk.
+- **FLARE never touches the bind** — 0 files newer than the rank start inside it, inode
+  identical, mount still present after serving, after a restart, and after a reboot.
+  **Option D is off the table.**
+- **The NAS cost is a one-time load cost** — steady-state inference pulled **0 MiB** over
+  the wire; combined with the earlier 1.29 s / 1.1 GB/s read of a TP4 rank slice, a
+  NAS-backed `deployments/` is effectively free per start.
+- **fstab persistence works** — installed the exact Phase 4 stanza on dgx4, rebooted it
+  (idle, 0 ranks): bind present at boot, `systemctl show <bind-unit> -p Requires`
+  contains `mnt-nas.mount`, and a rank came up from it **22 s after boot**.
+
+**Two of my own earlier conclusions died this round.** (1) R5's claim that a model dir
+needs the store's `blobs/ refs/ staging/` scaffolding was a **misattribution**: the
+`open core directory` error means `deployments/<id>/rank-N` is missing or one level off —
+which is exactly what R5's bind did by putting `<id>` where `deployments/` belonged. The
+doc is corrected so nobody goes copying scaffolding around. (2) My “2564 MiB proves the
+deployment was read over NFS” was **wrong**: the same 2608 MiB appeared with
+`deployments/` local, and a warm NFS client cache explains the rest. The corrected
+conclusion uses the negative control instead of a counter — and the ~2.6 GB per-startup
+NAS read is now an explicit open item, because since R4 `package_cache` points at the
+NAS, and it is not yet confirmed that a rank can start with the NAS unavailable. That
+check is now a pre-window item.
+
+Also learned about the test bed (recorded in the doc so nobody re-learns it): `--rank`
+and `--rendezvous` are rejected for a single-host core, a 262k-context rank reserves
+~109 GiB RAM, and after `pkill` a rank needs ~10 s to release its port (my 3 s restarts
+produced two silent failures).
+
+Cleanup done the taught way — fstab line removed, `daemon-reload`, **unmount
+deepest-first**, then `rm -rf --one-file-system`: 0 stray binds, `teststore` gone,
+`nodes/` holds only the real `gx10-dgx4` artefacts, store still
+`1 model · 1 version · gb10-tp4-202611120031`. dgx4 back at 215 G used / 1.5 T free.
+Serving untouched throughout, including dgx4's reboot.