|
|
@@ -361,3 +361,52 @@ dgx1 api=200.
|
|
|
- round 3 · -- · 36 turns · continue → continue · outcome
|
|
|
- round 4 · -- · 28 turns · continue → continue · outcome
|
|
|
- round 5 · -- · 54 turns · continue → continue · validate the bind-mount model on idle dgx4 — Phase 3 gate…
|
|
|
+
|
|
|
+## 2026-11-12 — R6 round (flare-storage-nas): the bind design is proven end to end, on one node, without a window
|
|
|
+
|
|
|
+**The round's premise was that R6b needed the window. It did not.** `local-inference-lab/Qwen3.8-27B-FLARE`
|
|
|
+(branch `gb10`, 25.5 GB, 317 s at ~80 MiB/s) yields a **TP 1 · ready** core —
|
|
|
+`Qwen3.8-27B-NVIDIA-NVFP4-QAD-MXFP8 · gb10-tp1-…`, 23.77 GiB, weights 22 G +
|
|
|
+deployments 2.5 G + envelope 18 M — which let dgx4 run a **complete, serving rank** with
|
|
|
+`deployments/` bind-mounted from `nodes/<node>/<model>/<version>/deployments` on the NAS.
|
|
|
+`api=200` in 20 s warm, 21 s cold (page cache dropped on the idle node), and a chat
|
|
|
+completion came back (reasoning model: `content: null`, `reasoning_content` populated).
|
|
|
+
|
|
|
+The four things that mattered, each measured rather than assumed:
|
|
|
+
|
|
|
+- **Reads do traverse the bind** — the negative control is the proof, not the traffic
|
|
|
+ counter: bind a NAS dir containing no rank slice and the rank dies with
|
|
|
+ `Verify core … open core directory: No such file or directory`. Bind one that contains
|
|
|
+ `<id>/rank-N` and it serves. No hidden fallback to local disk.
|
|
|
+- **FLARE never touches the bind** — 0 files newer than the rank start inside it, inode
|
|
|
+ identical, mount still present after serving, after a restart, and after a reboot.
|
|
|
+ **Option D is off the table.**
|
|
|
+- **The NAS cost is a one-time load cost** — steady-state inference pulled **0 MiB** over
|
|
|
+ the wire; combined with the earlier 1.29 s / 1.1 GB/s read of a TP4 rank slice, a
|
|
|
+ NAS-backed `deployments/` is effectively free per start.
|
|
|
+- **fstab persistence works** — installed the exact Phase 4 stanza on dgx4, rebooted it
|
|
|
+ (idle, 0 ranks): bind present at boot, `systemctl show <bind-unit> -p Requires`
|
|
|
+ contains `mnt-nas.mount`, and a rank came up from it **22 s after boot**.
|
|
|
+
|
|
|
+**Two of my own earlier conclusions died this round.** (1) R5's claim that a model dir
|
|
|
+needs the store's `blobs/ refs/ staging/` scaffolding was a **misattribution**: the
|
|
|
+`open core directory` error means `deployments/<id>/rank-N` is missing or one level off —
|
|
|
+which is exactly what R5's bind did by putting `<id>` where `deployments/` belonged. The
|
|
|
+doc is corrected so nobody goes copying scaffolding around. (2) My “2564 MiB proves the
|
|
|
+deployment was read over NFS” was **wrong**: the same 2608 MiB appeared with
|
|
|
+`deployments/` local, and a warm NFS client cache explains the rest. The corrected
|
|
|
+conclusion uses the negative control instead of a counter — and the ~2.6 GB per-startup
|
|
|
+NAS read is now an explicit open item, because since R4 `package_cache` points at the
|
|
|
+NAS, and it is not yet confirmed that a rank can start with the NAS unavailable. That
|
|
|
+check is now a pre-window item.
|
|
|
+
|
|
|
+Also learned about the test bed (recorded in the doc so nobody re-learns it): `--rank`
|
|
|
+and `--rendezvous` are rejected for a single-host core, a 262k-context rank reserves
|
|
|
+~109 GiB RAM, and after `pkill` a rank needs ~10 s to release its port (my 3 s restarts
|
|
|
+produced two silent failures).
|
|
|
+
|
|
|
+Cleanup done the taught way — fstab line removed, `daemon-reload`, **unmount
|
|
|
+deepest-first**, then `rm -rf --one-file-system`: 0 stray binds, `teststore` gone,
|
|
|
+`nodes/` holds only the real `gx10-dgx4` artefacts, store still
|
|
|
+`1 model · 1 version · gb10-tp4-202611120031`. dgx4 back at 215 G used / 1.5 T free.
|
|
|
+Serving untouched throughout, including dgx4's reboot.
|