|
@@ -299,7 +299,29 @@ previous phase failed.
|
|
|
4. ~~dgx8 has the NAS in fstab but not mounted~~ **fixed 2026-11-11** — mounted at
|
|
4. ~~dgx8 has the NAS in fstab but not mounted~~ **fixed 2026-11-11** — mounted at
|
|
|
`vers=4.1` and made non-blocking with the phase 1 options.
|
|
`vers=4.1` and made non-blocking with the phase 1 options.
|
|
|
|
|
|
|
|
-Gate: binary rebuilt and staged, NAS mounted on every participating node.
|
|
|
|
|
|
|
+**Pre-window disk math — do this before scheduling the window (2026-11-12).**
|
|
|
|
|
+The acquired core's `weights/` is **470 G**, i.e. *bigger* than the old 260 GiB the plan
|
|
|
|
|
+was written against, and per-node free space is the one thing a window cannot recover:
|
|
|
|
|
+
|
|
|
|
|
+| node | root | free now | superseded data freed by the window | note |
|
|
|
|
|
+|---|---|---|---|---|
|
|
|
|
|
+| dgx1 | 915 G | 283 G | 477 G package store (open by rank 0 until restart) | **tight**: ~760 G free afterwards |
|
|
|
|
|
+| dgx2 | 1.8 T | 467 G | 261 G model dir | comfortable |
|
|
|
|
|
+| dgx3 | 1.8 T | 470 G | 261 G model dir | comfortable |
|
|
|
|
|
+| dgx4 | 1.8 T | 1.5 T | — (idle) | the staging test bed |
|
|
|
|
|
+| dgx8 | 1.8 T | ~470 G | 261 G model dir | comfortable |
|
|
|
|
|
+
|
|
|
|
|
+A peer's local footprint is **not** the store total: on the old build dgx2 held 261 G
|
|
|
|
|
+where the package store was 477 G, and a rank slice is only 3 files / 1.3 GiB. The exact
|
|
|
|
|
+new per-rank number is unknown until a real stage happens — **measure it on dgx4 first**
|
|
|
|
|
+(1.5 T free), because if a rank needs the full 470 G locally then dgx1 lands at ~40 %
|
|
|
|
|
+free and must not receive any further local copies. Never start a window whose aggregate
|
|
|
|
|
+`weights/` exceeds a node's free space after its own deletion: a rank that cannot write
|
|
|
|
|
+its staging tree takes the whole group down.
|
|
|
|
|
+
|
|
|
|
|
+Gate: binary staged and verified on every node (`## Checks` probe), agent binary swapped
|
|
|
|
|
+wherever ranks are not running, NAS mounted on every participating node, and the table
|
|
|
|
|
+above re-measured the day of the window.
|
|
|
|
|
|
|
|
### Phase 1 — make the NAS mount boot-proof
|
|
### Phase 1 — make the NAS mount boot-proof
|
|
|
|
|
|