|
@@ -305,3 +305,46 @@ never renames over / hardlinks into `deployments/` — either would take option
|
|
|
|
|
|
|
|
Serving untouched all round: dgx1/2/3/8 `ranks=1`, dgx1 `api=200`. dgx4 has 1.5 T free
|
|
Serving untouched all round: dgx1/2/3/8 `ranks=1`, dgx1 `api=200`. dgx4 has 1.5 T free
|
|
|
and no ranks, and now holds the first real `nodes/<node>/…` artefacts (5.2 G).
|
|
and no ranks, and now holds the first real `nodes/<node>/…` artefacts (5.2 G).
|
|
|
|
|
+
|
|
|
|
|
+### Second half of the same round: R5 landed, and the window now has a tool
|
|
|
|
|
+
|
|
|
|
|
+**R5 is done**, and it is not just paperwork — it produced the pre-cutover runtime table
|
|
|
|
|
+and the mechanism behind the fleet's mystery:
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+dgx1 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
|
|
|
|
|
+dgx2 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
|
|
|
|
|
+dgx3 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
|
|
|
|
|
+dgx4 staged=e3fe697e /usrbin=e3fe697e agent=e3fe697e exe=/home/kilork/.local/lib/flare-agent/flare
|
|
|
|
|
+dgx8 staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+Every serving node runs a `(deleted)` inode, i.e. **production is running neither the
|
|
|
|
|
+Nov-5 apt build nor v0.1.0** — it is the pre-apt (PR-184) binary held alive by the
|
|
|
|
|
+process. The reason apt never fixed this is now proven instead of inferred: the unit
|
|
|
|
|
+ExecStarts `%h/.local/lib/flare-agent/flare` and `dpkg -S` owns no such file, so every
|
|
|
|
|
+apt upgrade moved `/usr/bin/flare` and left the running path alone. The `## Checks`
|
|
|
|
|
+probe prints `agent=` for exactly this reason — it is the only column describing what
|
|
|
|
|
+executes.
|
|
|
|
|
+
|
|
|
|
|
+Then the swap was **rehearsed on idle dgx4** (temp+`mv` at both paths,
|
|
|
|
|
+`systemctl --user restart flare-agent.service`, sha + `readlink /proc/PID/exe`
|
|
|
|
|
+verification, `flare.orig-20261112` backups) and turned into
|
|
|
|
|
+**`host-scripts/flare-swap.sh`** so the window runs one reviewed command per node
|
|
|
|
|
+instead of hand-typed lines. Its two tests both behaved: on dgx4 it was a verified
|
|
|
|
|
+no-op with exit 0; on **dgx8, with a rank running, it refused** (exit 1) leaving shas
|
|
|
|
|
+unchanged, zero `tmp-swap` files behind, and the rank untouched. That refusal is the
|
|
|
|
|
+behaviour that protects prod during a window.
|
|
|
|
|
+
|
|
|
|
|
+**A trap for the window, found by reading the numbers again:** the new core's `weights/`
|
|
|
|
|
+is **470 G**, not the 260 GiB this plan was written against. dgx1's root is 915 G with
|
|
|
|
|
+283 G free, and its 477 G package store is only released when rank 0 restarts — so dgx1
|
|
|
|
|
+comes to ~760 G free *during* the window, and if a rank turns out to need the full 470 G
|
|
|
|
|
+locally, that is the only node where the arithmetic is close. Hence a new gate: measure
|
|
|
|
|
+the real per-rank footprint on dgx4 (1.5 T free) before rolling sparks/prod4, and
|
|
|
|
|
+re-measure the free-space table the day of the window.
|
|
|
|
|
+
|
|
|
|
|
+Repair of my own damage is still running healthily on dgx4 (pid 2037594, `read_bytes`
|
|
|
|
|
++370 MB/s against a 475 G blob re-verification, NIC idle — the progress bar counts
|
|
|
|
|
+network bytes only, which is why it reads `32.4 MiB` the whole time). When it lands,
|
|
|
|
|
+`models list` returns to `474.66 GiB`.
|