ソースを参照

docs(progress): R5 done - pre-cutover runtime table, the dpkg-does-not-own-the-agent-copy mechanism, flare-swap.sh tested on dgx4+dgx8, and the 470 G weights/ trap for the window

Alexander Korolev 1 週間 前
親
コミット
84c8d632ac
1 ファイル変更、43 行追加、0 行削除
  1. 43 0
      specs/tech-architecture/RALPH_PROGRESS.md

+ 43 - 0
specs/tech-architecture/RALPH_PROGRESS.md

@@ -305,3 +305,46 @@ never renames over / hardlinks into `deployments/` — either would take option
 
 
 Serving untouched all round: dgx1/2/3/8 `ranks=1`, dgx1 `api=200`. dgx4 has 1.5 T free
 Serving untouched all round: dgx1/2/3/8 `ranks=1`, dgx1 `api=200`. dgx4 has 1.5 T free
 and no ranks, and now holds the first real `nodes/<node>/…` artefacts (5.2 G).
 and no ranks, and now holds the first real `nodes/<node>/…` artefacts (5.2 G).
+
+### Second half of the same round: R5 landed, and the window now has a tool
+
+**R5 is done**, and it is not just paperwork — it produced the pre-cutover runtime table
+and the mechanism behind the fleet's mystery:
+
+```
+dgx1  staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
+dgx2  staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
+dgx3  staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
+dgx4  staged=e3fe697e /usrbin=e3fe697e agent=e3fe697e exe=/home/kilork/.local/lib/flare-agent/flare
+dgx8  staged=e3fe697e /usrbin=03c8e463 agent=03c8e463 exe=…flare-agent/flare (deleted)
+```
+
+Every serving node runs a `(deleted)` inode, i.e. **production is running neither the
+Nov-5 apt build nor v0.1.0** — it is the pre-apt (PR-184) binary held alive by the
+process. The reason apt never fixed this is now proven instead of inferred: the unit
+ExecStarts `%h/.local/lib/flare-agent/flare` and `dpkg -S` owns no such file, so every
+apt upgrade moved `/usr/bin/flare` and left the running path alone. The `## Checks`
+probe prints `agent=` for exactly this reason — it is the only column describing what
+executes.
+
+Then the swap was **rehearsed on idle dgx4** (temp+`mv` at both paths,
+`systemctl --user restart flare-agent.service`, sha + `readlink /proc/PID/exe`
+verification, `flare.orig-20261112` backups) and turned into
+**`host-scripts/flare-swap.sh`** so the window runs one reviewed command per node
+instead of hand-typed lines. Its two tests both behaved: on dgx4 it was a verified
+no-op with exit 0; on **dgx8, with a rank running, it refused** (exit 1) leaving shas
+unchanged, zero `tmp-swap` files behind, and the rank untouched. That refusal is the
+behaviour that protects prod during a window.
+
+**A trap for the window, found by reading the numbers again:** the new core's `weights/`
+is **470 G**, not the 260 GiB this plan was written against. dgx1's root is 915 G with
+283 G free, and its 477 G package store is only released when rank 0 restarts — so dgx1
+comes to ~760 G free *during* the window, and if a rank turns out to need the full 470 G
+locally, that is the only node where the arithmetic is close. Hence a new gate: measure
+the real per-rank footprint on dgx4 (1.5 T free) before rolling sparks/prod4, and
+re-measure the free-space table the day of the window.
+
+Repair of my own damage is still running healthily on dgx4 (pid 2037594, `read_bytes`
++370 MB/s against a 475 G blob re-verification, NIC idle — the progress bar counts
+network bytes only, which is why it reads `32.4 MiB` the whole time). When it lands,
+`models list` returns to `474.66 GiB`.