|
|
@@ -98,24 +98,39 @@ ignore it).
|
|
|
### DGPP provisioning (native analog of an on-demand image build)
|
|
|
|
|
|
A recipe pins `[dgpp].repo` (git URL) + `[dgpp].rev` (commit/tag). Before
|
|
|
-launch, `serve` provisions DGPP on **every** launch node (in parallel) —
|
|
|
-`src/native/provision.rs`:
|
|
|
-
|
|
|
-1. Skip key: `<dir>/.sm121-dgpp-rev` (the marker, like the docker
|
|
|
- `sm121.checksum` label). If it already equals `rev`, the whole pass is a
|
|
|
- no-op — the second serve is free.
|
|
|
-2. Else, in a detached tmux session (`sm121-dgpp-<recipe>`, log
|
|
|
+launch, `serve` provisions DGPP — `src/native/provision.rs`:
|
|
|
+
|
|
|
+1. **Stage first:** the generated cluster JSON + site `.env` are written to
|
|
|
+ EVERY launch node (`dgpp-cluster install` validates `DGPP_NODES` per node;
|
|
|
+ `DGPP_NODES` is **space**-separated, rank order).
|
|
|
+2. **Provision on rank 0 only** — `dgpp-cluster install` is head-coordinated:
|
|
|
+ it installs the release on rank 0 AND SSHes to the peers (from `DGPP_NODES`)
|
|
|
+ to install there too (the docker build-once + sync analog). Running it on
|
|
|
+ every node double-installs the peers ("release already exists").
|
|
|
+3. Skip key: `<dir>/.sm121-dgpp-rev` (the marker, like the docker
|
|
|
+ `sm121.checksum` label). If it already equals `rev`, the pass is a no-op.
|
|
|
+4. Else, in a detached tmux session (`sm121-dgpp-<recipe>`, log
|
|
|
`/tmp/<recipe>-dgpp-build.log`, 30 s poll / 90 min cap — same shape as
|
|
|
- `docker/build.rs`): `git clone <repo> <dir>` (or `git fetch --all --tags`)
|
|
|
- → `git checkout <rev>` → `scripts/release.sh` (builds the CUDA kernels →
|
|
|
- `dist/dgpp-*.tar.zst`) → `scripts/dgpp-cluster install dist/dgpp-*.tar.zst`
|
|
|
- → `echo <rev> > .sm121-dgpp-rev` → `echo SM121-DGPP-DONE`.
|
|
|
-3. `set -e` + the marker/DONE echo last: a failed build leaves no marker, so
|
|
|
- the next serve retries; success is detected by grepping `SM121-DGPP-DONE`.
|
|
|
-
|
|
|
-No `[dgpp].repo`/`rev` → assume DGPP is pre-installed (the pre-provisioning
|
|
|
-behavior). Bumping `rev` cuts a new DGPP generation (the marker keys on it).
|
|
|
-`dir` is `~`-expanded to `$HOME` so the remote shell resolves it.
|
|
|
+ `docker/build.rs`): the build script is written to a **file** and run as
|
|
|
+ `bash <file> 2>&1 | tee <log>` (inline `cmd && ... && echo DONE | tee` would
|
|
|
+ pipe only the final echo). The script: `git clone/fetch` → `git checkout
|
|
|
+ <rev>` → `scripts/release.sh` (CUDA kernels → `dist/dgpp-*.tar.zst`) →
|
|
|
+ `dgpp-cluster install dist/dgpp-*.tar.zst --config <staged json>` →
|
|
|
+ `echo <rev> > .sm121-dgpp-rev` → `echo SM121-DGPP-DONE`.
|
|
|
+5. `install` is not idempotent ("release already exists"): the script tolerates
|
|
|
+ it via `|| dgpp-cluster releases --config <json> | grep -qF <rev-abbrev>`, so
|
|
|
+ a re-serve at the same pin (marker lost) is still safe. `set -e` + the
|
|
|
+ marker/DONE last: a genuinely failed build leaves no marker → retry.
|
|
|
+
|
|
|
+No `[dgpp].repo`/`rev` → assume DGPP is pre-installed. Bumping `rev` cuts a new
|
|
|
+generation (the marker keys on it). `dir` is `~`-expanded to `$HOME`.
|
|
|
+
|
|
|
+**DGPP deployment prereqs (validated live 2026-10-15, w2 on 2nodes-dac):**
|
|
|
+provisioning + `dgpp-cluster up` work; DGPP's own preflight then gates on
|
|
|
+things the launcher does NOT own — the model checkpoint must be in the node's
|
|
|
+HF cache (`python3 scripts/download_model.py --config <file>` on rank 0), and
|
|
|
+multi-fabric hosts need `DGPP_ROCE_DEVICES` set explicitly ("transport supports
|
|
|
+at most two lanes"). These are DGPP-side deployment steps, not launcher bugs.
|
|
|
|
|
|
## Phasing
|
|
|
|
