engine-drivers spec: correct DGPP provisioning model (rank0-only install, idempotent, space-nodes, script-file tee) + DGPP deploy prereqs from live w2 run

specs/tech-architecture/engine-drivers.md

@@ -98,24 +98,39 @@ ignore it).
 ### DGPP provisioning (native analog of an on-demand image build)
 
 A recipe pins `[dgpp].repo` (git URL) + `[dgpp].rev` (commit/tag). Before
-launch, `serve` provisions DGPP on **every** launch node (in parallel) —
-`src/native/provision.rs`:
-
-1. Skip key: `<dir>/.sm121-dgpp-rev` (the marker, like the docker
-   `sm121.checksum` label). If it already equals `rev`, the whole pass is a
-   no-op — the second serve is free.
-2. Else, in a detached tmux session (`sm121-dgpp-<recipe>`, log
+launch, `serve` provisions DGPP — `src/native/provision.rs`:
+
+1. **Stage first:** the generated cluster JSON + site `.env` are written to
+   EVERY launch node (`dgpp-cluster install` validates `DGPP_NODES` per node;
+   `DGPP_NODES` is **space**-separated, rank order).
+2. **Provision on rank 0 only** — `dgpp-cluster install` is head-coordinated:
+   it installs the release on rank 0 AND SSHes to the peers (from `DGPP_NODES`)
+   to install there too (the docker build-once + sync analog). Running it on
+   every node double-installs the peers ("release already exists").
+3. Skip key: `<dir>/.sm121-dgpp-rev` (the marker, like the docker
+   `sm121.checksum` label). If it already equals `rev`, the pass is a no-op.
+4. Else, in a detached tmux session (`sm121-dgpp-<recipe>`, log
    `/tmp/<recipe>-dgpp-build.log`, 30 s poll / 90 min cap — same shape as
-   `docker/build.rs`): `git clone <repo> <dir>` (or `git fetch --all --tags`)
-   → `git checkout <rev>` → `scripts/release.sh` (builds the CUDA kernels →
-   `dist/dgpp-*.tar.zst`) → `scripts/dgpp-cluster install dist/dgpp-*.tar.zst`
-   → `echo <rev> > .sm121-dgpp-rev` → `echo SM121-DGPP-DONE`.
-3. `set -e` + the marker/DONE echo last: a failed build leaves no marker, so
-   the next serve retries; success is detected by grepping `SM121-DGPP-DONE`.
-
-No `[dgpp].repo`/`rev` → assume DGPP is pre-installed (the pre-provisioning
-behavior). Bumping `rev` cuts a new DGPP generation (the marker keys on it).
-`dir` is `~`-expanded to `$HOME` so the remote shell resolves it.
+   `docker/build.rs`): the build script is written to a **file** and run as
+   `bash <file> 2>&1 | tee <log>` (inline `cmd && ... && echo DONE | tee` would
+   pipe only the final echo). The script: `git clone/fetch` → `git checkout
+   <rev>` → `scripts/release.sh` (CUDA kernels → `dist/dgpp-*.tar.zst`) →
+   `dgpp-cluster install dist/dgpp-*.tar.zst --config <staged json>` →
+   `echo <rev> > .sm121-dgpp-rev` → `echo SM121-DGPP-DONE`.
+5. `install` is not idempotent ("release already exists"): the script tolerates
+   it via `|| dgpp-cluster releases --config <json> | grep -qF <rev-abbrev>`, so
+   a re-serve at the same pin (marker lost) is still safe. `set -e` + the
+   marker/DONE last: a genuinely failed build leaves no marker → retry.
+
+No `[dgpp].repo`/`rev` → assume DGPP is pre-installed. Bumping `rev` cuts a new
+generation (the marker keys on it). `dir` is `~`-expanded to `$HOME`.
+
+**DGPP deployment prereqs (validated live 2026-10-15, w2 on 2nodes-dac):**
+provisioning + `dgpp-cluster up` work; DGPP's own preflight then gates on
+things the launcher does NOT own — the model checkpoint must be in the node's
+HF cache (`python3 scripts/download_model.py --config <file>` on rank 0), and
+multi-fabric hosts need `DGPP_ROCE_DEVICES` set explicitly ("transport supports
+at most two lanes"). These are DGPP-side deployment steps, not launcher bugs.
 
 ## Phasing