native/dgpp: on-demand DGPP provisioning (clone-at-pin + release.sh + install + rev-marker)

sm121 now installs DGPP itself when a recipe pins [dgpp].repo + [dgpp].rev —
the native analog of building a docker image on demand. src/native/provision.rs:
per launch node (parallel), skip if <dir>/.sm121-dgpp-rev == rev; else a detached
tmux session (30s poll / 90min cap, like docker/build.rs) runs git clone/fetch +
checkout <rev> + scripts/release.sh + dgpp-cluster install, then stamps the rev
marker + SM121-DGPP-DONE (set -e, marker last → a failed build leaves none and
retries). No repo/rev → assume pre-installed. Wired into native_serve before the
config upload. Unit tests for build_script/tmux_build/needs_provisioning. Spec +
AGENTS updated. make check green (378 tests).

AGENTS.md

@@ -348,6 +348,11 @@ The `deploy` field selects how the recipe is launched on each node:
   `[dgpp].dir` on rank 0, and runs `scripts/dgpp-cluster up --config <file>`
   (nohup, log under `<dir>/log/`); `stop` runs `scripts/dgpp-cluster down`.
   Readiness is `/v1/models` on `[dgpp].http_port` (default 18080).
+  **Provisioning:** pin `[dgpp].repo` + `[dgpp].rev` and `serve` clones the DGPP
+  repo at that rev on every launch node, runs `scripts/release.sh` +
+  `dgpp-cluster install`, and stamps `<dir>/.sm121-dgpp-rev` — a later serve at
+  the same pin skips the build (the native analog of the docker image build +
+  checksum label). No `repo`/`rev` → assume pre-installed.
 
 **First-class engines** (`engine` selects an `EngineDriver` in `src/engine.rs`):
 `vllm` (default), `sglang`, `gr8` (`kindlingai/gr8`, docker deploy, `--tp/--peers`

specs/tech-architecture/engine-drivers.md

@@ -95,6 +95,28 @@ ignore it).
   `<dir>/log/`). Stop runs `scripts/dgpp-cluster down` on rank 0.
 - Readiness is the uniform `/v1/models` probe on `http_port` (default 18080).
 
+### DGPP provisioning (native analog of an on-demand image build)
+
+A recipe pins `[dgpp].repo` (git URL) + `[dgpp].rev` (commit/tag). Before
+launch, `serve` provisions DGPP on **every** launch node (in parallel) —
+`src/native/provision.rs`:
+
+1. Skip key: `<dir>/.sm121-dgpp-rev` (the marker, like the docker
+   `sm121.checksum` label). If it already equals `rev`, the whole pass is a
+   no-op — the second serve is free.
+2. Else, in a detached tmux session (`sm121-dgpp-<recipe>`, log
+   `/tmp/<recipe>-dgpp-build.log`, 30 s poll / 90 min cap — same shape as
+   `docker/build.rs`): `git clone <repo> <dir>` (or `git fetch --all --tags`)
+   → `git checkout <rev>` → `scripts/release.sh` (builds the CUDA kernels →
+   `dist/dgpp-*.tar.zst`) → `scripts/dgpp-cluster install dist/dgpp-*.tar.zst`
+   → `echo <rev> > .sm121-dgpp-rev` → `echo SM121-DGPP-DONE`.
+3. `set -e` + the marker/DONE echo last: a failed build leaves no marker, so
+   the next serve retries; success is detected by grepping `SM121-DGPP-DONE`.
+
+No `[dgpp].repo`/`rev` → assume DGPP is pre-installed (the pre-provisioning
+behavior). Bumping `rev` cuts a new DGPP generation (the marker keys on it).
+`dir` is `~`-expanded to `$HOME` so the remote shell resolves it.
+
 ## Phasing
 
 1. **Trait + vLLM/SGLang drivers** — pure refactor, six call-sites delegate,

src/config/recipe.rs

@@ -242,6 +242,16 @@ pub struct DgppConfig {
     /// empty = DGPP default. Emitted as `paths.resident_cache` in the JSON.
     #[serde(default)]
     pub resident_cache: Option<String>,
+    /// DGPP source pin for on-demand provisioning (the native analog of a
+    /// docker image build). When BOTH `repo` and `rev` are set, serve clones
+    /// the repo at `rev` on each launch node, runs `scripts/release.sh` +
+    /// `dgpp-cluster install`, and stamps a rev marker so a later serve at the
+    /// same pin skips the build. Absent → assume DGPP is pre-installed.
+    #[serde(default)]
+    pub repo: Option<String>,
+    /// Pinned commit / tag / branch to check out (folded into the marker).
+    #[serde(default)]
+    pub rev: Option<String>,
     /// The DGPP `engine` block, serialized verbatim into the cluster JSON.
     #[serde(default)]
     pub engine: HashMap<String, serde_json::Value>,

src/native/mod.rs

@@ -15,6 +15,8 @@ use tracing::info;
 use crate::config::{Cluster, Config, DgppConfig, Node, Recipe};
 use crate::docker::ServeOpts;
 
+mod provision;
+
 /// DGPP HTTP port for a recipe (its `[dgpp].http_port`, else 18080).
 pub fn http_port(recipe: &Recipe) -> u16 {
     recipe
@@ -187,6 +189,16 @@ pub async fn native_serve(
     )
     .await?;
 
+    // Provision the DGPP checkout at the pinned rev on every launch node (the
+    // native analog of building the docker image on demand): clone-at-pin +
+    // release.sh + install, with a rev-marker skip. No-op when the recipe
+    // declares no [dgpp].repo/rev (assume pre-installed). Nodes in parallel.
+    let prov = cluster
+        .nodes
+        .iter()
+        .map(|n| provision::provision_node(config, n, recipe_name, &dgpp, dry_run));
+    futures::future::try_join_all(prov).await?;
+
     let dir = expand_home(&dgpp_dir(&dgpp));
     let cfg_name = format!("cluster_sm121_{recipe_name}.json");
     let cfg_path = format!("{dir}/{cfg_name}");

src/native/provision.rs

@@ -0,0 +1,211 @@
+//! DGPP native-engine provisioning — the native analog of an on-demand docker
+//! image build. When a recipe pins `[dgpp].repo` + `[dgpp].rev`, `serve`
+//! clones the DGPP repo at that rev on each launch node, builds the release
+//! (`scripts/release.sh` → `dist/dgpp-*.tar.zst`), installs it
+//! (`scripts/dgpp-cluster install`), and stamps a rev marker
+//! (`<dir>/.sm121-dgpp-rev`). A later serve at the same pin skips the whole
+//! pass (the marker is the skip key, like the docker `sm121.checksum` label).
+//!
+//! The build runs in a detached tmux session with a 30 s poll / 90 min cap —
+//! a CUDA-kernel compile must not ride on one SSH session (same shape as
+//! `docker/build.rs::build_image_on_node`).
+
+use anyhow::Result;
+use tracing::info;
+
+use crate::config::{Config, DgppConfig, Node};
+use crate::ssh::run_ssh;
+
+use super::{dgpp_dir, expand_home};
+
+const MAX_BUILD_SECS: u64 = 90 * 60;
+const DONE_MARKER: &str = "SM121-DGPP-DONE";
+
+/// The build script run inside the tmux session on a node. `dir` is already
+/// `$HOME`-expanded and is double-quoted (so `$HOME` expands); `repo`/`rev`
+/// are single-quoted literals. The rev marker + DONE echo run only if every
+/// prior step succeeded (`set -e`), so a failed build leaves no marker and the
+/// next serve retries.
+pub fn build_script(repo: &str, rev: &str, dir: &str) -> String {
+    format!(
+        "set -e; \
+         if [ -d \"{dir}/.git\" ]; then cd \"{dir}\" && git fetch --all --tags && git checkout '{rev}'; \
+         else git clone '{repo}' \"{dir}\" && cd \"{dir}\" && git checkout '{rev}'; fi; \
+         scripts/release.sh && \
+         scripts/dgpp-cluster install dist/dgpp-*.tar.zst && \
+         echo '{rev}' > .sm121-dgpp-rev && echo {DONE_MARKER}"
+    )
+}
+
+/// The detached tmux new-session command wrapping the build script (tee to a
+/// per-recipe log). Session/log names are per-recipe, like the docker build.
+pub fn tmux_build(recipe_name: &str, dir: &str, repo: &str, rev: &str) -> String {
+    let session = format!("sm121-dgpp-{recipe_name}");
+    let logfile = format!("/tmp/{recipe_name}-dgpp-build.log");
+    format!(
+        "tmux kill-session -t '{session}' 2>/dev/null || true; \
+         tmux new-session -d -s '{session}' '{script} 2>&1 | tee {logfile}'",
+        script = build_script(repo, rev, dir).replace('\'', r"'\''"),
+    )
+}
+
+/// Skip key: the node needs provisioning unless its marker already equals the
+/// pinned rev.
+pub fn needs_provisioning(marker: &str, rev: &str) -> bool {
+    marker.trim() != rev
+}
+
+/// Provision DGPP on one node at the recipe's pinned rev. No-op when the
+/// recipe declares no `[dgpp].repo`/`rev` (assume pre-installed) or when the
+/// node's marker already matches the pin.
+pub async fn provision_node(
+    config: &Config,
+    node: &Node,
+    recipe_name: &str,
+    d: &DgppConfig,
+    dry_run: bool,
+) -> Result<()> {
+    let (repo, rev) = match (d.repo.as_deref(), d.rev.as_deref()) {
+        (Some(r), Some(v)) if !r.is_empty() && !v.is_empty() => (r, v),
+        _ => return Ok(()),
+    };
+    let dir = expand_home(&dgpp_dir(d));
+    let marker_path = format!("{dir}/.sm121-dgpp-rev");
+
+    if dry_run {
+        run_ssh(
+            config,
+            node,
+            format!("cat \"{marker_path}\" 2>/dev/null || true"),
+            true,
+        )
+        .await?;
+        run_ssh(config, node, tmux_build(recipe_name, &dir, repo, rev), true).await?;
+        return Ok(());
+    }
+
+    let marker = run_ssh(
+        config,
+        node,
+        format!("cat \"{marker_path}\" 2>/dev/null || true"),
+        false,
+    )
+    .await
+    .unwrap_or_default();
+    if !needs_provisioning(&marker, rev) {
+        info!("DGPP {rev} already provisioned on '{}'", node.name);
+        return Ok(());
+    }
+
+    info!(
+        "Provisioning DGPP '{rev}' on '{}' (tmux: sm121-dgpp-{recipe_name})...",
+        node.name
+    );
+    run_ssh(
+        config,
+        node,
+        tmux_build(recipe_name, &dir, repo, rev),
+        false,
+    )
+    .await?;
+
+    let session = format!("sm121-dgpp-{recipe_name}");
+    let logfile = format!("/tmp/{recipe_name}-dgpp-build.log");
+    let start = std::time::Instant::now();
+    loop {
+        tokio::time::sleep(std::time::Duration::from_secs(30)).await;
+        let elapsed = start.elapsed();
+        if elapsed.as_secs() >= MAX_BUILD_SECS {
+            anyhow::bail!(
+                "DGPP provision on '{}' timed out after {}s — inspect tmux session \
+                 '{session}' / log '{logfile}' on the node.",
+                node.name,
+                elapsed.as_secs()
+            );
+        }
+        // `tmux has-session` prints nothing while alive; `|| echo DEAD` marks
+        // it gone (still exit 0). Empty output therefore means STILL RUNNING.
+        let alive = run_ssh(
+            config,
+            node,
+            format!("tmux has-session -t '{session}' 2>&1 || echo DEAD"),
+            false,
+        )
+        .await?;
+        if let Ok(logs) = run_ssh(
+            config,
+            node,
+            format!("tail -8 {logfile} 2>/dev/null"),
+            false,
+        )
+        .await
+        {
+            for line in logs.lines().filter(|l| !l.is_empty()) {
+                info!("  [dgpp-build] {}", line.trim());
+            }
+        }
+        if alive.trim().contains("DEAD") {
+            let ok = run_ssh(
+                config,
+                node,
+                format!("grep -F '{DONE_MARKER}' {logfile} >/dev/null 2>&1 && echo OK || echo NO"),
+                false,
+            )
+            .await
+            .map(|s| s.trim() == "OK")
+            .unwrap_or(false);
+            if ok {
+                info!("DGPP '{rev}' provisioned on '{}'", node.name);
+                return Ok(());
+            }
+            let exit_log = run_ssh(config, node, format!("tail -25 {logfile}"), false)
+                .await
+                .unwrap_or_default();
+            anyhow::bail!("DGPP provision on '{}' failed.\n{}", node.name, exit_log);
+        }
+        info!("  DGPP build running for {}s...", elapsed.as_secs());
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn build_script_pins_rev_and_stamps_marker_only_on_success() {
+        let s = build_script(
+            "https://github.com/HawkBearPig/dgpp.git",
+            "v1.2.3",
+            "$HOME/dgpp",
+        );
+        // dir double-quoted ($HOME expands), repo/rev single-quoted literals.
+        assert!(s.contains("git clone 'https://github.com/HawkBearPig/dgpp.git' \"$HOME/dgpp\""));
+        assert!(s.contains("git checkout 'v1.2.3'"));
+        assert!(s.contains("scripts/release.sh"));
+        assert!(s.contains("scripts/dgpp-cluster install dist/dgpp-*.tar.zst"));
+        // marker + DONE come last (after `&&`), so a failed build leaves none.
+        assert!(s.contains("echo 'v1.2.3' > .sm121-dgpp-rev && echo SM121-DGPP-DONE"));
+        assert!(s.find("scripts/release.sh").unwrap() < s.find(".sm121-dgpp-rev").unwrap());
+    }
+
+    #[test]
+    fn tmux_build_detaches_and_tees_to_per_recipe_log() {
+        let t = tmux_build(
+            "glm53-flash-nvfp4-fp8-dgpp-2x",
+            "$HOME/dgpp",
+            "https://x/dgpp.git",
+            "abc123",
+        );
+        assert!(t.contains("tmux new-session -d -s 'sm121-dgpp-glm53-flash-nvfp4-fp8-dgpp-2x'"));
+        assert!(t.contains("tee /tmp/glm53-flash-nvfp4-fp8-dgpp-2x-dgpp-build.log"));
+        // stale-session guard so a rerun doesn't hit "duplicate session".
+        assert!(t.contains("tmux kill-session -t 'sm121-dgpp-glm53-flash-nvfp4-fp8-dgpp-2x'"));
+    }
+
+    #[test]
+    fn needs_provisioning_keys_on_the_rev_marker() {
+        assert!(needs_provisioning("", "v1.2.3")); // no marker
+        assert!(needs_provisioning("v1.2.2\n", "v1.2.3")); // drifted pin
+        assert!(!needs_provisioning("v1.2.3\n", "v1.2.3")); // same pin → skip
+    }
+}