feat(serve): add composed_model for multi-repo model directories

A recipe may declare [composed_model] to combine multiple HF repos into a single model directory at serve time — e.g. a base checkpoint + an expert overlay — without mutating the HF cache. The launcher builds a /tmp dir per-node with symlinks into the HF cache (container-side absolute paths) + generated files, bind-mounts it at `mount`, and rewrites `model` to that path.

Symlinks point to /root/.cache/huggingface/... (the container-side HF cache mount), not the host-side path. They are dangling on the host but resolve correctly inside the container where the HF cache is mounted at that exact path — avoiding the host↔container mount-root mismatch. Later sources win (overwrite earlier symlinks), so order matters for overlays.

The NF4 Lloyd recipe (experts-only repo overlaid on the MXFP4 base checkpoint + generated nf4_layers.json) now uses [composed_model] instead of manual prerequisites — no operator-side symlink/cp/JSON-creation steps, no {{placeholder}} extra_mounts. The composed dir is cleaned up automatically on `sm121 stop`.

Also adds recipes/test-composed/ — an alpine recipe that exercises the composed_model mechanism (symlink composition + bind-mount + model rewrite) with a heartbeat entrypoint, without launching heavy vLLM containers.

AGENTS.md

@@ -178,6 +178,38 @@ VLLM_COMMIT = "00907fc9b982f973808a0193ad0f8c61fbb62854"  # bump fork commit for
 TORCH_VERSION = "2.11.0"
 ```
 
+## Composed model directories
+
+A recipe may declare `[composed_model]` to combine multiple HF repos into a
+single model directory at serve time — e.g. a base checkpoint + an expert
+overlay — without mutating the HF cache. The launcher builds a `/tmp` dir
+per-node with symlinks into the HF cache (container-side absolute paths) +
+generated files, bind-mounts it at `mount`, and rewrites `model` to that path.
+
+Symlinks point to `/root/.cache/huggingface/...` (the container-side HF cache
+mount), not the host-side path. They are dangling on the host but resolve
+correctly inside the container where the HF cache is mounted at that exact
+path — avoiding the host↔container mount-root mismatch. Later sources win
+(overwrite earlier symlinks), so order matters for overlays.
+
+```toml
+[composed_model]
+mount = "/root/model"               # container-side target; `model` is rewritten to this
+
+[[composed_model.sources]]          # symlinked into the composed dir (later wins)
+repo = "aidendle94/GLM-5.2-MXFP4-Experts-GPTQ"      # base: config, weights, tokenizer
+
+[[composed_model.sources]]
+repo = "aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ"  # overlay: Lloyd NF4 experts
+glob = "L*.safetensors"                             # only expert weight files
+
+[[composed_model.files]]            # generated real files (not symlinks)
+name = "nf4_layers.json"
+content = '{"layers":[3,4,...,77]}'
+```
+
+The composed dir is cleaned up automatically on `sm121 stop`.
+
 ## Config fields
 
 ```toml

recipes/glm52-nf4-lloyd-experts-gptq/recipe.toml

@@ -9,52 +9,36 @@
 # reconstruction error vs E2M1/MXFP4 baseline on the same GPTQ pipeline).
 # Codes are indices into per-layer grids/L{n}.json — NOT the E2M1 table.
 #
-# LOADING: the Lloyd repo is experts-only (L{n}.safetensors + grids/). It
-# cannot be served standalone. The base checkpoint is
-# aidendle94/GLM-5.2-MXFP4-Experts-GPTQ (full model: config.json with
-# quant_method="hybrid_mxfp4_ct", hybrid-ct-*.safetensors non-expert weights
-# in block-FP8/NVFP4, tokenizer, mtp-draft). The Lloyd NF4 experts REPLACE
-# the MXFP4 E2M1 experts for layers 3–77 (all 75 MoE layers): the Lloyd
-# L{n}.safetensors files have the same filenames and tensor names as the
-# MXFP4 repo's, so they must be copied into the MXFP4 snapshot directory
-# (filesystem overlay). vLLM loads them through the standard weight loader;
-# the nf4_e8m0_k32 kernel (baked in production-hybrid-1.4) decodes them with
-# the global _NF4_CODEBOOK instead of the E2M1 bit-twiddle. Which layers use
-# the NF4 path is controlled by VLLM_NF4_LAYERS — a path to a JSON file
-# {"layers": [3, 4, ..., 77]}, NOT a range string.
+# LOADING via [composed_model]: the Lloyd repo is experts-only (no config,
+# no tokenizer, no non-expert weights). The launcher builds a /tmp dir per
+# node with symlinks into the HF cache and bind-mounts it at /root/model:
+#   1. aidendle94/GLM-5.2-MXFP4-Experts-GPTQ (base checkpoint — config.json
+#      with quant_method="hybrid_mxfp4_ct", hybrid-ct-*.safetensors non-expert
+#      weights, tokenizer, mtp-draft) → all files symlinked.
+#   2. aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ (Lloyd NF4 experts) →
+#      L*.safetensors symlinked, OVERWRITING the MXFP4 E2M1 experts
+#      (later sources win). Same filenames, same tensor names — the Lloyd
+#      codes replace the E2M1 codes.
+#   3. nf4_layers.json (generated) — tells the kernel which layers use the
+#      nf4_e8m0_k32 decode path vs the default fp4_e8m0_k32 E2M1 bit-twiddle.
+#
+# The symlinks point to container-side paths (/root/.cache/huggingface/...)
+# so they resolve inside the container where the HF cache is mounted. The
+# _NF4_CODEBOOK is GLOBAL and hardcoded in the b12x binary (fitted as the
+# mean of 71 per-layer Lloyd–Max tables) — the per-layer grids/L{n}.json
+# files are reference artifacts, NOT loaded at runtime.
 #
 # Image: sparkrun-vllm-ds4-gb10:glm52
 #   Built from docker-images/glm52/Dockerfile, which FROMs
 #   aidendle94/sparkrun-vllm-ds4-gb10:production-hybrid-1.4 (bakes the
 #   nf4_e8m0_k32 / nf4_packed kernel — inert without VLLM_NF4_LAYERS).
 #
-# PREREQUISITES (manual, on EVERY node):
-#   1. Download both repos via HF_HOME:
-#        huggingface-cli download aidendle94/GLM-5.2-MXFP4-Experts-GPTQ
-#        huggingface-cli download aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ
-#      (MXFP4 base ~390 GB, Lloyd experts ~385 GB)
-#   2. Overlay the Lloyd NF4 experts onto the MXFP4 snapshot directory (same
-#      filenames, same tensor names — they replace the E2M1 experts with
-#      NF4-packed codes). Use RELATIVE symlinks (ln -snrf), not copies: the
-#      HF cache is mounted at /root/.cache/huggingface inside the container
-#      (vs /home/<user>/.cache/huggingface on the host), so absolute symlinks
-#      would break — relative paths resolve the same under both roots, like
-#      HF's own snapshot→blob links. -n is critical: it stops ln from
-#      dereferencing the existing MXFP4 symlink (without -n, ln would replace
-#      the BLOB the link points to, corrupting the 390 GB MXFP4 cache).
-#      Lightweight: ~75 symlinks, no 385 GB copy.
-#        MXFP4=$(readlink -f ~/.cache/huggingface/hub/models--aidendle94--GLM-5.2-MXFP4-Experts-GPTQ/snapshots/main)
-#        LLOYD=$(readlink -f ~/.cache/huggingface/hub/models--aidendle94--GLM-5.2-NF4-Lloyd-Experts-GPTQ/snapshots/main)
-#        for f in "$LLOYD"/L*.safetensors; do ln -snrf "$f" "$MXFP4/$(basename "$f")"; done
-#      Without this overlay the kernel decodes E2M1 weights with the NF4
-#      codebook → garbage output (the model card warns about this).
-#   3. Create the NF4 layer-list JSON in the Lloyd snapshot dir (the kernel
-#      reads this via VLLM_NF4_LAYERS — {{lloyd_experts}}/nf4_layers.json):
-#        echo "{\"layers\":[$(seq -s, 3 77)]}" > "$LLOYD"/nf4_layers.json
-#
-# REFERENCE:
-#   https://huggingface.co/aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ
-#   https://huggingface.co/aidendle94/GLM-5.2-MXFP4-Experts-GPTQ
+# PREREQUISITES:
+#   - Both repos cached on every node via HF_HOME:
+#       huggingface-cli download aidendle94/GLM-5.2-MXFP4-Experts-GPTQ
+#       huggingface-cli download aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ
+#     (MXFP4 base ~390 GB, Lloyd experts ~385 GB)
+#   No manual overlay or JSON creation — [composed_model] handles everything.
 
 engine = "vllm"
 name = "GLM-5.2 NF4 Lloyd-Experts-GPTQ (vLLM)"
@@ -95,10 +79,10 @@ args = [
     "--max-num-seqs", "3",
     "--gpu-memory-utilization", "0.89",
     # Speculative decode — MTP k=4 with adaptive depths
-    # {{mtp_draft}} is substituted per-node with the resolved HF cache path
-    # (refs/main → latest snapshot) so symlinks resolve correctly.
+    # /root/model/mtp-draft is symlinked from the MXFP4 base checkpoint
+    # (the composed dir includes it via source #1).
     "--speculative-config", """
-'{"model":"{{mtp_draft}}","method":"mtp","num_speculative_tokens":4,"draft_attention_backend":"B12X_MLA_SPARSE","adaptive_speculative_tokens_window":32}'""",
+'{"model":"/root/model/mtp-draft","method":"mtp","num_speculative_tokens":4,"draft_attention_backend":"B12X_MLA_SPARSE","adaptive_speculative_tokens_window":32}'""",
     # Compilation — FULL_AND_PIECEWISE with all custom ops, 8 capture sizes
     "--compilation-config", """
 '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"],"cudagraph_capture_sizes":[1,2,3,4,5,6,8]}'""",
@@ -107,6 +91,20 @@ args = [
     "--cpu-distributed-timeout-seconds", "3600",
 ]
 
+[composed_model]
+mount = "/root/model"
+
+[[composed_model.sources]]
+repo = "aidendle94/GLM-5.2-MXFP4-Experts-GPTQ"
+
+[[composed_model.sources]]
+repo = "aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ"
+glob = "L*.safetensors"
+
+[[composed_model.files]]
+name = "nf4_layers.json"
+content = '{"layers":[3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77]}'
+
 # Persistent JIT/torch-inductor cache (avoids ~20 min recompilation on cold boot).
 # `alias = "jit_cache"` resolves via [mount_aliases] in config.toml; if absent,
 # falls back to host "~/.cache/jit" (expanded to /home/<ssh_user>/.cache/jit).
@@ -117,37 +115,13 @@ host = "~/.cache/jit"
 container = "/root/.cache/jit"
 env = "XDG_CACHE_HOME"
 
-# MTP draft model — from the MXFP4 base checkpoint's repo. Resolved at serve
-# time, per node, from the HF cache (refs/main → latest snapshot), so the
-# path survives `huggingface-cli download` updates without a fragile symlink.
-# The recipe references {{mtp_draft}} in --speculative-config; the launcher
-# substitutes the resolved full HF cache path per-node (no bind-mount — the
-# HF cache is already mounted at /root/.cache/huggingface, so symlinks
-# resolve correctly).
-[[extra_mounts]]
-repo = "aidendle94/GLM-5.2-MXFP4-Experts-GPTQ"
-subfolder = "mtp-draft"
-container = "mtp_draft"
-
-# Lloyd repo path reference — resolved at serve time, per node, from the HF
-# cache (refs/main → latest snapshot). {{lloyd_experts}} is substituted into
-# VLLM_NF4_LAYERS below so the kernel can find the nf4_layers.json file
-# (created by the operator in step 3 of the prerequisites above). This is a
-# path reference, NOT a bind-mount — the HF cache is already mounted at
-# /root/.cache/huggingface, so the resolved path's symlinks resolve correctly.
-[[extra_mounts]]
-repo = "aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ"
-container = "lloyd_experts"
-
 
 [env]
 # ── NF4 Lloyd expert overlay ──────────────────────────────────────────────
 # Path to a JSON file listing which layers use the nf4_e8m0_k32 decode path
-# (vs the default fp4_e8m0_k32 E2M1 bit-twiddle). The kernel reads this file
-# at init: {"layers": [3, 4, ..., 77]}. {{lloyd_experts}} is substituted
-# per-node with the resolved HF cache path to the Lloyd repo snapshot, where
-# the operator creates nf4_layers.json (see PREREQUISITES step 3 above).
-VLLM_NF4_LAYERS = "{{lloyd_experts}}/nf4_layers.json"
+# (vs the default fp4_e8m0_k32 E2M1 bit-twiddle). Generated by [composed_model]
+# at /root/model/nf4_layers.json — no placeholder needed (known container path).
+VLLM_NF4_LAYERS = "/root/model/nf4_layers.json"
 
 # ── b12x suite ────────────────────────────────────────────────────────────
 VLLM_USE_V2_MODEL_RUNNER = "1"

recipes/test-composed/recipe.toml

@@ -0,0 +1,46 @@
+# Test Composed Model — verifies the [composed_model] mechanism (symlink
+# composition + bind-mount + model rewrite) without launching heavy vLLM.
+#
+# Uses alpine (like test-sleep/test-mods) with a heartbeat entrypoint that
+# first inspects the composed dir, then loops. Requires both model repos
+# cached on every node — the composed dir's symlinks point into the HF cache
+# (container-side /root/.cache/huggingface), so they only resolve inside the
+# container where the HF cache is mounted.
+#
+# What this verifies:
+#   1. The composed dir is built on the node (/tmp/sm121_test-composed_<rank>/)
+#   2. MXFP4 base files are symlinked (config.json, tokenizer, etc.)
+#   3. Lloyd NF4 experts overlay the MXFP4 experts (L*.safetensors)
+#   4. The generated nf4_layers.json file is present
+#   5. The bind-mount makes /root/model/ accessible inside the container
+#   6. model= is rewritten to /root/model (visible in the docker run command)
+#
+# Usage:
+#   cargo run -- -d serve --recipe test-composed    # dry-run: inspect cmd
+#   cargo run serve --recipe test-composed           # real: starts alpine
+#   cargo run logs --recipe test-composed            # see the composed dir listing
+
+engine = "vllm"
+name = "Test Composed Model"
+docker_image = "alpine:latest"
+model = "noop"
+
+[composed_model]
+mount = "/root/model"
+
+[[composed_model.sources]]
+repo = "aidendle94/GLM-5.2-MXFP4-Experts-GPTQ"
+
+[[composed_model.sources]]
+repo = "aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ"
+glob = "L*.safetensors"
+
+[[composed_model.files]]
+name = "nf4_layers.json"
+content = '{"layers":[3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77]}'
+
+entrypoint = ["sh", "-c", "echo '=== composed model test ===' && echo 'dir contents (first 15):' && ls /root/model/ | head -15 && echo '--- nf4_layers.json ---' && cat /root/model/nf4_layers.json && echo '' && echo '=== test OK ===' && i=0; while true; do echo \"$(date -u +%H:%M:%S) [$(hostname)] composed $i\"; i=$((i+1)); sleep 2; done"]
+
+args = []
+
+[env]

src/after_stop.rs

@@ -204,6 +204,7 @@ mod tests {
             shm_size: String::new(),
             extra_mounts: vec![],
             build: Default::default(),
+            composed_model: None,
         }
     }
 

src/before_serve.rs

@@ -314,6 +314,7 @@ mod tests {
             shm_size: String::new(),
             extra_mounts: vec![],
             build: Default::default(),
+            composed_model: None,
         }
     }
 

src/config/types.rs

@@ -167,6 +167,13 @@ pub struct Recipe {
     /// the `versions.env` names the image's `Dockerfile` declares as `ARG`.
     #[serde(default)]
     pub build: RecipeBuild,
+    /// When set, the launcher builds a /tmp composed-model dir per-node
+    /// (symlinks into the HF cache + generated files), bind-mounts it at
+    /// `mount`, and rewrites `model` to that path. Lets a recipe combine
+    /// multiple HF repos (e.g. a base checkpoint + an expert overlay)
+    /// without mutating the HF cache. See [`ComposedModel`].
+    #[serde(default)]
+    pub composed_model: Option<ComposedModel>,
     /// Extra bind-mounts to add to `docker run` (e.g. persistent JIT cache).
     /// Each entry resolves its host path from `Config.mount_aliases` when
     /// `alias` is set and present there (alias wins); otherwise falls back to
@@ -187,6 +194,52 @@ pub struct RecipeBuild {
     pub versions: HashMap<String, String>,
 }
 
+/// A composed model directory: the launcher builds a `/tmp` dir per-node
+/// with symlinks into the HF cache (container-side absolute paths) +
+/// generated files, then bind-mounts it into the container. Lets a recipe
+/// combine multiple HF repos (e.g. a base checkpoint + an expert overlay)
+/// without mutating the HF cache.
+///
+/// Symlinks point to `/root/.cache/huggingface/...` (the container-side HF
+/// cache mount), not the host-side path. They are dangling on the host but
+/// resolve correctly inside the container where the HF cache is mounted at
+/// that exact path — avoiding the host↔container mount-root mismatch.
+#[derive(Clone, Debug, Default, Deserialize)]
+pub struct ComposedModel {
+    /// Container-side mount target for the composed dir (e.g. `/root/model`).
+    /// `recipe.model` is rewritten to this path at serve time.
+    pub mount: String,
+    /// Source repos to symlink into the composed dir. Later entries win
+    /// (overwrite earlier symlinks), so order matters for overlays.
+    #[serde(default)]
+    pub sources: Vec<ComposedSource>,
+    /// Real files (not symlinks) to write into the composed dir.
+    #[serde(default)]
+    pub files: Vec<ComposedFile>,
+}
+
+/// A source repo for [`ComposedModel`]. The repo's HF cache snapshot is
+/// symlinked into the composed dir, optionally filtered by `glob`.
+#[derive(Clone, Debug, Default, Deserialize)]
+pub struct ComposedSource {
+    /// HF Hub repo id (e.g. `aidendle94/GLM-5.2-MXFP4-Experts-GPTQ`).
+    pub repo: String,
+    /// Optional glob to filter which snapshot files are symlinked (e.g.
+    /// `L*.safetensors` — only expert weight files). Empty/omitted = all
+    /// files. Only glob-safe characters are allowed (alphanumeric + `*?._-`).
+    #[serde(default)]
+    pub glob: Option<String>,
+}
+
+/// A generated file written into the composed dir (not a symlink).
+#[derive(Clone, Debug, Default, Deserialize)]
+pub struct ComposedFile {
+    /// Filename in the composed dir (e.g. `nf4_layers.json`).
+    pub name: String,
+    /// Raw file content (written verbatim via base64).
+    pub content: String,
+}
+
 impl Config {
     pub fn resolved_hf_cache_dir(&self) -> std::path::PathBuf {
         if let Some(ref dir) = self.hf_cache_dir {

src/docker/command.rs

@@ -391,6 +391,7 @@ mod tests {
             docker_image_dir: None,
             extra_mounts: vec![],
             build: Default::default(),
+            composed_model: None,
         }
     }
 

src/docker/compose.rs

@@ -0,0 +1,146 @@
+//! Composed model directories: build a /tmp dir per-node with symlinks into
+//! the HF cache (container-side absolute paths) + generated files, then
+//! bind-mount it into the container. Lets a recipe combine multiple HF repos
+//! (e.g. a base checkpoint + an expert overlay) without mutating the HF cache.
+//!
+//! Symlinks point to `/root/.cache/huggingface/...` (the container-side HF
+//! cache mount path), NOT the host-side path. They are dangling on the host
+//! (the path `/root/.cache/...` doesn't exist there) but resolve correctly
+//! inside the container, where the HF cache is mounted at that exact path.
+//! This avoids the host↔container mount-root mismatch without requiring
+//! relative-path gymnastics across different mount roots.
+
+use super::shq;
+use crate::config::{ComposedModel, ComposedSource};
+use crate::ssh::run_ssh;
+use anyhow::{Context, Result};
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD;
+use std::path::Path;
+use tracing::info;
+
+/// Build a composed model directory on a node. Creates
+/// `/tmp/sm121_<recipe>_<rank>/` with symlinks into the HF cache
+/// (container-side absolute paths) + generated files. Returns
+/// `(host_path, container_path)` for the bind-mount.
+pub(crate) async fn build_composed_dir_on_node(
+    config: &crate::config::Config,
+    node: &crate::config::Node,
+    recipe_name: &str,
+    rank: u32,
+    composed: &ComposedModel,
+    hf_cache: &Path,
+    dry_run: bool,
+) -> Result<(String, String)> {
+    let host_path = format!("/tmp/sm121_{}_{}", recipe_name, rank);
+    let container_path = composed.mount.clone();
+
+    if dry_run {
+        println!(
+            "[dry-run] would build composed model dir '{}' on '{}' \
+             (mount: {}, {} sources, {} files)",
+            host_path,
+            node.name,
+            container_path,
+            composed.sources.len(),
+            composed.files.len()
+        );
+        return Ok((host_path, container_path));
+    }
+
+    let cache_str = hf_cache.display().to_string();
+
+    // Create the composed dir.
+    run_ssh(config, node, format!("mkdir -p {}", shq(&host_path)), false).await?;
+
+    // Symlink each source's snapshot files into the composed dir.
+    for source in &composed.sources {
+        symlink_source(config, node, source, &host_path, &cache_str).await?;
+    }
+
+    // Write generated files (base64-encoded to avoid quoting issues).
+    for file in &composed.files {
+        let b64 = STANDARD.encode(file.content.as_bytes());
+        run_ssh(
+            config,
+            node,
+            format!(
+                "printf '%s' '{}' | base64 -d > {}/{}",
+                b64,
+                shq(&host_path),
+                shq(&file.name),
+            ),
+            false,
+        )
+        .await?;
+    }
+
+    info!(
+        "Built composed model dir '{}' on '{}' ({} sources, {} files)",
+        host_path,
+        node.name,
+        composed.sources.len(),
+        composed.files.len()
+    );
+
+    Ok((host_path, container_path))
+}
+
+/// Resolve a source repo's snapshot on the node and symlink its files into
+/// the composed dir. Symlink targets are container-side absolute paths
+/// (`/root/.cache/huggingface/...`) so they resolve inside the container.
+async fn symlink_source(
+    config: &crate::config::Config,
+    node: &crate::config::Node,
+    source: &ComposedSource,
+    host_path: &str,
+    cache_str: &str,
+) -> Result<()> {
+    let model_dir = crate::config::model_cache_dir_name(&source.repo);
+    let refs_path = format!("{}/hub/{}/refs/main", cache_str, model_dir);
+    let glob = source.glob.as_deref().unwrap_or("*");
+
+    // Validate the glob: only allow glob-safe characters to prevent shell
+    // injection (the glob is inserted raw into the shell `for` statement).
+    if !glob
+        .chars()
+        .all(|c| c.is_ascii_alphanumeric() || "*?._-".contains(c))
+    {
+        anyhow::bail!(
+            "composed_model source '{}': glob '{}' contains invalid characters \
+             (allowed: alphanumeric, * ? . _ -)",
+            source.repo,
+            glob
+        );
+    }
+
+    // One SSH round-trip: resolve refs/main, then symlink each matching file.
+    // The symlink TARGET is the CONTAINER-side path (/root/.cache/huggingface/...)
+    // not the host-side path — the link is dangling on the host but resolves
+    // inside the container where the HF cache is mounted at that exact path.
+    // `ln -snf`: -s symlink, -n no-deref (treat existing symlink as a file,
+    // don't dive into the dir it points to), -f force (overwrite earlier).
+    let cmd = format!(
+        "ref=\"$(cat {refs_q} 2>/dev/null)\" && test -n \"$ref\" && \
+         snap=\"{cache}/hub/{model}/snapshots/$ref\" && \
+         test -d \"$snap\" && \
+         csnap=\"/root/.cache/huggingface/hub/{model}/snapshots/$ref\" && \
+         for f in \"$snap\"/{glob}; do \
+           [ -e \"$f\" ] || [ -L \"$f\" ] || continue; \
+           ln -snf \"$csnap/$(basename \"$f\")\" {host_q}/; \
+         done",
+        refs_q = shq(&refs_path),
+        cache = shq(cache_str),
+        model = shq(&model_dir),
+        host_q = shq(host_path),
+    );
+
+    run_ssh(config, node, cmd, false).await.with_context(|| {
+        format!(
+            "composed_model source '{}' not found in HF cache on '{}' \
+             — run `huggingface-cli download {}` on that node",
+            source.repo, node.name, source.repo
+        )
+    })?;
+    Ok(())
+}

src/docker/mod.rs

@@ -1,5 +1,6 @@
 pub mod build;
 pub mod command;
+pub mod compose;
 pub mod deploy;
 pub mod health;
 pub mod logs;
@@ -233,9 +234,10 @@ pub async fn stop_cluster(
             config,
             node,
             format!(
-                "docker ps -q --filter {} --filter {} | xargs -r docker stop",
+                "docker ps -q --filter {} --filter {} | xargs -r docker stop; rm -rf {} 2>/dev/null; true",
                 shq(&format!("label=sm121.recipe={}", recipe_name)),
-                shq(&format!("label=sm121.cluster={}", cluster.name))
+                shq(&format!("label=sm121.cluster={}", cluster.name)),
+                shq(&format!("/tmp/sm121_{}_{}", recipe_name, node.rank)),
             ),
             dry_run,
         )
@@ -378,9 +380,32 @@ async fn start_cluster(
                 };
                 resolved_refs.push((spec.name.clone(), resolved_path));
             }
-            let recipe = command::substitute_path_refs(&dm_ctx.recipe, &resolved_refs);
+            let mut recipe = command::substitute_path_refs(&dm_ctx.recipe, &resolved_refs);
+            let mut extra_mounts = dm_ctx.extra_mounts.clone();
+            // Build composed model dir if configured: symlinks into the HF
+            // cache + generated files, bind-mounted at `mount`. `model` is
+            // rewritten to the mount path so vLLM loads from the composed dir.
+            if let Some(ref cm) = recipe.composed_model {
+                match compose::build_composed_dir_on_node(
+                    &config, &node, &rn, node.rank, cm, &hf_cache, dry,
+                )
+                .await
+                {
+                    Ok((hp, cp)) => {
+                        extra_mounts.push(command::ResolvedMount {
+                            host: hp,
+                            container: cp.clone(),
+                            read_only: false,
+                            env: None,
+                        });
+                        recipe.model = cp;
+                    }
+                    Err(e) => return (node.name, node.rank, engine, Err(e)),
+                }
+            }
             let ctx = command::DockerCommandCtx {
                 recipe,
+                extra_mounts,
                 ..dm_ctx.clone()
             };
             let cmd = command::build_docker_command(&ctx, &node, is_head).join(" ");

src/docker/tests.rs

@@ -67,6 +67,7 @@ fn trec() -> Recipe {
         docker_image_dir: None,
         extra_mounts: vec![],
         build: Default::default(),
+        composed_model: None,
     }
 }
 
@@ -255,6 +256,7 @@ fn sglang_recipe() -> Recipe {
         docker_image_dir: None,
         extra_mounts: vec![],
         build: Default::default(),
+        composed_model: None,
     }
 }
 

src/doctor.rs

@@ -377,6 +377,7 @@ mod tests {
             shm_size: String::new(),
             extra_mounts: vec![],
             build: Default::default(),
+            composed_model: None,
         }
     }