launcher/specs/tech-architecture/engine-drivers.md

First-class engine drivers (EngineDriver trait)

Goal

Make third-party inference engines (gr8, DGPP, future ones) first-class engine values in a recipe — served/stopped/logged/health-probed/benched by the same sm121 commands as vllm/sglang — instead of hand-deployed outside the launcher. The operator directive: engines like dgpp/gr8 must be "равноправные члены наравне с vllm/sglang/tmux".

Current state (the scattered branches this replaces)

engine is a free String (default "vllm"); only vllm/sglang have any launcher logic, and it is duplicated across six sites, each branching on recipe.engine == "sglang":

Site	Engine-specific behaviour
engine_flags.rs::build_engine_args	model arg (--model-path vs positional), distributed rendezvous flags (vLLM set vs SGLang set), --headless for vLLM workers
docker/command.rs::collect_nccl_env	worker-IP env: SGLANG_WORKER_IP vs VLLM_HOST_IP
tmux/commands.rs	entrypoint fallback (python -m vllm… / python -m sglang…), port on all nodes (SGLang) vs head-only (vLLM)
tmux/mod.rs::build_host_ip_export	host-IP env var per engine
doctor.rs	install hint (pip install vllm / sglang)
publish_render.rs	SGLang special-case in the rendered bundle

Design

Three orthogonal axes, each a first-class concept:

1. engine → an EngineDriver (the inference-engine semantics).
2. deploy → Docker | Tmux | Native (new) — how the process is started on a host (container / tmux session / bare SSH command).
3. launch_shape → PerNode (every launch node runs the engine; vLLM, SGLang, gr8) | HeadCoordinated (rank 0 runs a cluster manager that brings up the peers itself; DGPP).

EngineDriver trait (in src/engine.rs)

// Head-coordination is the deploy target (`deploy = "native"` = rank 0 runs
// the engine's cluster manager), not a separate trait axis.
pub trait EngineDriver: Send + Sync {
    // Model argument (positional for vLLM, --model-path for SGLang,
    // --model for gr8, none for DGPP — the cluster JSON carries it).
    fn model_args(&self, recipe: &Recipe) -> Vec<String>;
    // Distributed rendezvous flags (vLLM/SGLang/gr8 sets; DGPP: none — the
    // JSON carries world_size + rendezvous).
    fn distributed_args(&self, ctx: &EngineFlagsCtx, node: &Node) -> Vec<String>;
    // Non-head workers run headless (vLLM yes; SGLang/gr8/DGPP no).
    fn headless_workers(&self) -> bool { false }
    // Env var carrying the node's bind IP (VLLM_HOST_IP / SGLANG_WORKER_IP / none).
    fn engine_ip_env(&self) -> Option<&'static str> { None }
    // Takes --host/--port (vLLM head, SGLang all). Engines with a fixed API
    // port (gr8 :8100, DGPP :18080) return false — no --host/--port, and
    // `validate` exempts them from the port-convention rule.
    fn takes_host_port(&self) -> bool { true }
    // tmux entrypoint fallback when the recipe sets none.
    fn entrypoint_default(&self) -> Vec<String>;
    // doctor install hint.
    fn install_hint(&self) -> &'static str;
}

A registry driver_for(engine: &str) -> &dyn EngineDriver maps the recipe's engine string to a driver (vllm, sglang, gr8, dgpp; unknown → the vLLM driver, matching today's default). build_engine_args and the five other sites delegate to it. Phase 1 is a pure refactor — vLLM/SGLang drivers reproduce the current behaviour byte-for-byte, all existing tests stay green.

EngineFlagsCtx gains peers

gr8 takes --peers <ip0>,<ip1>,… (the ordered launch-node IPs; each rank finds itself in the list). Add peers: &[String] to the ctx (vLLM/SGLang ignore it).

gr8 driver (kindlingai/gr8)

• deploy = docker (image ghcr.io/kindlingai/gr8), per-node launch.
• entrypoint ["/opt/gr8/gr8", "glm53", "up"]; model_args → --model <repo>; distributed_args → --tp <N> --peers <list>; no --headless, no engine-IP env, takes_host_port = false (API :8100 on the first peer).
• Recipe carries --model-cache, --drafter, --shape in args; the vm.compaction_proactiveness=0 prerequisite is a before_serve host hook.
• Stop: label-based docker stop works (the container is the engine).

DGPP driver (HawkBearPig/dgpp)

• deploy = native (new), head-coordinated: rank 0 runs the manager.
• The recipe's [dgpp] block carries http_port, dir (the DGPP checkout on the nodes), optional release, and a free-form [dgpp.engine] table. The launcher generates DGPP's cluster JSON (model, world_size, http, and the engine table serialized verbatim) plus a sibling .env (DGPP_NODES/DGPP_SSH_USER/ports), uploads both to <dir> on rank 0, and runs scripts/dgpp-cluster up --config <file> there (nohup, log under <dir>/log/). Stop runs scripts/dgpp-cluster down on rank 0.
• Readiness is the uniform /v1/models probe on http_port (default 18080).

DGPP provisioning (native analog of an on-demand image build)

A recipe pins [dgpp].repo (git URL) + [dgpp].rev (commit/tag). Before launch, serve provisions DGPP — src/native/provision.rs:

1. Stage first: the generated cluster JSON + site .env are written to EVERY launch node (dgpp-cluster install validates DGPP_NODES per node; DGPP_NODES is space-separated, rank order).
2. Provision on rank 0 only — dgpp-cluster install is head-coordinated: it installs the release on rank 0 AND SSHes to the peers (from DGPP_NODES) to install there too (the docker build-once + sync analog). Running it on every node double-installs the peers ("release already exists").
3. Skip key: <dir>/.sm121-dgpp-rev (the marker, like the docker sm121.checksum label). If it already equals rev, the pass is a no-op.
4. Else, in a detached tmux session (sm121-dgpp-<recipe>, log /tmp/<recipe>-dgpp-build.log, 30 s poll / 90 min cap — same shape as docker/build.rs): the build script is written to a file and run as bash <file> 2>&1 | tee <log> (inline cmd && ... && echo DONE | tee would pipe only the final echo). The script: git clone/fetch → git checkout <rev> → scripts/release.sh (CUDA kernels → dist/dgpp-*.tar.zst) → dgpp-cluster install dist/dgpp-*.tar.zst --config <staged json> → echo <rev> > .sm121-dgpp-rev → echo SM121-DGPP-DONE.
5. install is not idempotent ("release already exists"): the script tolerates it via || dgpp-cluster releases --config <json> | grep -qF <rev-abbrev>, so a re-serve at the same pin (marker lost) is still safe. set -e + the marker/DONE last: a genuinely failed build leaves no marker → retry.

No [dgpp].repo/rev → assume DGPP is pre-installed. Bumping rev cuts a new generation (the marker keys on it). dir is ~-expanded to $HOME.

DGPP deployment prereqs (validated live 2026-10-15, w2 on 2nodes-dac): provisioning + dgpp-cluster up work; DGPP's own preflight then gates on things the launcher does NOT own — the model checkpoint must be in the node's HF cache (python3 scripts/download_model.py --config <file> on rank 0), and multi-fabric hosts need DGPP_ROCE_DEVICES set explicitly ("transport supports at most two lanes"). These are DGPP-side deployment steps, not launcher bugs.

Phasing

1. Trait + vLLM/SGLang drivers — pure refactor, six call-sites delegate, all tests green. ✅ (launcher engine: Phase 1 commit)
2. gr8 driver + peers ctx + takes_host_port + a glm53-full-tp4-gr8 recipe; validate + dry-run render the exact gr8 glm53 up contract. ✅
3. Native deploy (head-coordinated) + DGPP driver + [dgpp] config-JSON generation + a qwen38-27b-nvfp4-dgpp recipe; dry-run renders the JSON/.env upload + dgpp-cluster up/down. ✅ (native serve/stop live-tested on first real DGPP deploy)
4. New-model recipes (Ling-3.1-flash, Step-5-Preview) — separate, gated on the bleeding-edge-base decision (nightly/dedicated image, 5–8 Sparks).

Constraints

• Files ≤500 lines — engine.rs holds the trait + registry + the four drivers; split drivers into engine/gr8.rs etc. if it grows.
• No #[allow] for clippy — refactor instead.
• i18n: any new user-facing strings get keys in all six locales.