# AGENTS.md — vllm-spark-launcher Rust CLI tool for launching vLLM/SGLang containers across a DGX Spark cluster. SSH-based provisioning over `russh`. Supports two runtimes: **Docker** (default) and **tmux** (alternative). ## Project Layout - `Cargo.toml` — binary crate `vllm-spark-launcher`. Edition 2024. - `src/main.rs` — CLI entrypoint, command dispatch, i18n init. - `src/cli.rs` — `clap` derive-based CLI definitions (Commands, PowerAction, TmuxCommands). - `src/config.rs` — Config, Cluster, Node, Recipe, NcclConfig, HomeAssistantConfig deserialization; cluster resolution (exact + prefix match, auto-select smallest). - `src/ssh.rs` — SSH client session management, key loading, host key verification, retry logic. - `src/docker/mod.rs` — Docker command builder: serve, stop, health, logs, log streaming. - `src/docker/tests.rs` — Docker command builder tests (MP, SGLang, Ray, keep, port, env overrides). - `src/tmux/mod.rs` — tmux module root (session manager, serve, ops). - `src/tmux/serve.rs` — tmux serve logic (window/panel layout). - `src/tmux/ops.rs` — tmux stop/status/logs/attach operations. - `src/health.rs` — tmux health-check helpers. - `src/power.rs` — `power on/off/switch` flows (SSH shutdown + HA toggle). - `src/home_assistant.rs` — HA REST API client (turn_on/turn_off). - `src/host_keys.rs` — `init-host-keys` command (auto-write SSH host_keys). - `src/exec.rs` — `exec` subcommand: run commands on cluster/nodes, resolve aliases. - `src/doctor.rs` — Doctor mode: pre-flight checks and `--repair` (Dockerfile generation). - `src/pre_serve.rs` — BeforeServe/PreServeCommand types. - `src/before_serve.rs` — Pre-serve hook execution logic. - `locales/{en,ru,es,it,fr,de}/main.ftl` — Fluent i18n translation files. - `config.toml` — runtime config: SSH key, cluster definitions, per-cluster NCCL settings, HA, exec aliases. - `recipes//recipe.toml` — recipe files. ## Commands ### Docker runtime ``` cargo run serve --recipe glm51 # Auto-cluster selection cargo run serve --recipe glm51 --cluster 4nodes-switch # Explicit cluster cargo run serve --recipe glm51 --skip-before-serve # Bypass pre-serve hooks cargo run -- -d serve --recipe glm51 # Dry-run (print only) cargo run stop --recipe glm51 # Stop containers on all nodes cargo run health # Health check all clusters cargo run health --cluster 2nodes-dac # Single cluster cargo run health --fingerprint # Show host keys for config.toml cargo run init-host-keys # Auto-write SSH host_keys cargo run exec --cluster 4nodes -- "..." # Run on all nodes cargo run exec --cluster 4nodes --node dgx1 -- "uptime" # Run on one node cargo run exec --recipe glm51 -- "cat /tmp/foo" # Use recipe's cluster cargo run exec --cluster 4nodes --alias gpu-lock # Use exec alias cargo run logs --recipe glm51 # Last 100 lines per node cargo run logs --recipe glm51 --follow # Follow head node logs cargo run doctor --recipe glm51 # Check node prerequisites cargo run doctor --recipe glm51 --repair # Auto-fix (build repaired image) ### Power management ``` cargo run power off --cluster 4nodes-switch # SSH shutdown + HA turn_off cargo run power off --cluster 4 --node dgx1 # Prefix matching + single node cargo run power on --cluster 4nodes-switch # HA turn_on (wake nodes) cargo run power switch --from 4 --to 2 # Switch clusters (disable → enable) cargo run -- -d power off --cluster 4nodes-switch # Dry-run ``` ### SGLang recipe example ``` cargo run serve --recipe minimax-m3 # SGLang, 4 nodes, TP=4 cargo run -- -d serve --recipe minimax-m3 # Dry-run (see the commands) cargo run stop --recipe minimax-m3 # Stop SGLang containers cargo run logs --recipe minimax-m3 # Check SGLang logs ``` ### Tmux runtime ``` cargo run tmux serve --recipe glm51 # Launch via tmux (no Docker) cargo run tmux stop --recipe glm51 # Stop tmux sessions cargo run tmux status --recipe glm51 # Detailed health per node cargo run tmux logs --recipe glm51 # Capture pane output cargo run tmux logs --recipe glm51 --follow # Stream pane output cargo run tmux attach --recipe glm51 # Interactive attach (head node) ``` All subprocess calls support `-d` / `--dry-run` as a global flag. ## config.toml User-specific. Has hardcoded SSH key paths — won't work without editing. Structure: ```toml ssh_user = "username" ssh_key_path = "/absolute/path/to/key" [[clusters]] name = "my-cluster" master_ib_ip = "1.2.3.4" nodes = [ # host_key = "SHA256:..." # run 'cargo run health --fingerprint' to discover { name = "dgx1.lan", ssh_ip = "10.0.0.1", ib_ip = "1.2.3.4", rank = 0, entity = "switch.dgx1" }, { name = "dgx2.lan", ssh_ip = "10.0.0.2", ib_ip = "1.2.3.5", rank = 1, entity = "switch.dgx2" }, ] extra_entities = ["switch.network_switch"] # cluster-level switches (PDU, network) [nccl."my-cluster"] ib_hca = "rocep1s0f0,roceP2p1s0f0" socket_ifname = "enp1s0f0np0" ib_gid_index = 3 ib_disable = 0 ``` ### Home Assistant config Optional `[home_assistant]` section for power management: ```toml [home_assistant] url = "http://homeassistant.local:8123" token = "long-lived-access-token" shutdown_delay_seconds = 30 ``` ### Exec aliases ```toml [exec_aliases] gpu-lock = "sudo nvidia-smi -lgc 300,2300" shutdown = "sudo shutdown now" ``` Use: `cargo run exec --cluster 4 --alias gpu-lock` ### Recipe-level cluster override A recipe can pin itself to a specific cluster (bypasses `--cluster` / auto-pick): ```toml cluster = "4nodes-switch" ``` ### Pre-serve hooks (`[before_serve.""]`) Run shell commands on every node of a cluster **before** `serve` / `tmux serve` launches containers or tmux sessions. Defined in `config.toml`: ```toml [before_serve."4nodes-switch"] commands = [ # Default: abort the serve if this fails "sudo nvidia-smi -lgc 300,2300", # Explicit on_failure: # "abort" — fail-fast (default) # "warn" — log warning, continue # "continue" — stay silent, continue { command = "sudo nvidia-smi -pm 1", on_failure = "warn" }, ] ``` Recipes can also declare pre-serve commands that **append** to the cluster config (so recipe-level commands run after cluster-level): ```toml before_serve = [ { command = "echo recipe-level", on_failure = "abort" }, ] ``` Commands are run sequentially on each node (so order matters) but nodes run in parallel. Use `--skip-before-serve` to bypass for a one-off serve. ### `exec` aliases (`[exec_aliases]`) Map short names to full shell commands for use with `cargo run exec --alias`. Defined in `config.toml`: ```toml [exec_aliases] gpu-lock = "sudo nvidia-smi -lgc 300,2300" shutdown = "sudo shutdown now" reboot = "sudo reboot" ``` Then: ```bash cargo run exec --cluster 4nodes --alias gpu-lock cargo run exec --cluster 4nodes --alias shutdown ``` `--alias ` and a trailing command are mutually exclusive. If the alias name is unknown, the error message lists the configured aliases. ## recipes/ TOML format. Directory name is intentionally misspelled. Each recipe lives in its own subdirectory: ```toml engine = "vllm" # or "sglang" (default: "vllm") name = "Human Readable Name" docker_image = "my-image:tag" model = "org/model-name" port = 30000 # default: 30000 required_nodes = 4 # optional, omitted = no check pipeline_parallel_size = 0 # 0 = auto (equals nnodes), vLLM only tensor_parallel_size = 1 # per-node TP for vLLM, total TP for SGLang distributed_backend = "mp" # "mp" (multiprocessing) or "ray" shm_size = "64gb" # optional, only added when non-empty cluster = "4nodes-switch" # optional: pin to a specific cluster args = [ "--kv-cache-dtype", "fp8", "--max-model-len", "32768", ] entrypoint = ["vllm", "serve"] # optional, appended after image name before_serve = [ # recipe-level pre-serve hooks (after cluster) { command = "echo recipe", on_failure = "abort" }, ] [env] SOME_VAR = "value" ``` ### Engine support Both `vllm` and `sglang` engines are supported. The engine field controls: **Host IP env var:** - `vllm` → `-e VLLM_HOST_IP=` - `sglang` → `-e SGLANG_WORKER_IP=` **Model argument:** - `vllm` → bare model name (e.g. `org/model`) - `sglang` → `--model-path org/model` **Distributed flags:** - `vllm` → `--tensor-parallel-size 1`, `--pipeline-parallel-size `, `--distributed-executor-backend mp|ray` - `sglang` → `--tp `, no `--pipeline-parallel-size`, no `--distributed-executor-backend` Both engines use `--nnodes`, `--node-rank`, and `--master-addr`. ## How Docker launch works `build_docker_command` in `src/docker.rs` constructs the `docker run` command for each node. It is engine-aware: ### vLLM engine (default) Appends these flags: ``` --tensor-parallel-size --pipeline-parallel-size --nnodes --node-rank --master-addr --master-port # default 25000 --distributed-executor-backend ray # only for ray backend (mp is auto-detected) --host 0.0.0.0 --port ``` Non-head nodes also receive `--headless`. ### SGLang engine Appends: ``` --tp --nnodes --node-rank --dist-init-addr : --host 0.0.0.0 --port ``` ### Ray backend For vLLM Ray backend, additionally appends: ``` -e RAY_ADDRESS=:6379 -e RAY_PORT= -e RAY_NODE_IP_ADDRESS= -e RAY_OVERRIDE_NODE_IP_ADDRESS= ``` ### Env vars passed to every container **Cluster-driven** (from `[nccl.""]`): `NCCL_IB_DISABLE`, `NCCL_IB_HCA`, `NCCL_SOCKET_IFNAME`, `NCCL_IB_GID_INDEX`, `GLOO_SOCKET_IFNAME`, `TP_SOCKET_IFNAME` **Network interface** (from `nccl.socket_ifname`): `MN_IF_NAME`, `UCX_NET_DEVICES`, `OMPI_MCA_btl_tcp_if_include` **Defaults**: `OMP_NUM_THREADS=4`, `NCCL_IGNORE_CPU_AFFINITY=1`, `NCCL_NET=IB`, `NCCL_DEBUG=WARN` **Engine IP** (from `node.ib_ip`): `VLLM_HOST_IP` (vLLM) or `SGLANG_WORKER_IP` (SGLang) **Head node only**: `-e VLLM_LOGGING_COLOR=1 -e TERM=xterm-256color` (colored logs) Recipe `[env]` entries are appended last and override any default. ### Common to all runs NCCL env vars come from the cluster's `[nccl.""]` section. Port mapping `-p :` on the head node only. `HF_HOME=/root/.cache/huggingface` with volume mount. Containers are labelled: - `sm121.recipe=` - `sm121.rank=` `--cluster auto` picks the **smallest** cluster (fewest nodes) that matches the recipe's `required_nodes` and has an NCCL config. Prefix matching works: `--cluster 4` matches `4nodes-switch`. ## How tmux launch works `tmux serve` builds sessions on each node with a 3-window layout: | Window | Content | |--------|---------| | 0: `model` | Main inference process (vLLM/SGLang) | | 1: `gpu` | `nvidia-smi -l 5` GPU monitor | | 2: `shell` | Debug shell | For **Ray backend**, nodes start sequentially: head node first (polled for API readiness up to 150s), then workers. For **MP backend**, all nodes start in parallel. ## Doctor mode `cargo run doctor --recipe ` checks each node for: - Docker availability - NVIDIA GPU (nvidia-smi) - NCCL libraries - Engine version inside the image (`vllm version` / `sglang version`) - Ray availability (if backend is "ray") - Container entrypoint With `--repair`, generates a Dockerfile that installs missing packages and rebuilds the image with a `.repaired` tag. The repaired image is auto-detected on subsequent `serve` runs. ## Dependencies - `clap` 4.6 + `derive` — derive-based CLI - `russh` 0.61 + `russh-keys` 0.50-beta — SSH client - `tokio` full — async runtime - `toml` 1 + `serde` 1 — config/recipe deserialization - `tracing` + `tracing-subscriber` — logging - `anyhow` — error handling - `futures` 0.3 — `join_all` for concurrent SSH - `sha2` 0.11 — SHA-256 for host key fingerprints - `base64` 0.22 — Base64 encoding for fingerprints - `dirs` 6 — OS config dir for default config fallback - `reqwest` 0.13 — HTTP client for Home Assistant REST API - `serde_json` 1 — JSON for HA requests - `fluent-i18n` — i18n via Mozilla Fluent (locales in `locales/`)