# AGENTS.md — sm121 Rust CLI (`edition 2024`) for launching vLLM/SGLang inference containers across a DGX Spark cluster via SSH (`russh`). Docker default, tmux alternative. ## Dev quick-ref | Action | Command | |--------|---------| | Build | `cargo build` | | All gates | `make check` (fmt → clippy → test → ≤500 lines) | | Dry-run | `cargo run -- -d serve --recipe deepseek-v4-flash` | | Tests | `cargo test` (65, all passing) | | l10n | `locales/{en,ru,es,it,fr,de}/main.ftl` — Fluent FTL files | See [`DEVELOPMENT.md`](DEVELOPMENT.md) for architecture, module layout, and dev workflow. ## Commands ``` serve/stop/logs/health/doctor/exec/power/tmux/init-host-keys ``` Cluster: exact name, prefix (`4`→`4nodes-switch`), or `auto` (smallest with NCCL + matching `required_nodes`). `serve --solo` overrides cluster selection: finds the first free node across all clusters and runs one single-node container there (no `--nnodes`/`--tp`/ `--master-addr` flags — the engine defaults to single-node). `--no-solo` forces multi-node even if the recipe sets `solo = true`. A recipe can also set `solo = true` as the default. > ⚠️ **Cluster may be serving live traffic.** A model (often the one serving this > very agent) may be running on the cluster at any time. Before running `stop`, > `power off`, or `power switch`, ALWAYS first run `health` and confirm with the > user that the cluster is not serving before proceeding. Do NOT assume the > cluster is idle — a stray `stop`/`power off` has killed running agents mid-task > before. `logs`, `health`, `doctor`, and dry-runs (`-d`) are always safe. ## Recipe format (`recipes//recipe.toml`) ```toml engine = "vllm" # or "sglang" docker_image = "img:tag" model = "org/model" port = 30000 # default required_nodes = 2 # optional; also caps the launch set (N nodes, not all) solo = false # optional; single-node on first free node, no dist flags tensor_parallel_size = 2 pipeline_parallel_size = 1 # vLLM only, 0 = auto distributed_backend = "mp" # or "ray" shm_size = "64gb" # optional, only set when non-empty entrypoint = ["vllm", "serve"] # optional, after image before_serve = [{ command = "echo", on_failure = "abort" }] after_stop = [{ script = "restore_dgx_services" }] # runs after `sm121 stop` args = ["--kv-cache-dtype", "fp8"] [[extra_mounts]] # extra bind-mounts for `docker run -v` alias = "jit_cache" # resolves via [mount_aliases] in config.toml; wins over `host` host = "~/.cache/jit" # fallback default if alias absent; `~` expanded to ssh_user home container = "/root/.cache/jit" # container-side target (required) read_only = false # optional, default false (adds :ro) env = "XDG_CACHE_HOME" # optional: injects -e = # Model path reference (mutually exclusive with alias/host): resolves at # serve time, per node, from the HF cache — refs/main → latest snapshot, so # the path survives `huggingface-cli download` updates with no fragile # symlink. `container` is the placeholder name (e.g. `mtp_draft`) that the # recipe references as `{{mtp_draft}}` in args/env. The launcher substitutes # the resolved **container-side** HF cache path per-node — the host cache-dir # prefix (e.g. `/home//.cache/huggingface`) is rewritten to the # container mount target `/root/.cache/huggingface` (no bind-mount — the HF # cache is already mounted there, so symlinks resolve correctly). # Absent subfolder → serve aborts with a hint to `huggingface-cli download `. [[extra_mounts]] repo = "aidendle94/GLM-5.2-Hybrid-FP8-MXPF4" subfolder = "mtp-draft" # optional; omit = whole snapshot dir container = "mtp_draft" [mods] commands = [ # run *inside* container before engine "pip install flash-attn", { command = "./patch.sh", on_failure = "warn" }, ] [env] KEY = "val" # overrides launcher defaults ``` ## Env vars (auto-set) Cluster-driven: `NCCL_IB_DISABLE/HCA/SOCKET_IFNAME/GID_INDEX`, `GLOO_/TP_SOCKET_IFNAME`, `MN_IF_NAME`, `UCX_NET_DEVICES`, `OMPI_MCA_btl_tcp_if_include` Defaults: `OMP_NUM_THREADS=4`, `NCCL_IGNORE_CPU_AFFINITY=1`, `NCCL_NET=IB`, `NCCL_DEBUG=WARN` Engine IP: `VLLM_HOST_IP` / `SGLANG_WORKER_IP` Head: `VLLM_LOGGING_COLOR=1`, `TERM=xterm-256color`, `-p :` Ray: `RAY_ADDRESS/PORT/NODE_IP_ADDRESS/OVERRIDE_NODE_IP_ADDRESS` Recipe `[env]` wins (last wins in `-e`). ## Mods Mods run **inside** the container before the engine process via entrypoint wrapping. When `[mods]` is present, the recipe **must** also set `entrypoint` (the known process to `exec` at the end). Same `on_failure` semantics as `before_serve` (default: `abort`). On failure the container exits; configure `docker run --restart` externally to restart on crash. Implementation: `--entrypoint sh image -c 'mods && exec '` ## Built-in scripts (baked into the binary) Host scripts under `host-scripts/` are embedded at compile time (`include_str!` in `src/builtin_scripts.rs`) and can be referenced by **alias** from any `before_serve` / `after_stop` list. The launcher uploads + runs them via SSH per node — no `scp` from the operator's machine. Entry shape (untagged enum, three forms): ```toml before_serve = [ "sudo nvidia-smi -lgc", # bare string = raw shell command { command = "echo hi", on_failure = "warn" }, # raw command + on_failure { script = "stop_dgx_services" }, # built-in alias { script = "stop_dgx_services", on_failure = "warn" }, ] ``` A bare string is always a raw command (today's behavior, unchanged); a built-in is referenced only via `{ script = "" }`. Unknown aliases error at resolution time with the list of known ones. Built-in aliases: | Alias | File | Purpose | |-------|------|---------| | `stop_dgx_services` | `host-scripts/stop-dgx-services.sh` | Stop non-essential services on a DGX node (frees unified memory) | | `restore_dgx_services` | `host-scripts/restore-dgx-services.sh` | Restore services stopped by `stop_dgx_services` | To add a built-in: drop the `.sh` into `host-scripts/`, add an `include_str!` + `BuiltinScript` entry in `src/builtin_scripts.rs`, document the alias here. Scripts stay diffable / shellcheck-able as real files. ## Flags auto-generated - vLLM: `--tensor-parallel-size`, `--pipeline-parallel-size`, `--nnodes`, `--node-rank`, `--master-addr`, `--master-port`, `--host`, `--port`, `--headless` (workers) - SGLang: `--tp`, `--nnodes`, `--node-rank`, `--dist-init-addr`, `--host`, `--port` - Both: `--distributed-executor-backend` (ray only, mp is auto-detected) ## Config fields ```toml ssh_user = "u" ssh_key_path = "/path" # PathBuf [[clusters]] name = "c" master_ib_ip = "10.0.0.1" nodes = [{ name = "n", ssh_ip = "...", ib_ip = "...", rank = 0, entity = "switch.n" }] extra_entities = ["switch.network"] # cluster-level HA entities [nccl."c"] ib_hca = "..." socket_ifname = "..." ib_gid_index = 3 ib_disable = 0 [home_assistant] url = "..." token = "..." shutdown_delay_seconds = 30 [exec_aliases] name = "command" [mount_aliases] # host paths referenced by name from recipe [[extra_mounts]] jit_cache = "/mnt/nas/ai/jit" # alias wins over recipe `host`; lets operators repoint mounts site-wide hf_cache = "/mnt/nas/ai/huggingface" [before_serve."c"] commands = ["cmd", { command = "cmd", on_failure = "warn" }] ``` ## Power - `off`: SSH shutdown → wait → HA `turn_off` per-node entities + cluster extras - `on`: HA `turn_on` (no SSH, requires HA config) - `switch --from --to `: disable nodes/extras unique to `from`, enable all in `to` ## i18n `fluent-i18n` crate, auto-detect via `set_locale(None)`. FTL in `locales/`. ## Constraints - Files ≤500 lines (enforced by `make check-lines`); extract submodules when a file approaches the limit - `#![deny(clippy::wildcard_imports)]` — no `use foo::*;` - **Clippy warnings**: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do **not** add `#[allow(...)]` attributes. If the only viable fix is an allow attribute, consult the user for guidance.