# AGENTS.md — sm121 Rust CLI (`edition 2024`) for launching vLLM/SGLang inference containers across a DGX Spark cluster via SSH (`russh`). Docker default, tmux alternative. ## Dev quick-ref | Action | Command | |--------|---------| | Build | `cargo build` | | All gates | `make check` (fmt → clippy → test → ≤500 lines) | | Dry-run | `cargo run -- -d serve --recipe deepseek-v4-flash` | | Tests | `cargo test` (65, all passing) | | l10n | `locales/{en,ru,es,it,fr,de}/main.ftl` — Fluent FTL files | See [`DEVELOPMENT.md`](DEVELOPMENT.md) for architecture, module layout, and dev workflow. ## Commands ``` serve/stop/logs/health/doctor/exec/power/tmux/init-host-keys ``` Cluster: exact name, prefix (`4`→`4nodes-switch`), or `auto` (smallest with NCCL + matching `required_nodes`). `serve --solo` overrides cluster selection: finds the first free node across all clusters and runs one single-node container there (no `--nnodes`/`--tp`/ `--master-addr` flags — the engine defaults to single-node). `--no-solo` forces multi-node even if the recipe sets `solo = true`. A recipe can also set `solo = true` as the default. `serve --force` skips the occupancy check during cluster selection — occupied nodes are kept in the available set (reachable is still required). Use to run a lightweight test recipe alongside a live serve, or to force onto a busy node. Dangerous for GPU recipes (OOM, port conflicts); the operator is responsible for coexistence. > ⚠️ **Cluster may be serving live traffic.** A model (often the one serving this > very agent) may be running on the cluster at any time. Before running `stop`, > `power off`, or `power switch`, ALWAYS first run `health` and confirm with the > user that the cluster is not serving before proceeding. Do NOT assume the > cluster is idle — a stray `stop`/`power off` has killed running agents mid-task > before. `logs`, `health`, `doctor`, and dry-runs (`-d`) are always safe. ## Recipe format (`recipes//recipe.toml`) ```toml engine = "vllm" # or "sglang" docker_image = "img:tag" model = "org/model" port = 30000 # default required_nodes = 2 # optional; also caps the launch set (N nodes, not all) solo = false # optional; single-node on first free node, no dist flags tensor_parallel_size = 2 pipeline_parallel_size = 1 # vLLM only, 0 = auto distributed_backend = "mp" # or "ray" shm_size = "64gb" # optional, only set when non-empty entrypoint = ["vllm", "serve"] # optional, after image docker_image_dir = "glm52-nf3-hybrid" # optional; build dir under docker-images/ (default: recipe name) before_serve = [{ command = "echo", on_failure = "abort" }] after_stop = [{ script = "restore_dgx_services" }] # runs after `sm121 stop` args = ["--kv-cache-dtype", "fp8"] [[extra_mounts]] # extra bind-mounts for `docker run -v` alias = "jit_cache" # resolves via [mount_aliases] in config.toml; wins over `host` host = "~/.cache/jit" # fallback default if alias absent; `~` expanded to ssh_user home container = "/root/.cache/jit" # container-side target (required) read_only = false # optional, default false (adds :ro) env = "XDG_CACHE_HOME" # optional: injects -e = # Model path reference (mutually exclusive with alias/host): resolves at # serve time, per node, from the HF cache — refs/main → latest snapshot, so # the path survives `huggingface-cli download` updates with no fragile # symlink. `container` is the placeholder name (e.g. `mtp_draft`) that the # recipe references as `{{mtp_draft}}` in args/env. The launcher substitutes # the resolved **container-side** HF cache path per-node — the host cache-dir # prefix (e.g. `/home//.cache/huggingface`) is rewritten to the # container mount target `/root/.cache/huggingface` (no bind-mount — the HF # cache is already mounted there, so symlinks resolve correctly). # Absent subfolder → serve aborts with a hint to `huggingface-cli download `. [[extra_mounts]] repo = "aidendle94/GLM-5.2-Hybrid-FP8-MXPF4" subfolder = "mtp-draft" # optional; omit = whole snapshot dir container = "mtp_draft" [mods] commands = [ # run *inside* container before engine "pip install flash-attn", { command = "./patch.sh", on_failure = "warn" }, ] [env] KEY = "val" # overrides launcher defaults [build.versions] # optional; overrides pins in docker-images//versions.env VLLM_COMMIT = "00907fc..." # recipe wins over the image's pin; folded into build checksum ``` ## Env vars (auto-set) Cluster-driven: `NCCL_IB_DISABLE/HCA/SOCKET_IFNAME/GID_INDEX`, `GLOO_/TP_SOCKET_IFNAME`, `MN_IF_NAME`, `UCX_NET_DEVICES`, `OMPI_MCA_btl_tcp_if_include` Defaults: `OMP_NUM_THREADS=4`, `NCCL_IGNORE_CPU_AFFINITY=1`, `NCCL_NET=IB`, `NCCL_DEBUG=WARN` Engine IP: `VLLM_HOST_IP` / `SGLANG_WORKER_IP` Head: `VLLM_LOGGING_COLOR=1`, `TERM=xterm-256color`, `-p :` Ray: `RAY_ADDRESS/PORT/NODE_IP_ADDRESS/OVERRIDE_NODE_IP_ADDRESS` Recipe `[env]` wins (last wins in `-e`). ## Mods Mods run **inside** the container before the engine process via entrypoint wrapping. When `[mods]` is present, the recipe **must** also set `entrypoint` (the known process to `exec` at the end). Same `on_failure` semantics as `before_serve` (default: `abort`). On failure the container exits; configure `docker run --restart` externally to restart on crash. Implementation: `--entrypoint sh image -c 'mods && exec '` ## Built-in scripts (baked into the binary) Host scripts under `host-scripts/` are embedded at compile time (`include_str!` in `src/builtin_scripts.rs`) and can be referenced by **alias** from any `before_serve` / `after_stop` list. The launcher uploads + runs them via SSH per node — no `scp` from the operator's machine. Entry shape (untagged enum, three forms): ```toml before_serve = [ "sudo nvidia-smi -lgc", # bare string = raw shell command { command = "echo hi", on_failure = "warn" }, # raw command + on_failure { script = "stop_dgx_services" }, # built-in alias { script = "stop_dgx_services", on_failure = "warn" }, ] ``` A bare string is always a raw command (today's behavior, unchanged); a built-in is referenced only via `{ script = "" }`. Unknown aliases error at resolution time with the list of known ones. Built-in aliases: | Alias | File | Purpose | |-------|------|---------| | `stop_dgx_services` | `host-scripts/stop-dgx-services.sh` | Stop non-essential services on a DGX node (frees unified memory) | | `restore_dgx_services` | `host-scripts/restore-dgx-services.sh` | Restore services stopped by `stop_dgx_services` | To add a built-in: drop the `.sh` into `host-scripts/`, add an `include_str!` + `BuiltinScript` entry in `src/builtin_scripts.rs`, document the alias here. Scripts stay diffable / shellcheck-able as real files. ## Flags auto-generated - vLLM: `--tensor-parallel-size`, `--pipeline-parallel-size`, `--nnodes`, `--node-rank`, `--master-addr`, `--master-port`, `--host`, `--port`, `--headless` (workers) - SGLang: `--tp`, `--nnodes`, `--node-rank`, `--dist-init-addr`, `--host`, `--port` - Both: `--distributed-executor-backend` (ray only, mp is auto-detected) ## Docker image builds A recipe's `docker_image` is built on demand from `docker-images//` (default `` = recipe name). `sm121 serve --build-node ` (or the standalone `sm121 build`) uploads the dir to a node, runs `bash build.sh` in a tmux session, and stamps a `sm121.checksum` label; later serves skip the build when the label matches a recursive SHA-256 of the dir. **Pinned builds** (`docker-images//versions.env`): every build input (CUDA base, torch/NCCL/flashinfer/vLLM fork+commit, GPU arch, …) is a pinned `KEY=VAL`. `build.sh` is a generic driver — it forwards every pin in `versions.env` as a `--build-arg` to the `Dockerfile` (no hardcoded variable list). Add a pin to `versions.env` and it is picked up automatically. `locks/` holds `--require-hashes` lockfiles; `checksums/` holds verified out-of-band downloads (e.g. tiktoken encodings). See `docker-images/glm52-nf3-hybrid/` for the reference cu130 / aarch64 / sm_121a stack; `docker-images/solar-open2-nvfp4/` is the same stack minus the b12x suite (stock vLLM serve). **Recipe overrides** (`[build.versions]`): a recipe may override any pin without editing the image dir. The launcher merges recipe pins over the image's `versions.env` (recipe wins), writes the merged file to the node, and folds the overrides into the checksum so a pin change triggers a rebuild. Keys must match `versions.env` names the `Dockerfile` declares as `ARG`. ```toml [build.versions] VLLM_COMMIT = "00907fc9b982f973808a0193ad0f8c61fbb62854" # bump fork commit for this recipe only TORCH_VERSION = "2.11.0" ``` ## Composed model directories A recipe may declare `[composed_model]` to combine multiple HF repos into a single model directory at serve time — e.g. a base checkpoint + an expert overlay — without mutating the HF cache. The launcher builds a `/tmp` dir per-node with symlinks into the HF cache (container-side absolute paths) + generated files, bind-mounts it at `mount`, and rewrites `model` to that path. Symlinks point to `/root/.cache/huggingface/...` (the container-side HF cache mount), not the host-side path. They are dangling on the host but resolve correctly inside the container where the HF cache is mounted at that exact path — avoiding the host↔container mount-root mismatch. Later sources win (overwrite earlier symlinks), so order matters for overlays. ```toml [composed_model] mount = "/root/model" # container-side target; `model` is rewritten to this [[composed_model.sources]] # symlinked into the composed dir (later wins) repo = "aidendle94/GLM-5.2-MXFP4-Experts-GPTQ" # base: config, weights, tokenizer # (no glob → whole hierarchy symlinked recursively, incl. subdirs) [[composed_model.sources]] repo = "aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ" # overlay: Lloyd NF4 experts glob = "L*.safetensors" # only expert weight files (glob, may include / for subdirs) [[composed_model.files]] # generated real files (not symlinks) name = "nf4_layers.json" content = '{"layers":[3,4,...,77]}' ``` With **no glob**, the entire snapshot hierarchy is symlinked recursively (every file in every subdir, as individual symlinks with real parent dirs) — so a later source can overlay individual files at any path. With a `glob`, only matching entries are symlinked (one level; `/` allowed for a specific subdir, e.g. `grids/*.json`) and the directory structure is preserved. Allowed glob characters: alphanumeric, `* ? . _ - /`. The composed dir is cleaned up automatically on `sm121 stop`. ## Config fields ```toml ssh_user = "u" ssh_key_path = "/path" # PathBuf [[clusters]] name = "c" master_ib_ip = "10.0.0.1" nodes = [{ name = "n", ssh_ip = "...", ib_ip = "...", rank = 0, entity = "switch.n" }] extra_entities = ["switch.network"] # cluster-level HA entities [nccl."c"] ib_hca = "..." socket_ifname = "..." ib_gid_index = 3 ib_disable = 0 [home_assistant] url = "..." token = "..." shutdown_delay_seconds = 30 [exec_aliases] name = "command" [mount_aliases] # host paths referenced by name from recipe [[extra_mounts]] jit_cache = "/mnt/nas/ai/jit" # alias wins over recipe `host`; lets operators repoint mounts site-wide hf_cache = "/mnt/nas/ai/huggingface" [before_serve."c"] commands = ["cmd", { command = "cmd", on_failure = "warn" }] ``` ## Power - `off`: SSH shutdown → wait → HA `turn_off` per-node entities + cluster extras - `on`: HA `turn_on` (no SSH, requires HA config) - `switch --from --to `: disable nodes/extras unique to `from`, enable all in `to` ## i18n `fluent-i18n` crate, auto-detect via `set_locale(None)`. FTL in `locales/`. ## Constraints - Files ≤500 lines (enforced by `make check-lines`); extract submodules when a file approaches the limit - `#![deny(clippy::wildcard_imports)]` — no `use foo::*;` - **Clippy warnings**: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do **not** add `#[allow(...)]` attributes. If the only viable fix is an allow attribute, consult the user for guidance.