# AGENTS.md — sm121 Rust CLI (`edition 2024`) for launching vLLM/SGLang inference containers across a DGX Spark cluster via SSH (`russh`). Docker default, tmux alternative. ## Dev quick-ref | Action | Command | |--------|---------| | Build | `cargo build` | | All gates | `make check` (fmt → clippy → test → ≤500 lines) | | Dry-run | `cargo run -- -d serve --recipe deepseek-v4-flash` | | Tests | `cargo test` (65, all passing) | | l10n | `locales/{en,ru,es,it,fr,de}/main.ftl` — Fluent FTL files | See [`DEVELOPMENT.md`](DEVELOPMENT.md) for architecture, module layout, and dev workflow. ## Commands ``` serve/stop/logs/health/doctor/exec/power/tmux/init-host-keys ``` Cluster: exact name, prefix (`4`→`4nodes-switch`), or `auto` (smallest with NCCL + matching `required_nodes`). `serve --solo` overrides cluster selection: finds the first free node across all clusters and runs one single-node container there (no `--nnodes`/`--tp`/ `--master-addr` flags — the engine defaults to single-node). `--no-solo` forces multi-node even if the recipe sets `solo = true`. A recipe can also set `solo = true` as the default. `serve --force` skips the occupancy check during cluster selection — occupied nodes are kept in the available set (reachable is still required). Use to run a lightweight test recipe alongside a live serve, or to force onto a busy node. Dangerous for GPU recipes (OOM, port conflicts); the operator is responsible for coexistence. > ⚠️ **Cluster may be serving live traffic.** A model (often the one serving this > very agent) may be running on the cluster at any time. Before running `stop`, > `power off`, or `power switch`, ALWAYS first run `health` and confirm with the > user that the cluster is not serving before proceeding. Do NOT assume the > cluster is idle — a stray `stop`/`power off` has killed running agents mid-task > before. `logs`, `health`, `doctor`, and dry-runs (`-d`) are always safe. ## Recipe format (`recipes//recipe.toml`) ```toml engine = "vllm" # or "sglang" docker_image = "img:tag" model = "org/model" port = 30000 # default required_nodes = 2 # optional; also caps the launch set (N nodes, not all) solo = false # optional; single-node on first free node, no dist flags deploy = "docker" # optional; "docker" (default) or "tmux" venv = "~/venvs/vllm025" # optional; Python venv for tmux (activates before entrypoint) venv_setup = [...] # optional; commands to create venv if it doesn't exist (tmux only) venv_setup_script = "venv.sh" # optional; script file (relative to recipe dir) to create venv (tmux only) tensor_parallel_size = 2 pipeline_parallel_size = 1 # vLLM only, 0 = auto distributed_backend = "mp" # or "ray" shm_size = "64gb" # optional, only set when non-empty entrypoint = ["vllm", "serve"] # optional, after image; used by both docker and tmux docker_image_dir = "glm52-nf3-hybrid" # optional; build dir under docker-images/ (default: recipe name) before_serve = [{ command = "echo", on_failure = "abort" }] after_stop = [{ script = "restore_dgx_services" }] # runs after `sm121 stop` args = ["--kv-cache-dtype", "fp8"] [[extra_mounts]] # extra bind-mounts for `docker run -v` alias = "jit_cache" # resolves via [mount_aliases] in config.toml; wins over `host` host = "~/.cache/jit" # fallback default if alias absent; `~` expanded to ssh_user home container = "/root/.cache/jit" # container-side target (required) read_only = false # optional, default false (adds :ro) env = "XDG_CACHE_HOME" # optional: injects -e = # Model path reference (mutually exclusive with alias/host): resolves at # serve time, per node, from the HF cache — refs/main → latest snapshot, so # the path survives `huggingface-cli download` updates with no fragile # symlink. `container` is the placeholder name (e.g. `mtp_draft`) that the # recipe references as `{{mtp_draft}}` in args/env. The launcher substitutes # the resolved **container-side** HF cache path per-node — the host cache-dir # prefix (e.g. `/home//.cache/huggingface`) is rewritten to the # container mount target `/root/.cache/huggingface` (no bind-mount — the HF # cache is already mounted there, so symlinks resolve correctly). # Absent subfolder → serve aborts with a hint to `huggingface-cli download `. [[extra_mounts]] repo = "aidendle94/GLM-5.2-Hybrid-FP8-MXPF4" subfolder = "mtp-draft" # optional; omit = whole snapshot dir container = "mtp_draft" [mods] commands = [ # run *inside* container before engine "pip install flash-attn", { command = "./patch.sh", on_failure = "warn" }, ] [env] KEY = "val" # overrides launcher defaults [build.versions] # optional; overrides pins in docker-images//versions.env VLLM_COMMIT = "00907fc..." # recipe wins over the image's pin; folded into build checksum ``` ### Deployment target (`deploy`) The `deploy` field selects how the recipe is launched on each node: - `docker` (default): `docker run` with the recipe's `entrypoint`. Supports all features (mods, composed_model, image sync, extra_mounts). - `tmux`: a detached `tmux` session running the engine directly (no Docker). The recipe's `entrypoint` is used when present (e.g. `["vllm", "serve"]` → `vllm serve`); otherwise the launcher falls back to the engine's Python module path. Docker-only features are rejected: `[mods]`, `[composed_model]`, image sync, and `--sync-image`/`--build-node`/ `--force-build` CLI opts are ignored with a warning. The `serve` command dispatches based on `recipe.deploy`: ```toml engine = "vllm" deploy = "tmux" # or "docker" (default) entrypoint = ["vllm", "serve"] ``` Engine-specific args (`args`) are interpreted by the engine itself — the launcher doesn't validate them, so the same `args` list works for both `vllm` and `sglang` recipes. The `entrypoint` field is shared: both deployment paths use it, so a single recipe can be served via either `docker` or `tmux` without changes. ### Venv creation (`venv` + `venv_setup`) When `deploy = "tmux"`, the `venv` field specifies a Python virtualenv path that is activated before the entrypoint (`source /bin/activate && ...`). The `venv_setup` field is an optional set of shell commands to create and initialize the venv **if it does not already exist** on a node. Before serve, the launcher checks `test -f /bin/activate` on each node; if absent, it runs the setup commands in order, then re-checks. A failure aborts the serve. `venv_setup` accepts either an inline multi-line string (one command per line; blank lines and `#` comments are skipped) or a `Vec` list: ```toml deploy = "tmux" venv = "~/venvs/vllm025" venv_setup = """ uv venv ~/venvs/vllm025 -p 3.12 uv pip install vllm==0.25.1 --torch-backend=cu130 uv pip install flashinfer-python==0.6.15.dev20260712 \ flashinfer-cubin==0.6.15.dev20260712 \ flashinfer-jit-cache==0.6.15.dev20260712 \ --extra-index-url https://flashinfer.ai/whl/nightly/ \ --extra-index-url https://flashinfer.ai/whl/nightly/cu130/ hf download poolside/Laguna-S-2.1-NVFP4 hf download poolside/Laguna-S-2.1-DFlash-NVFP4 """ ``` Or as a list: ```toml venv_setup = [ "uv venv ~/venvs/vllm025 -p 3.12", "uv pip install vllm==0.25.1 --torch-backend=cu130", ] ``` Alternatively, `venv_setup_script` references a shell script file (relative to the recipe directory) that is read and run on each node if the venv doesn't exist. Script file takes precedence over `venv_setup` if both are set: ```toml venv_setup_script = "venv_setup.sh" ``` If the venv already exists on a node, the setup commands are skipped. `venv_setup` and `venv_setup_script` are ignored in Docker mode (the container has its own Python). ## Env vars (auto-set) Cluster-driven: `NCCL_IB_DISABLE/HCA/SOCKET_IFNAME/GID_INDEX`, `GLOO_/TP_SOCKET_IFNAME`, `MN_IF_NAME`, `UCX_NET_DEVICES`, `OMPI_MCA_btl_tcp_if_include` Defaults: `OMP_NUM_THREADS=4`, `NCCL_IGNORE_CPU_AFFINITY=1`, `NCCL_NET=IB`, `NCCL_DEBUG=WARN` Engine IP: `VLLM_HOST_IP` / `SGLANG_WORKER_IP` Head: `VLLM_LOGGING_COLOR=1`, `TERM=xterm-256color`, `-p :` Ray: `RAY_ADDRESS/PORT/NODE_IP_ADDRESS/OVERRIDE_NODE_IP_ADDRESS` Recipe `[env]` wins (last wins in `-e`). ## Mods Mods run **inside** the container before the engine process via entrypoint wrapping. When `[mods]` is present, the recipe **must** also set `entrypoint` (the known process to `exec` at the end). Same `on_failure` semantics as `before_serve` (default: `abort`). On failure the container exits; configure `docker run --restart` externally to restart on crash. Implementation: `--entrypoint sh image -c 'mods && exec '` ## Built-in scripts (baked into the binary) Host scripts under `host-scripts/` are embedded at compile time (`include_str!` in `src/builtin_scripts.rs`) and can be referenced by **alias** from any `before_serve` / `after_stop` list. The launcher uploads + runs them via SSH per node — no `scp` from the operator's machine. Entry shape (untagged enum, three forms): ```toml before_serve = [ "sudo nvidia-smi -lgc", # bare string = raw shell command { command = "echo hi", on_failure = "warn" }, # raw command + on_failure { script = "stop_dgx_services" }, # built-in alias { script = "stop_dgx_services", on_failure = "warn" }, ] ``` A bare string is always a raw command (today's behavior, unchanged); a built-in is referenced only via `{ script = "" }`. Unknown aliases error at resolution time with the list of known ones. Built-in aliases: | Alias | File | Purpose | |-------|------|---------| | `stop_dgx_services` | `host-scripts/stop-dgx-services.sh` | Stop non-essential services on a DGX node (frees unified memory) | | `restore_dgx_services` | `host-scripts/restore-dgx-services.sh` | Restore services stopped by `stop_dgx_services` | To add a built-in: drop the `.sh` into `host-scripts/`, add an `include_str!` + `BuiltinScript` entry in `src/builtin_scripts.rs`, document the alias here. Scripts stay diffable / shellcheck-able as real files. ## Flags auto-generated - vLLM: `--tensor-parallel-size`, `--pipeline-parallel-size`, `--nnodes`, `--node-rank`, `--master-addr`, `--master-port`, `--host`, `--port`, `--headless` (workers) - SGLang: `--tp`, `--nnodes`, `--node-rank`, `--dist-init-addr`, `--host`, `--port` - Both: `--distributed-executor-backend` (ray only, mp is auto-detected) ## Docker image builds A recipe's `docker_image` is built on demand from `docker-images//` (default `` = recipe name). `sm121 serve --build-node ` (or the standalone `sm121 build`) uploads the dir to a node, runs `bash build.sh` in a tmux session, and stamps a `sm121.checksum` label; later serves skip the build when the label matches a recursive SHA-256 of the dir. **Pinned builds** (`docker-images//versions.env`): every build input (CUDA base, torch/NCCL/flashinfer/vLLM fork+commit, GPU arch, …) is a pinned `KEY=VAL`. `build.sh` is a generic driver — it forwards every pin in `versions.env` as a `--build-arg` to the `Dockerfile` (no hardcoded variable list). Add a pin to `versions.env` and it is picked up automatically. `locks/` holds `--require-hashes` lockfiles; `checksums/` holds verified out-of-band downloads (e.g. tiktoken encodings). See `docker-images/glm52-nf3-hybrid/` for the reference cu130 / aarch64 / sm_121a stack; `docker-images/solar-open2-nvfp4/` is the same stack minus the b12x suite (stock vLLM serve). **Recipe overrides** (`[build.versions]`): a recipe may override any pin without editing the image dir. The launcher merges recipe pins over the image's `versions.env` (recipe wins), writes the merged file to the node, and folds the overrides into the checksum so a pin change triggers a rebuild. Keys must match `versions.env` names the `Dockerfile` declares as `ARG`. ```toml [build.versions] VLLM_COMMIT = "00907fc9b982f973808a0193ad0f8c61fbb62854" # bump fork commit for this recipe only TORCH_VERSION = "2.11.0" ``` ### Structured output under speculative decoding (glm52) The glm52 image patches vLLM's structured-output reasoning-boundary logic (upstream PRs #44297 + #44993) to fix guided decoding (`response_format`) under MTP + reasoning parser. Without the patch, when the reasoning parser emits the reasoning-end marker (``) under speculative decoding, the grammar FSM fails to advance, so the content channel is unconstrained — models emit prose or markdown-fence prefixes before JSON (tool-eval-bench Structured Output: 50% instead of 100%). The patch is applied at image build time by `docker-images/glm52/patch-structured-output.py`, which modifies three files in the installed vLLM package: 1. `v1/structured_output/request.py` — adds `reasoning_end_token_index` field 2. `v1/structured_output/__init__.py` — `should_advance()` accepts `new_token_ids` (authoritative step delta), returns `True` for all constraint types at the boundary, records the index; `grammar_bitmask()` detects mid-window reasoning end and constrains subsequent drafts; adds `_find_reasoning_end_index()` and `trim_reasoning_for_advance()` 3. `v1/core/sched/scheduler.py` — `update_from_output()` passes `new_token_ids` to `should_advance()` and trims reasoning content before `accept_tokens()` Fixes: vLLM #34650, #43388, #48228. The b12x fork predates the upstream merge (Jul 23, 2026); this patch is a forward-port until the fork rebases. ## Composed model directories A recipe may declare `[composed_model]` to combine multiple HF repos into a single model directory at serve time — e.g. a base checkpoint + an expert overlay — without mutating the HF cache. The launcher builds a `/tmp` dir per-node with symlinks into the HF cache (container-side absolute paths) + generated files, bind-mounts it at `mount`, and rewrites `model` to that path. Symlinks point to `/root/.cache/huggingface/...` (the container-side HF cache mount), not the host-side path. They are dangling on the host but resolve correctly inside the container where the HF cache is mounted at that exact path — avoiding the host↔container mount-root mismatch. Later sources win (overwrite earlier symlinks), so order matters for overlays. ```toml [composed_model] mount = "/root/model" # container-side target; `model` is rewritten to this [[composed_model.sources]] # symlinked into the composed dir (later wins) repo = "aidendle94/GLM-5.2-MXFP4-Experts-GPTQ" # base: config, weights, tokenizer # (no glob → whole hierarchy symlinked recursively, incl. subdirs) [[composed_model.sources]] repo = "aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ" # overlay: Lloyd NF4 experts glob = "L*.safetensors" # only expert weight files (glob, may include / for subdirs) exclude = ["L5.safetensors"] # optional: skip these paths (keep earlier source's version) [[composed_model.files]] # generated real files (not symlinks) name = "nf4_layers.json" content = '{"layers":[3,4,...,77]}' ``` With **no glob**, the entire snapshot hierarchy is symlinked recursively (every file in every subdir, as individual symlinks with real parent dirs) — so a later source can overlay individual files at any path. With a `glob`, only matching entries are symlinked (one level; `/` allowed for a specific subdir, e.g. `grids/*.json`) and the directory structure is preserved. `exclude` (optional list of path patterns) skips matched files so an earlier source's version survives — e.g. `exclude = ["L5.safetensors"]` keeps the base checkpoint's `L5.safetensors` instead of the overlay's. Allowed characters (glob + exclude): alphanumeric, `* ? . _ - /`. The composed dir is cleaned up automatically on `sm121 stop`. ## Serving notes (DeepSeek-V4-Flash / DSpark on GB10) Operational findings from the SvangenStudios agent-serving benchmark (v0.1.1, measured on this exact stack: DS4-Flash-0731, DSpark k=5, 2× DGX Spark, TP=2). See `specs/tech-architecture/dgx-spark-agent-serving-findings.md` for full detail. **`repetition_penalty` crashes the DSpark path** — an illegal-memory-access that kills the serving container. Any gateway/proxy in front of an `sm121`-served DSpark model **must filter `repetition_penalty`** from requests before forwarding. `presence_penalty` and `frequency_penalty` are verified safe. This is a DSpark-stack bug, not model behavior; the benchmark explicitly did not repeat the destructive test. **UMA memory gate (vLLM #48140)** — on GB10 (unified memory), vLLM's startup KV-cache budget check effectively reads Linux `MemFree`, so ~20 GB of reclaimable page cache (created by *reading the weights*) is booked as unavailable. The effect: large `--max-model-len` profiles (128K–262K+) may **deterministically refuse to boot** even when the memory should fit, while 32K–64K boots normally. Two candidate mitigations: - run `echo 3 > /proc/sys/vm/drop_caches` (or the built-in `drop_caches` mod from eugr's repo) as a `before_serve` hook before the engine starts; - a local UMA-aware patch of `gpu_worker.py` to use `MemAvailable` instead of `MemFree`. The `drop_caches` workaround remains unverified on this deployment — test before relying on it for a large-context recipe. **`--gpu-memory-utilization` is a KV lever, not a safety lever** on GB10: weights take ~77.7 GiB of a ~119.6 GiB node, so the GMU ceiling directly squeezes the KV-cache budget. The benchmark found **0.80 boots but dies under traffic**; **0.78** is the safe ceiling. Do not raise GMU to "be safe" — that *reduces* usable KV capacity. The deepseek-v4-flash-eugr / -src / -eugr-agent recipes all ship 0.78. **Three engines** — an LLM server is three nearly-independent resources: prefill (compute-bound, serializes; one 32K prefill saturates the box), decode (bandwidth-bound, scales to ~N=4 concurrency), and admission (scheduler-bound, **blocked during long prefill** — a short request submitted mid-prefill waits until prefill completes, ~221× normal TTFT). `--max-num-batched-tokens` governs how often decode gets a scheduling slot between prefill chunks; **2048 gives 2.9× the decode share of 8192** under a concurrent prefill, for 7.7% prefill throughput. Use the `deepseek-v4-flash-eugr-agent` recipe (2048) for interactive agent traffic; use `deepseek-v4-flash-eugr` (8192) for exclusive long-context batch jobs. ## Config fields ```toml ssh_user = "u" ssh_key_path = "/path" # PathBuf [[clusters]] name = "c" master_ib_ip = "10.0.0.1" nodes = [{ name = "n", ssh_ip = "...", ib_ip = "...", rank = 0, entity = "switch.n" }] extra_entities = ["switch.network"] # cluster-level HA entities [nccl."c"] ib_hca = "..." socket_ifname = "..." ib_gid_index = 3 ib_disable = 0 [home_assistant] url = "..." token = "..." shutdown_delay_seconds = 30 [exec_aliases] name = "command" [mount_aliases] # host paths referenced by name from recipe [[extra_mounts]] jit_cache = "/mnt/nas/ai/jit" # alias wins over recipe `host`; lets operators repoint mounts site-wide hf_cache = "/mnt/nas/ai/huggingface" [before_serve."c"] commands = ["cmd", { command = "cmd", on_failure = "warn" }] ``` ## Power - `off`: SSH shutdown → wait → HA `turn_off` per-node entities + cluster extras - `on`: HA `turn_on` (no SSH, requires HA config) - `switch --from --to `: disable nodes/extras unique to `from`, enable all in `to` ## i18n `fluent-i18n` crate, auto-detect via `set_locale(None)`. FTL in `locales/`. ## Constraints - Files ≤500 lines (enforced by `make check-lines`); extract submodules when a file approaches the limit - `#![deny(clippy::wildcard_imports)]` — no `use foo::*;` - **Clippy warnings**: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do **not** add `#[allow(...)]` attributes. If the only viable fix is an allow attribute, consult the user for guidance.