launcher/AGENTS.md

AGENTS.md — sm121

Rust CLI (edition 2024) for launching vLLM/SGLang inference containers across a DGX Spark cluster via SSH (russh). Docker default, tmux alternative.

Dev quick-ref

Action	Command
Build	cargo build
All gates	make check (fmt → clippy → test → ≤500 lines)
Dry-run	cargo run -- -d serve --recipe deepseek-v4-flash
Tests	cargo test (65, all passing)
l10n	locales/{en,ru,es,it,fr,de}/main.ftl — Fluent FTL files

See DEVELOPMENT.md for architecture, module layout, and dev workflow.

Commands

serve/stop/logs/health/doctor/exec/power/tmux/init-host-keys

Cluster: exact name, prefix (4→4nodes-switch), or auto (smallest with NCCL + matching required_nodes).

⚠️ Cluster may be serving live traffic. A model (often the one serving this very agent) may be running on the cluster at any time. Before running stop, power off, or power switch, ALWAYS first run health and confirm with the user that the cluster is not serving before proceeding. Do NOT assume the cluster is idle — a stray stop/power off has killed running agents mid-task before. logs, health, doctor, and dry-runs (-d) are always safe.

Recipe format (recipes/<name>/recipe.toml)

engine = "vllm"                     # or "sglang"
docker_image = "img:tag"
model = "org/model"
port = 30000                        # default
required_nodes = 2                  # optional
tensor_parallel_size = 2
pipeline_parallel_size = 1          # vLLM only, 0 = auto
distributed_backend = "mp"          # or "ray"
shm_size = "64gb"                   # optional, only set when non-empty
entrypoint = ["vllm", "serve"]      # optional, after image
before_serve = [{ command = "echo", on_failure = "abort" }]
after_stop = [{ script = "restore_dgx_services" }]   # runs after `sm121 stop`
args = ["--kv-cache-dtype", "fp8"]

[[extra_mounts]]                    # extra bind-mounts for `docker run -v`
alias = "jit_cache"                # resolves via [mount_aliases] in config.toml; wins over `host`
host = "~/.cache/jit"              # fallback default if alias absent; `~` expanded to ssh_user home
container = "/root/.cache/jit"     # container-side target (required)
read_only = false                 # optional, default false (adds :ro)
env = "XDG_CACHE_HOME"            # optional: injects -e <name>=<container path>

# Model path reference (mutually exclusive with alias/host): resolves at
# serve time, per node, from the HF cache — refs/main → latest snapshot, so
# the path survives `huggingface-cli download` updates with no fragile
# symlink. `container` is the placeholder name (e.g. `mtp_draft`) that the
# recipe references as `{{mtp_draft}}` in args/env. The launcher substitutes
# the resolved **container-side** HF cache path per-node — the host cache-dir
# prefix (e.g. `/home/<user>/.cache/huggingface`) is rewritten to the
# container mount target `/root/.cache/huggingface` (no bind-mount — the HF
# cache is already mounted there, so symlinks resolve correctly).
# Absent subfolder → serve aborts with a hint to `huggingface-cli download <repo>`.
[[extra_mounts]]
repo = "aidendle94/GLM-5.2-Hybrid-FP8-MXPF4"
subfolder = "mtp-draft"           # optional; omit = whole snapshot dir
container = "mtp_draft"

[mods]
commands = [                         # run *inside* container before engine
    "pip install flash-attn",
    { command = "./patch.sh", on_failure = "warn" },
]

[env]
KEY = "val"                         # overrides launcher defaults

Env vars (auto-set)

Cluster-driven: NCCL_IB_DISABLE/HCA/SOCKET_IFNAME/GID_INDEX, GLOO_/TP_SOCKET_IFNAME, MN_IF_NAME, UCX_NET_DEVICES, OMPI_MCA_btl_tcp_if_include Defaults: OMP_NUM_THREADS=4, NCCL_IGNORE_CPU_AFFINITY=1, NCCL_NET=IB, NCCL_DEBUG=WARN Engine IP: VLLM_HOST_IP / SGLANG_WORKER_IP Head: VLLM_LOGGING_COLOR=1, TERM=xterm-256color, -p <port>:<port> Ray: RAY_ADDRESS/PORT/NODE_IP_ADDRESS/OVERRIDE_NODE_IP_ADDRESS Recipe [env] wins (last wins in -e).

Mods

Mods run inside the container before the engine process via entrypoint wrapping. When [mods] is present, the recipe must also set entrypoint (the known process to exec at the end). Same on_failure semantics as before_serve (default: abort). On failure the container exits; configure docker run --restart externally to restart on crash.

Implementation: --entrypoint sh image -c 'mods && exec <original entrypoint>'

Built-in scripts (baked into the binary)

Host scripts under host-scripts/ are embedded at compile time (include_str! in src/builtin_scripts.rs) and can be referenced by alias from any before_serve / after_stop list. The launcher uploads

• runs them via SSH per node — no scp from the operator's machine.

Entry shape (untagged enum, three forms):

before_serve = [
  "sudo nvidia-smi -lgc",                       # bare string = raw shell command
  { command = "echo hi", on_failure = "warn" },  # raw command + on_failure
  { script = "stop_dgx_services" },              # built-in alias
  { script = "stop_dgx_services", on_failure = "warn" },
]

A bare string is always a raw command (today's behavior, unchanged); a built-in is referenced only via { script = "<alias>" }. Unknown aliases error at resolution time with the list of known ones.

Built-in aliases:

Alias	File	Purpose
stop_dgx_services	host-scripts/stop-dgx-services.sh	Stop non-essential services on a DGX node (frees unified memory)
restore_dgx_services	host-scripts/restore-dgx-services.sh	Restore services stopped by stop_dgx_services

To add a built-in: drop the .sh into host-scripts/, add an include_str! + BuiltinScript entry in src/builtin_scripts.rs, document the alias here. Scripts stay diffable / shellcheck-able as real files.

Flags auto-generated

• vLLM: --tensor-parallel-size, --pipeline-parallel-size, --nnodes, --node-rank, --master-addr, --master-port, --host, --port, --headless (workers)
• SGLang: --tp, --nnodes, --node-rank, --dist-init-addr, --host, --port
• Both: --distributed-executor-backend (ray only, mp is auto-detected)

Config fields

ssh_user = "u"
ssh_key_path = "/path"              # PathBuf

[[clusters]]
name = "c"
master_ib_ip = "10.0.0.1"
nodes = [{ name = "n", ssh_ip = "...", ib_ip = "...", rank = 0, entity = "switch.n" }]
extra_entities = ["switch.network"] # cluster-level HA entities

[nccl."c"]
ib_hca = "..."
socket_ifname = "..."
ib_gid_index = 3
ib_disable = 0

[home_assistant]
url = "..."
token = "..."
shutdown_delay_seconds = 30

[exec_aliases]
name = "command"

[mount_aliases]                     # host paths referenced by name from recipe [[extra_mounts]]
jit_cache = "/mnt/nas/ai/jit"      # alias wins over recipe `host`; lets operators repoint mounts site-wide
hf_cache = "/mnt/nas/ai/huggingface"

[before_serve."c"]
commands = ["cmd", { command = "cmd", on_failure = "warn" }]

Power

• off: SSH shutdown → wait → HA turn_off per-node entities + cluster extras
• on: HA turn_on (no SSH, requires HA config)
• switch --from <src> --to <tgt>: disable nodes/extras unique to from, enable all in to

i18n

fluent-i18n crate, auto-detect via set_locale(None). FTL in locales/.

Constraints

• Files ≤500 lines (enforced by make check-lines); extract submodules when a file approaches the limit
• #![deny(clippy::wildcard_imports)] — no use foo::*;
• Clippy warnings: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do not add #[allow(...)] attributes. If the only viable fix is an allow attribute, consult the user for guidance.