launcher/AGENTS.md

AGENTS.md — sm121

Rust CLI (edition 2024) for launching vLLM/SGLang inference containers across a DGX Spark cluster via SSH (russh). Docker default, tmux alternative.

Dev quick-ref

Action	Command
Build	cargo build
All gates	make check (fmt → clippy → test → ≤500 lines)
Dry-run	cargo run -- -d serve --recipe deepseek-v4-flash
Tests	cargo test (65, all passing)
l10n	locales/{en,ru,es,it,fr,de}/main.ftl — Fluent FTL files

See DEVELOPMENT.md for architecture, module layout, and dev workflow.

Commands

serve/stop/logs/health/doctor/exec/power/tmux/init-host-keys

Cluster: exact name, prefix (4→4nodes-switch), or auto (smallest with NCCL + matching required_nodes).

Recipe format (recipes/<name>/recipe.toml)

engine = "vllm"                     # or "sglang"
docker_image = "img:tag"
model = "org/model"
port = 30000                        # default
required_nodes = 2                  # optional
tensor_parallel_size = 2
pipeline_parallel_size = 1          # vLLM only, 0 = auto
distributed_backend = "mp"          # or "ray"
shm_size = "64gb"                   # optional, only set when non-empty
entrypoint = ["vllm", "serve"]      # optional, after image
before_serve = [{ command = "echo", on_failure = "abort" }]
args = ["--kv-cache-dtype", "fp8"]

[mods]
commands = [                         # run *inside* container before engine
    "pip install flash-attn",
    { command = "./patch.sh", on_failure = "warn" },
]

[env]
KEY = "val"                         # overrides launcher defaults

Env vars (auto-set)

Cluster-driven: NCCL_IB_DISABLE/HCA/SOCKET_IFNAME/GID_INDEX, GLOO_/TP_SOCKET_IFNAME, MN_IF_NAME, UCX_NET_DEVICES, OMPI_MCA_btl_tcp_if_include Defaults: OMP_NUM_THREADS=4, NCCL_IGNORE_CPU_AFFINITY=1, NCCL_NET=IB, NCCL_DEBUG=WARN Engine IP: VLLM_HOST_IP / SGLANG_WORKER_IP Head: VLLM_LOGGING_COLOR=1, TERM=xterm-256color, -p <port>:<port> Ray: RAY_ADDRESS/PORT/NODE_IP_ADDRESS/OVERRIDE_NODE_IP_ADDRESS Recipe [env] wins (last wins in -e).

Mods

Mods run inside the container before the engine process via entrypoint wrapping. When [mods] is present, the recipe must also set entrypoint (the known process to exec at the end). Same on_failure semantics as before_serve (default: abort). On failure the container exits; configure docker run --restart externally to restart on crash.

Implementation: --entrypoint sh image -c 'mods && exec <original entrypoint>'

Flags auto-generated

• vLLM: --tensor-parallel-size, --pipeline-parallel-size, --nnodes, --node-rank, --master-addr, --master-port, --host, --port, --headless (workers)
• SGLang: --tp, --nnodes, --node-rank, --dist-init-addr, --host, --port
• Both: --distributed-executor-backend (ray only, mp is auto-detected)

Config fields

ssh_user = "u"
ssh_key_path = "/path"              # PathBuf

[[clusters]]
name = "c"
master_ib_ip = "10.0.0.1"
nodes = [{ name = "n", ssh_ip = "...", ib_ip = "...", rank = 0, entity = "switch.n" }]
extra_entities = ["switch.network"] # cluster-level HA entities

[nccl."c"]
ib_hca = "..."
socket_ifname = "..."
ib_gid_index = 3
ib_disable = 0

[home_assistant]
url = "..."
token = "..."
shutdown_delay_seconds = 30

[exec_aliases]
name = "command"

[before_serve."c"]
commands = ["cmd", { command = "cmd", on_failure = "warn" }]

Power

• off: SSH shutdown → wait → HA turn_off per-node entities + cluster extras
• on: HA turn_on (no SSH, requires HA config)
• switch --from <src> --to <tgt>: disable nodes/extras unique to from, enable all in to

i18n

fluent-i18n crate, auto-detect via set_locale(None). FTL in locales/.

Constraints

• Files ≤800 lines unless impractical; extract submodules when a file exceeds 700 lines (make check-lines)
• #![deny(clippy::wildcard_imports)] — no use foo::*;
• Clippy warnings: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do not add #[allow(...)] attributes. If the only viable fix is an allow attribute, consult the user for guidance.