AGENTS.md — sm121

Rust CLI (edition 2024) for launching vLLM/SGLang inference containers across a DGX Spark cluster via SSH (russh). Docker default, tmux alternative.

Dev quick-ref

Action	Command
Build	`cargo build`
All gates	`make check` (fmt → clippy → test → ≤500 lines)
Dry-run	`cargo run -- -d serve --recipe deepseek-v4-flash`
Tests	`cargo test` (65, all passing)
l10n	`locales/{en,ru,es,it,fr,de}/main.ftl` — Fluent FTL files

See DEVELOPMENT.md for architecture, module layout, and dev workflow.

Commands

serve/stop/logs/health/doctor/exec/power/tmux/init-host-keys

Cluster: exact name, prefix (4→4nodes-switch), or auto (smallest with NCCL + matching required_nodes).

Recipe format (`recipes/<name>/recipe.toml`)

engine = "vllm"                     # or "sglang"
docker_image = "img:tag"
model = "org/model"
port = 30000                        # default
required_nodes = 2                  # optional
tensor_parallel_size = 2
pipeline_parallel_size = 1          # vLLM only, 0 = auto
distributed_backend = "mp"          # or "ray"
shm_size = "64gb"                   # optional, only set when non-empty
entrypoint = ["vllm", "serve"]      # optional, after image
before_serve = [{ command = "echo", on_failure = "abort" }]
args = ["--kv-cache-dtype", "fp8"]

[[extra_mounts]]                    # extra bind-mounts for `docker run -v`
alias = "jit_cache"                # resolves via [mount_aliases] in config.toml; wins over `host`
host = "~/.cache/jit"              # fallback default if alias absent; `~` expanded to ssh_user home
container = "/root/.cache/jit"     # container-side target (required)
read_only = false                 # optional, default false (adds :ro)
env = "XDG_CACHE_HOME"            # optional: injects -e <name>=<container path>

# Model path reference (mutually exclusive with alias/host): resolves at
# serve time, per node, from the HF cache — refs/main → latest snapshot, so
# the path survives `huggingface-cli download` updates with no fragile
# symlink. `container` is the placeholder name (e.g. `mtp_draft`) that the
# recipe references as `{{mtp_draft}}` in args/env. The launcher substitutes
# the resolved **container-side** HF cache path per-node — the host cache-dir
# prefix (e.g. `/home/<user>/.cache/huggingface`) is rewritten to the
# container mount target `/root/.cache/huggingface` (no bind-mount — the HF
# cache is already mounted there, so symlinks resolve correctly).
# Absent subfolder → serve aborts with a hint to `huggingface-cli download <repo>`.
[[extra_mounts]]
repo = "aidendle94/GLM-5.2-Hybrid-FP8-MXPF4"
subfolder = "mtp-draft"           # optional; omit = whole snapshot dir
container = "mtp_draft"

[mods]
commands = [                         # run *inside* container before engine
    "pip install flash-attn",
    { command = "./patch.sh", on_failure = "warn" },
]

[env]
KEY = "val"                         # overrides launcher defaults

Env vars (auto-set)

Cluster-driven: NCCL_IB_DISABLE/HCA/SOCKET_IFNAME/GID_INDEX, GLOO_/TP_SOCKET_IFNAME, MN_IF_NAME, UCX_NET_DEVICES, OMPI_MCA_btl_tcp_if_include Defaults: OMP_NUM_THREADS=4, NCCL_IGNORE_CPU_AFFINITY=1, NCCL_NET=IB, NCCL_DEBUG=WARN Engine IP: VLLM_HOST_IP / SGLANG_WORKER_IP Head: VLLM_LOGGING_COLOR=1, TERM=xterm-256color, -p <port>:<port> Ray: RAY_ADDRESS/PORT/NODE_IP_ADDRESS/OVERRIDE_NODE_IP_ADDRESS Recipe [env] wins (last wins in -e).

Mods

Mods run inside the container before the engine process via entrypoint wrapping. When [mods] is present, the recipe must also set entrypoint (the known process to exec at the end). Same on_failure semantics as before_serve (default: abort). On failure the container exits; configure docker run --restart externally to restart on crash.

Implementation: --entrypoint sh image -c 'mods && exec <original entrypoint>'

Flags auto-generated

vLLM: --tensor-parallel-size, --pipeline-parallel-size, --nnodes, --node-rank, --master-addr, --master-port, --host, --port, --headless (workers)
SGLang: --tp, --nnodes, --node-rank, --dist-init-addr, --host, --port
Both: --distributed-executor-backend (ray only, mp is auto-detected)

Config fields

ssh_user = "u"
ssh_key_path = "/path"              # PathBuf

[[clusters]]
name = "c"
master_ib_ip = "10.0.0.1"
nodes = [{ name = "n", ssh_ip = "...", ib_ip = "...", rank = 0, entity = "switch.n" }]
extra_entities = ["switch.network"] # cluster-level HA entities

[nccl."c"]
ib_hca = "..."
socket_ifname = "..."
ib_gid_index = 3
ib_disable = 0

[home_assistant]
url = "..."
token = "..."
shutdown_delay_seconds = 30

[exec_aliases]
name = "command"

[mount_aliases]                     # host paths referenced by name from recipe [[extra_mounts]]
jit_cache = "/mnt/nas/ai/jit"      # alias wins over recipe `host`; lets operators repoint mounts site-wide
hf_cache = "/mnt/nas/ai/huggingface"

[before_serve."c"]
commands = ["cmd", { command = "cmd", on_failure = "warn" }]

Power

off: SSH shutdown → wait → HA turn_off per-node entities + cluster extras
on: HA turn_on (no SSH, requires HA config)
switch --from <src> --to <tgt>: disable nodes/extras unique to from, enable all in to

i18n

fluent-i18n crate, auto-detect via set_locale(None). FTL in locales/.

Constraints

Files ≤800 lines unless impractical; extract submodules when a file exceeds 700 lines (make check-lines)
#![deny(clippy::wildcard_imports)] — no use foo::*;
Clippy warnings: solve by refactoring the code (extract structs, reduce arg count, restructure logic). Do not add #[allow(...)] attributes. If the only viable fix is an allow attribute, consult the user for guidance.

AGENTS.md 5.8 KB تاريخچه خام